Metrics

Prometheus-format metrics from every GitLab process.

Source

Concern	Location
Core API	lib/gitlab/metrics/
Exporter abstraction	lib/gitlab/metrics/exporter/
SLIs	lib/gitlab/metrics/sli/
Frontend SLIs	config/user_experience_slis/
Sidekiq metrics	lib/gitlab/metrics/sidekiq.rb
Web metrics	lib/gitlab/metrics/web.rb
Workhorse	workhorse/internal/metrics/
Rack-attack metrics	lib/gitlab/rack_attack/instrumentation.rb

GitLab uses prometheus-client-mmap for in-process accumulation and a separate metrics exporter port for scraping.

Exporter ports

Process	Default port	Endpoint
Web (Puma + metrics-server)	8083	/metrics
Sidekiq	8082	/metrics
Workhorse	9229	/metrics
Gitaly	9236	/metrics
Pages	9235	/metrics

config/initializers/0_metrics.rb boots the in-process exporter; metrics_server/ runs as a sibling process for the web role.

SLIs

Gitlab::Metrics::Sli abstracts the SLI pattern (numerator = good events, denominator = total). Common SLIs:

• gitlab_sli_rails_request_apdex — % of web requests under their urgency target.
• gitlab_sli_rails_request_error — % of web requests with 5xx.
• gitlab_sli_sidekiq_execution_apdex — % of jobs under their urgency target.
• gitlab_sli_sidekiq_execution_error — % of jobs that errored.
• gitlab_sli_graphql_request_apdex — same for GraphQL.
• gitlab_sli_graphql_subscription_messages_apdex — subscriptions.
• gitlab_sli_active_context_* — Active Context retrieval SLIs.

Each SLI has a target threshold defined per-endpoint or per-worker class via metadata.

User experience SLIs

config/user_experience_slis/ declares user-perceptive SLIs (page load times, interactivity). The frontend reports them via Gitlab::Metrics::Sli::Apdex.

Adding a counter / histogram

class MyService
  HISTOGRAM = Gitlab::Metrics.histogram(
    :gitlab_my_service_duration_seconds,
    'My service operation duration',
    {},
    [0.05, 0.1, 0.25, 0.5, 1, 5]
  )

  def execute
    HISTOGRAM.observe({ feature: 'foo' }, time { ... })
  end
end

Conventions:

• Names start with gitlab_.
• Labels: low-cardinality only. Don't put project IDs in labels.
• Add the metric to doc/administration/monitoring/prometheus/gitlab_metrics.md.

Multiprocess pitfalls

Puma, Sidekiq, and other multi-worker processes use prometheus-client-mmap to aggregate counters across forks. Histograms work correctly; gauges with live_sum aggregation work correctly; resetting a counter resets across forks.

Dashboards

GitLab.com uses internal Grafana dashboards. Self-managed customers can use the included Grafana dashboards in omnibus-gitlab or the GitLab metrics dashboards project.

Application performance metrics

The Gitlab::Metrics::WebTransaction and Gitlab::Metrics::BackgroundTransaction instrument:

• DB query duration & count.
• Redis call duration.
• Gitaly call duration.
• Cache hit ratio.
• Object allocation count.

They show up in the perf bar and in production dashboards.

Related

• Logging, Tracing, Alerting.