Debugging

How to find what's wrong, fast.

Logs in development (GDK)

GDK funnels all service logs into one tailable stream:

gdk tail

Per-service logs:

Service	File
Rails	log/development.log
Sidekiq	log/sidekiq.log
Workhorse	log/workhorse.log
Gitaly	log/gitaly.log

JSON logs in production are described in Logging.

Common errors

PG::ConnectionBad: could not connect to server

The database isn't running or config/database.yml is wrong. Run gdk status and gdk start postgresql.

Errno::EADDRINUSE on port 3000

Another GDK or Puma is already running. gdk stop and try again.

ActiveRecord::PendingMigrationError

You pulled new code with new migrations. Run bin/rails db:migrate (and db:migrate:ci / db:migrate:sec for decomposed databases).

Webpack assets are not compiled

Run gdk start webpack or yarn webpack once.

Gitaly is not responding

Look at log/gitaly.log. Often it's a stale socket. gdk restart gitaly clears it.

Sidekiq queues backed up

In the Rails console:

require 'sidekiq/api'
Sidekiq::Queue.all.map { |q| [q.name, q.size] }

You can drain a queue with Sidekiq::Queue.new('foo').clear (development only).

The Rails console

The console is the primary inspection tool:

bin/rails console            # main DB
bin/rails console -e production  # live in production-like env (rare locally)

The full application loads — services, models, finders, all available. Common patterns:

user = User.find_by(username: 'root')
project = Project.find_by_full_path('flightjs/Flight')

# Run a service the way a controller would
Projects::CreateService.new(user, name: 'foo', namespace_id: user.namespace_id).execute

# Inspect Sidekiq
Sidekiq::Stats.new.queues
Sidekiq::ScheduledSet.new.size

# Look at app context
Gitlab::ApplicationContext.current

Performance debugging

The Rails performance bar is enabled in development by default. Click the bar at the top of any page to see:

• SQL queries with EXPLAIN.
• Redis commands.
• Gitaly calls.
• Bullet (N+1) warnings (config/bullet.yml).
• Object allocation.
• Tracepoints.

Backend profiling tools:

• Gitlab::StackProf (lib/gitlab/stack_prof.rb) — sampling profiler.
• Bullet — N+1 query detection.
• gitlab-active_record-explain_analyze — EXPLAIN ANALYZE wrapper.
• lib/gitlab/database/explain.rb — programmatic EXPLAIN.
• peek-mongo, peek-redis, peek-rblineprof, etc., visible in the perf bar.

Memory leaks

lib/gitlab/memory/ contains memory watchers:

• Gitlab::Memory::Watchdog runs in production to restart Sidekiq workers exceeding RSS limits.
• Gitlab::Memory::Reporter dumps heap snapshots.

Locally, bin/rake gitlab:memory:report snapshots a heap dump.

Slow tests

Use --profile 10 with RSpec:

bin/rspec --profile 10 spec/some_spec.rb

Or the test-prof tools that the project includes:

TEST_PROF=1 bin/rspec spec/some_spec.rb

Frontend debugging

• Browser devtools work as expected.
• Vue devtools recommended for component inspection.
• Apollo client devtools for GraphQL.
• Webpack source maps are enabled in development; production builds disable them by default (NO_SOURCEMAPS).

Troubleshooting Gitaly issues

Gitaly is a separate gRPC service. Common failure modes:

• "RPC method not found" — Gitaly version pinned in GITALY_SERVER_VERSION is older than the Rails code expects. Update GDK.
• "context deadline exceeded" — long-running operation hit the Gitaly N+1 detector or call deadline. Set GITALY_DISABLE_REQUEST_LIMITS=1 for local debugging only.
• "broken pipe" — Gitaly process crashed. Look in log/gitaly.log.

Gitaly's own admin/console is described in https://docs.gitlab.com/ee/administration/gitaly/.

When in doubt

The development docs at doc/development/ contain runbooks for many specific subsystems:

• doc/development/database/
• doc/development/sidekiq/
• doc/development/feature_flags/
• doc/development/graphql_guide/
• doc/development/workhorse/
• doc/development/geo/

Search across them with git grep -i 'your error' from the repo root.

Related

• Logging
• Metrics
• Tracing