CI/CD

Pipelines, jobs, runners, the catalog, and the auto-deploy story.

Source

app/models/ci/                    # ~50 ActiveRecord models
├── pipeline.rb, pipeline_schedule.rb
├── build.rb, build_metadata.rb
├── job_artifact.rb, pipeline_artifact.rb
├── stage.rb, status.rb
├── runner.rb, runner_machine.rb, runner_namespace.rb
├── runner_token.rb
├── trigger.rb, trigger_request.rb
├── pipeline_message.rb
└── ...
app/services/ci/                  # ~30 service namespaces
├── create_pipeline_service.rb
├── pipeline_creation/             # parsing and processing pipeline trees
├── pipeline_processing/           # status transitions
├── jobs_finder.rb, runners/, retry_pipeline_service.rb
├── runners/                        # registration, heartbeat
├── job_artifacts/, archive_trace_service.rb
├── catalog/                        # CI/CD Catalog
└── ...
app/workers/ci/                   # ~30 namespaces of pipeline / job workers
lib/gitlab/ci/                    # Pipeline-config DSL, parsers, validators
├── config/
├── status/, build/, runner_releases/
├── reports/                        # security and test reports
└── ...
ee/app/                            # license-aware features (security gates, multi-project pipelines)
ee/lib/api/ci/                    # EE-only CI APIs

Frontend: app/assets/javascripts/ci/, app/assets/javascripts/pipelines/, app/assets/javascripts/runner/, app/assets/javascripts/jobs/, app/assets/javascripts/pipeline_editor/.

Pipeline creation

graph TD
    Push[git push] -->|post-receive| Service[Ci::CreatePipelineService]
    MR[MR event] --> Service
    Sched[Schedule] --> Service
    Service --> Parse[Parse .gitlab-ci.yml]
    Parse --> Resolve[Resolve includes, components]
    Resolve --> Validate[Validate config]
    Validate --> Build[Build pipeline tree]
    Build --> Persist[Persist pipeline + builds]
    Persist --> Process[Ci::PipelineProcessing]
    Process --> Schedule[Schedule first stage]

Highlights:

• Configuration parser is in lib/gitlab/ci/config/. It supports include:, extends:, !reference, default:, parallel:, etc.
• The catalog (lib/gitlab/ci/components/) lets pipelines reuse components from other projects with semantic versioning.
• Validation runs through Ci::Lint (used by the linter UI and CI).
• Persistence creates Ci::Pipeline, Ci::Stage, and Ci::Build records (across the main and ci databases).

Pipeline processing

Ci::PipelineProcessing (app/services/ci/pipeline_processing/) is a state machine driver:

• After a stage finishes, it computes the next stage's runnable jobs.
• It enforces needs: DAG semantics.
• It marks pipelines as success/failure/canceled/skipped.
• It writes pipeline_message records for parser warnings.

Runners

• Ci::Runner, Ci::RunnerMachine, Ci::RunnerNamespace, Ci::RunnerToken model runners and their tokens.
• Registration / heartbeat APIs in lib/api/ci/runners.rb, lib/api/ci/runner.rb.
• Group, project, instance scopes via Ci::Runners::* services.

Job execution

sequenceDiagram
    Runner->>API: POST /api/v4/jobs/request
    API->>Picker: Ci::JobQueue::Picker
    Picker-->>API: pending build (or 204)
    API-->>Runner: build payload (script, vars, artifacts spec)
    Runner->>API: PATCH /api/v4/jobs/:id/trace (chunks)
    API->>Workhorse: Send-Data signed URL for next chunk
    Runner->>API: PUT /api/v4/jobs/:id/artifacts
    API->>Workhorse: Direct upload to object storage
    Runner->>API: PUT /api/v4/jobs/:id (status)
    API->>Service: Ci::Builds::Process

The picker uses Ci::Queue::EntityProxy to find a runnable build for the runner's tags / scope.

Trace storage

CI logs ("traces") are large and append-only. The pipeline:

1. Runner streams chunks to the API (Workhorse-accelerated).
2. Chunks are persisted in Ci::BuildTraceChunk (Redis → object storage migration).
3. After the build completes, the chunks are concatenated and stored as one artifact.
4. After expiry, the trace moves to long-term object storage.

app/services/ci/archive_trace_service.rb orchestrates the move.

Artifacts

• Ci::JobArtifact — per-job artifact (junit, codeclimate, sarif, dotenv, etc.).
• Ci::PipelineArtifact — per-pipeline aggregate (e.g., merged code coverage).
• Stored in object storage; uploaded direct via Workhorse.
• Reports (junit, security findings) parsed by lib/gitlab/ci/reports/.

CI/CD Catalog

app/models/ci/catalog/, app/services/ci/catalog/, app/graphql/types/ci/catalog/. Components are versioned packages of .gitlab-ci.yml snippets that other projects include.

Auto DevOps

app/services/auto_devops/, lib/gitlab/auto_devops_service.rb, plus opinionated CI templates in lib/gitlab/ci/templates/.

EE additions

• Security gates and merge-train automation.
• Multi-project pipelines.
• Compliance frameworks.
• Codequality ratchet.
• DAST, SAST, dependency scanning, container scanning template integrations.

API

• REST: lib/api/ci/*.rb, lib/api/jobs.rb, lib/api/runners.rb, lib/api/pipeline_*.rb.
• GraphQL: app/graphql/types/ci/*, app/graphql/mutations/ci/*.

Related

• Source code management — what triggers pipelines.
• Sidekiq jobs — pipeline processing workers.
• Object storage — artifact storage.
• Workhorse — accelerates artifact uploads / log streaming.