QA framework

The end-to-end black-box test framework for GitLab. Lives under qa/ in this repo with the test cases and the Capybara/Selenium framework that runs them.

Purpose

Validate that GitLab works as a whole — UI flows, API flows, and component integration — against a running GitLab instance (GDK locally, ephemeral Omnibus in CI).

Source layout

qa/
├── Gemfile, Gemfile.lock      # Independent gem set
├── README.md                  # Framework intro
├── qa.rb                      # Entry point loaded by tests
├── qa/                        # Framework code
│   ├── runtime/               # Test-runner state, env vars
│   ├── support/               # Custom matchers, helpers
│   ├── page/                  # Page Objects (the heart of the framework)
│   ├── resource/              # Test data factories (project, group, MR, issue, ...)
│   ├── service/               # External service helpers (mailhog, smtp, omniauth)
│   ├── specs/                 # The test cases
│   │   └── features/          # User-facing scenarios
│   └── flow/                  # Reusable navigation flows (login, mr review, etc.)
├── spec/                      # Tests *for* the QA framework itself
├── tasks/                     # Rake tasks to run E2E
├── allure/                    # Allure report integration
├── knapsack/                  # Knapsack test parallelization config
├── perf/                      # Performance scenarios
├── performance_test/          # Locust-style performance harness
├── tls_certificates/          # Cert fixtures for SSL tests
└── gdk/                       # Helpers for GDK test environments

How the framework runs

graph TD
    GDK[GDK or Omnibus instance]
    QA[bin/qa Test::Instance::All]
    Capybara[Capybara + Selenium]
    Browser[Headless Chrome]
    API[GitLab REST/GraphQL API]
    Pages[Page Objects]
    Tests[Test specs in qa/specs/features]

    QA --> Tests
    Tests --> Pages
    Pages --> Capybara
    Capybara --> Browser
    Browser --> GDK
    Tests --> API
    API --> GDK

Page objects

Every page in GitLab has a corresponding page object in qa/qa/page/. A page object exposes high-level actions:

class Page::Project::Show < Page::Base
  view 'app/views/projects/_home_panel.html.haml' do
    element :project_name_content
  end

  def project_name
    find_element(:project_name_content).text
  end
end

The view declarations couple page objects to their HAML templates. The CI job qa:selectors enforces that the linked elements actually exist in the views.

Resources

qa/qa/resource/ is a factory layer that creates GitLab data via the API:

project = QA::Resource::Project.fabricate_via_api! do |p|
  p.name = 'my-test'
  p.description = '...'
end

Factories use the GitLab REST/GraphQL API to set up state quickly without driving the browser.

Run modes

# Against GDK
bin/qa Test::Instance::All http://gdk.test:3000

# Smoke-tag only
bin/qa Test::Instance::All http://gdk.test:3000 --tag smoke

# Single spec file
bin/qa Test::Instance::All http://gdk.test:3000 -- qa/specs/features/login_spec.rb

# Parallel via Knapsack
KNAPSACK_GENERATE_REPORT=true bin/qa ...

In CI, the package-and-qa manual job spins up an ephemeral Omnibus container and runs the full e2e suite against it. The pipeline template is at .gitlab/ci/test-on-omnibus/main.gitlab-ci.yml.

Tags and selectors

Tests are tagged for selective execution:

• :smoke — must-pass core flows.
• :reliable — quarantine candidates.
• :transient — known flaky.
• :health_check
• :ee / :foss — edition-specific.
• :orchestrated — needs special infra (mailhog, ldap, etc.).

Extending

New e2e test:

1. Add a spec under qa/qa/specs/features/<feature>/<scenario>_spec.rb.
2. Use existing page objects and resource factories.
3. Tag appropriately.
4. Run locally before pushing: bin/qa Test::Instance::All http://gdk.test:3000 -- <new_spec>.rb.

Allure reporting

qa/allure/ integrates with Allure. CI publishes per-job HTML reports for inspection.

Related

• Testing — entire test pyramid.
• Frontend — the UI under test.