Development workflow

A practical walkthrough from idea to merged PR.

1. Find or create a JIRA ticket

Every change is tracked by a JIRA ticket of the form SPARK-NNNNN. The PR title must reference it. Create a ticket with the helper script if you do not have one:

python3 dev/create_spark_jira.py "Title goes here" -c <component> -t <type>

• -c is the JIRA component name (e.g., SQL, Spark Core, PySpark, Connect). python3 dev/create_spark_jira.py --list-components prints the full list.
• -t is one of Bug, Improvement, New Feature, Test, Documentation, Dependency upgrade.
• For a sub-task, pass -p SPARK-PARENT instead of -t.

The script defaults the affected version to the latest unreleased version.

2. Set up remotes

The convention used by the project's developer guides:

# origin is your fork
git remote add origin git@github.com:<you>/spark.git

# upstream is the canonical repo
git remote add upstream https://github.com/apache/spark.git
git fetch upstream master

Always rebase or branch off upstream/master, not your fork's stale tracking branch.

3. Branch and code

git checkout -b SPARK-12345-short-title upstream/master
# ... edit ...

The repo's AGENTS.md recommends working in a git worktree for parallel work on multiple PRs.

Style and ASCII

• The codebase is ASCII-only outside of strings. Smart quotes, em-dashes, and ellipses sneak in easily. Spot-check before committing:

  grep -rn -P "[^\x00-\x7F]" <files>

• Scalastyle, dev/lint-python, and dev/lint-r will catch most issues. See patterns-and-conventions.md.

4. Tests

See testing.md. At minimum, run the suite that covers your change before pushing.

5. Commit and push

• Squash work-in-progress commits before pushing. PRs are usually merged as a single commit.

• Push to your fork (origin), never to apache/spark:

  git push origin SPARK-12345-short-title

• Do not force-push after reviewers have started leaving comments unless asked. Add new commits and let the maintainer squash on merge.

6. Open the PR

• Title format: [SPARK-NNNNN][COMPONENT] Title. The component tag is the uppercase last word of the JIRA component (e.g., Project Infra -> [INFRA], Spark Core -> [CORE], Structured Streaming -> [STREAMING]).
• Base branch: master on apache/spark.
• Description: fill in every section of .github/PULL_REQUEST_TEMPLATE (What changes / Why are the changes needed / Does this PR introduce any user-facing change / How was this patch tested).

7. Investigating CI failures

The Report test results check on a PR is the fastest way to find failures. Do not download full job logs; use the API:

# 1. Get fork owner and head SHA
gh api repos/apache/spark/pulls/<PR>/ --jq '{owner: .head.repo.owner.login, sha: .head.sha}'

# 2. Find the check run id
gh api repos/<owner>/spark/commits/<sha>/check-runs \
  --jq '.check_runs[] | select(.name == "Report test results")'

# 3. Pull annotations (test class, test name, message)
gh api repos/<owner>/spark/check-runs/<id>/annotations

8. Merge

• Committers merge using GitHub's "Squash and merge" or the dev/merge_spark_pr.py helper, which preserves the JIRA tag in the commit message and closes the JIRA ticket automatically.
• Do not merge your own PR unless you are a committer and the change is trivial.

Backports

• Bug fixes are backported by maintainers to the active maintenance branches (branch-4.1, branch-4.0, branch-3.5, ...). New features generally are not.
• A backport PR title is prefixed with the target branch, e.g., [SPARK-12345][CORE][3.5].

Release pulse

The release.yml, publish_snapshot.yml, and build_* workflows under .github/workflows/ gate releases. The dev/create-release/ directory holds the release-cut scripts. Read docs/release-process.md if you ever need to run a release.