Contributing
Debugging playbook
Symptoms that have happened before, what caused them, and where to look first.
Many of Ghost's hardest bugs passed every test and appeared only at real size, real latency or real concurrency. This table is the institutional memory.
| Symptom | Likely cause | Where |
|---|---|---|
Push fails with no ref update commands | The request stream was left unattended across an await and the command section was dropped | ADR 0013, git-raw-body.middleware.ts |
Pushes above ~1 MiB fail with HTTP 400 … the remote end hung up before data | The flush-only probe answered as a malformed push | ADR 0014, isProbeRequest |
RangeError: data is too long on a huge push | Something holds a pack in a Buffer past 2 GiB | ADR 0015 |
| Clones of one repository return 500 after a strange push | A ref in the log names an object nobody has; pre-0034 logs | ADR 0034, repair-repositories.ts |
| A clone comes back empty with no error | HEAD names a branch that does not exist | ensureHead in the materializer |
| A ref you wrote into a cache disappears | The next materialize reconciles refs to the snapshot | ADR 0012 |
| A fork's pull request reports "conflicts" that are not real | The head's objects were not lent; merge-tree exit 1 with stderr is an error, not a conflict | ADR 0016 |
| Dates off by a few hours | A timestamp without time zone | ADR 0018 |
| 500s under concurrent pushes that look like S3 flakiness | A 409 ConditionalRequestConflict treated as an error instead of a retry | ADR 0005 |
WalContentionError (503) | Eight lost compare-and-swaps in a row on one repository | ADR 0007 |
SSH fetch reports remote transport reported error though it worked | A pipe ended the channel before exit-status | ADR 0019 |
| Large SSH fetches truncated | The connection closed before the channel drained | same |
gh fails with a GraphQL validation error naming a field | The field or type is missing; check gh-capture.spec and the conformance test | GitHub compatibility |
gh can't paginate | Missing Link header | linkHeader |
| e2e tests time out or collide | Run with --no-file-parallelism; check the root .env is not leaking config into the test app | Testing |
| An integration test changed your data | It ran against your dev database. Never set TEST_DATABASE_URL to it | Testing |
Docker image build fails at bun install --frozen-lockfile | A workspace was added without copying its package.json in that Dockerfile | Self-hosting |
General moves
- Reproduce at size. Push a real, large repository (a 2 GiB one has found two bugs). Use a remote bucket, not just RustFS on localhost, when latency could matter.
- Read the log. Download
repos/<id>/indexand decode it (the Storage chapter's script). Compare refs with the cache'sgit for-each-ref. The log is the truth; if the cache disagrees, the cache is wrong. - Wipe the cache. Deleting
$TMPDIR/ghost/<id>.gitis always safe and rules the cache out. - Trace it. With
lgtmrunning, Grafana's Tempo shows each request's spans down topgqueries and S3 calls. - Search the
ponytail:comments near the code; known shortcuts are written there with their ceilings.