Bound the queue in front of the gate, and count the waits #3

Merged

LNK merged 1 commit from claude/bounded-queue into main

2026-09-04 17:19:32 +00:00

LNK commented

2026-09-04 17:19:31 +00:00

Contributor

What changes. A budget in front of the gate thread, 32 MiB on the wire and 512 jobs. A reader admits each record against it before sending and holds the ticket until the gate has applied the record. While the budget is spent the reader waits, its socket stops being drained, and the collector's own write blocks, which is where the slowness belongs. Nothing is dropped; a collector made to wait captures its next frame later, from the screen as it is then. Control readers do not admit against the budget, so a Stop is never more than the budget deep. The single channel and single thread stay, and with them the ordering argument in the daemon's header.

What is recorded. Slow was the one state the store never marked. Two counters now are that mark, written per heartbeat rather than per record: ingest_backlog_waits and ingest_backlog_high_water_bytes.

Proof. Three tests in here_gate::backlog: a stalled consumer against eight producers sending sixteen thousand records stays inside the budget and loses nothing; an oversized record is admitted alone and the next waits for it; the job cap binds when records are small. The first fails against a budget that never waits, checked by making it one. Linux verifier green on the pinned toolchain: 278 Rust + 86 TypeScript = 364 tests, every stage, exit 0. The end-to-end suite that drives the real gate over real pipes is Windows-only and was not run here; the daemon wiring is compiled and linted.

The third thing the review named: the channel between the reader threads and the gate thread was unbounded, so a gate that fell behind held every frame in memory with no high-water mark, and the Stop button waited behind all of it. **What changes.** A budget in front of the gate thread, 32 MiB on the wire and 512 jobs. A reader admits each record against it before sending and holds the ticket until the gate has applied the record. While the budget is spent the reader waits, its socket stops being drained, and the collector's own write blocks, which is where the slowness belongs. Nothing is dropped; a collector made to wait captures its next frame later, from the screen as it is then. Control readers do not admit against the budget, so a Stop is never more than the budget deep. The single channel and single thread stay, and with them the ordering argument in the daemon's header. **What is recorded.** Slow was the one state the store never marked. Two counters now are that mark, written per heartbeat rather than per record: `ingest_backlog_waits` and `ingest_backlog_high_water_bytes`. **Proof.** Three tests in `here_gate::backlog`: a stalled consumer against eight producers sending sixteen thousand records stays inside the budget and loses nothing; an oversized record is admitted alone and the next waits for it; the job cap binds when records are small. The first fails against a budget that never waits, checked by making it one. Linux verifier green on the pinned toolchain: 278 Rust + 86 TypeScript = 364 tests, every stage, exit 0. The end-to-end suite that drives the real gate over real pipes is Windows-only and was not run here; the daemon wiring is compiled and linted.

LNK added 1 commit

2026-09-04 17:19:31 +00:00

Bound the queue in front of the gate, and count the waits cd34964a96

Every pipe connection has a reader thread, and every reader hands its records to the one
thread that owns the gate. The channel between them was `std::sync::mpsc::channel()`, which
is unbounded. A reader that had pulled a frame off its socket — up to eight mebibytes of
base64 RGBA, as the pipe carries it — put it on that channel and went back for the next one.
So when the gate fell behind, the backlog was in this process's memory, growing, with no
high-water mark and nothing that said so. The Stop button sat on the same channel, behind all
of it: for a client whose promise is that you can turn it off, off was queued behind data.

Now a reader admits each record against a budget before it sends it — 32 MiB on the wire and
512 jobs, both, because six frames fill one and six thousand key-count windows fill the
other — and holds the ticket until the gate has applied it. While the budget is spent the
reader waits, its socket stops being drained, the kernel buffer fills, and the collector's own
`write` blocks, which is where the slowness belongs. A collector made to wait captures its
next frame later, from the screen as it is then, rather than queueing the screen as it was.
Nothing is dropped. Control readers do not admit against the budget, so a Stop is never more
than the budget deep.

The one channel and the one thread stay: the ordering argument in this file's header is real
and this does not touch it. What changes is that the channel can no longer hold more than the
budget, so the argument's cost is now bounded too.

Slow was the one state the store never recorded — off closes the session with a reason, dead
shows as a stale heartbeat, behind left no mark. Two counters are that mark now, written on
each heartbeat rather than each record: how often a reader has had to wait, and the most that
has ever been in flight.

A record larger than the whole budget is admitted when the backlog is empty, rather than
never: a reader waiting for room that cannot exist would hold its collector silent for ever.

Three tests: a stalled consumer against eight producers sending sixteen thousand records
bounds what is in flight and loses nothing; an oversized record goes through alone and the
next one waits for it; the job cap binds when the records are small. The first fails against
a budget that never waits — checked, by making it one.

Not run here: the end-to-end suite that drives the real gate over real pipes is Windows-only.
The daemon wiring is compiled and linted; the bound itself is what the tests prove.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

LNK merged commit 6972fa5fd0 into main

2026-09-04 17:19:32 +00:00

LNK referenced this pull request from a commit

2026-09-04 17:19:33 +00:00

Merge pull request #3 from claude/bounded-queue

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

mikebrandon/here-telemetry-rnd!3

No description provided.

Rows
Columns