fix(hub): index flows(minute) so retention cannot stall ingest #33
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/flows-minute-index"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
retention::downsampletrims flows withDELETE FROM flows WHERE minute < ?, once every five minutes, inside one transaction. The only index on the table is(server, minute), which leads withserver, so the delete could not use it:EXPLAIN QUERY PLANsaidSCAN flowsover 3,445,105 rows.That transaction holds the hub's single
Arc<Mutex<Connection>>, and every agent's metric insert goes through the same mutex. So the scan froze all agent connections at once for tens of seconds,mark_seenstopped, and the hub declared healthy agents offline and pushed "netwatch: down" to ntfy.Observed on ewtg-ops after two more agents were enrolled (three agents, 519 MB database, 3.4M flow rows):
Gaps are measured on
metrics.ts, which the hub stamps at receipt (unix_now()), so they are processing stalls, not agent sampling. The full-scan transaction is the one that fits: periodic, fleet-wide, 10-110 s, worse asflowsgrew.What
Migration v14:
CREATE INDEX IF NOT EXISTS flows_minute ON flows (minute). The delete becomesSEARCH flows USING INDEX flows_minute (minute<?).Also updates the two tests that pinned
SCHEMA_VERSIONat 13, and adds an assertion that v14 creates the index.Verification
EXPLAIN QUERY PLAN DELETE FROM flows WHERE minute < 1:SCAN flowsbefore,SEARCH flows USING INDEX flows_minute (minute<?)after.cargo test -p hub --locked: 177 + 28 + 58 + 20 + 4 pass, 0 fail.The index was applied to the live database by hand during the incident, so the running hub already has it; this PR makes it survive a rebuild, a fresh install, or a restore from an older backup.
Retention trims flows with DELETE FROM flows WHERE minute < ?, once every five minutes, inside one transaction. The only index on the table is (server, minute), so the delete could not use it: EXPLAIN QUERY PLAN said SCAN flows over 3.4M rows while holding the hub's single connection. Every agent's metric insert queues behind that transaction, so fortysomething seconds of scan look like an offline agent. Measured on ewtg-ops (3.4M flow rows, 519MB database): before EXPLAIN QUERY PLAN DELETE FROM flows WHERE minute < 1 -> SCAN flows after -> SEARCH flows USING INDEX flows_minute (minute<?) Metric-arrival gaps per agent, worst over the window: before the index, 13:05-13:29Z dev 58s, prod 53s, notes 12s after the index, 13:29-13:36Z dev 3s, prod 3s, notes 3s Adds migration v14 and a test that the index exists.View command line instructions
Manual merge helper
Use this merge commit message when completing the merge manually.
Checkout
From your project repository, check out a new branch and test the changes.