david/netwatch

Watch

Fork 0

fix(agent): bound every send so a stalled socket redials instead of waiting #34

Open

david wants to merge 1 commit from fix/agent-send-timeout into main

david commented

2026-11-01 15:06:44 +01:00

Owner

Why

The agent heartbeats, samples metrics and snapshots flows from one select! loop, and every message awaits its write:

_ = metrics_ticker.tick() => {
    if let Some(sample) = sampler.sample() {
        send(&mut socket, &AgentMsg::Metrics { sample }).await?
    }
}

When the TCP path stalls, the write stays pending while the kernel backs the retransmission timer off (0.2 s doubling to 51 s). Nothing leaves the socket, so the hub sees a silent agent and pushes a false "down", then "back online" once the timer recovers. On remote hosts this ran 40 to 110 s.

Evidence

Captured on ewtg-prod while the hub had it marked down (thread dump plus ss -tin at the moment the agent stopped writing):

Send-Q 196854  notsent:192780  cwnd:1  ssthresh:7
rto:51712  backoff:8  lastsnd:13420  bytes_retrans:42661  lost:3  retrans:1/37

All ten agent threads idle in their normal wait states, no blocked syscalls, load 0.16, 16.6 GB free. The agent was not sampling slowly or stuck in a lock: its metrics were queued behind a write the kernel would not put on the wire.

The same host pings the hub with 0% loss but heavy jitter (max 82 to 159 ms, mdev up to 38 ms), which is what collapses cwnd to 1.

What

send() now wraps the write in SEND_TIMEOUT (15 s):

timeout(SEND_TIMEOUT, socket.send(Message::Text(text.into())))
    .await
    .with_context(|| format!("socket stalled for {} s", SEND_TIMEOUT.as_secs()))??;

On expiry the error propagates through the existing .context("sending ...")? sites, the connection is dropped, and the existing reconnect path redials with a fresh congestion window. Worst case silence falls from about two minutes to 15 s, inside the hub's 40 s presence window (10 s stale + 30 s grace). Queued payload is sampled metrics and flows, so nothing worth keeping is lost, and alerts that fail to send already behave this way on any socket error.

Verification

cargo check -p agent --locked: clean.
cargo test -p agent --locked: 126 + 3 + 2 pass, 0 fail.
Reproduced the stall live by restarting the hub, with a probe sampling ss once a second and dumping thread state on silence; the socket counters above are from that capture.

The transport itself (bursty loss and jitter on the IPv6 path between the Contabo agents and the hub) is outside this change; the agent now survives it instead of going quiet.

## Why The agent heartbeats, samples metrics and snapshots flows from one `select!` loop, and every message awaits its write: ```rust _ = metrics_ticker.tick() => { if let Some(sample) = sampler.sample() { send(&mut socket, &AgentMsg::Metrics { sample }).await? } } ``` When the TCP path stalls, the write stays pending while the kernel backs the retransmission timer off (0.2 s doubling to 51 s). Nothing leaves the socket, so the hub sees a silent agent and pushes a false "down", then "back online" once the timer recovers. On remote hosts this ran 40 to 110 s. ## Evidence Captured on `ewtg-prod` while the hub had it marked down (thread dump plus `ss -tin` at the moment the agent stopped writing): ``` Send-Q 196854 notsent:192780 cwnd:1 ssthresh:7 rto:51712 backoff:8 lastsnd:13420 bytes_retrans:42661 lost:3 retrans:1/37 ``` All ten agent threads idle in their normal wait states, no blocked syscalls, load 0.16, 16.6 GB free. The agent was not sampling slowly or stuck in a lock: its metrics were queued behind a write the kernel would not put on the wire. The same host pings the hub with 0% loss but heavy jitter (max 82 to 159 ms, mdev up to 38 ms), which is what collapses `cwnd` to 1. ## What `send()` now wraps the write in `SEND_TIMEOUT` (15 s): ```rust timeout(SEND_TIMEOUT, socket.send(Message::Text(text.into()))) .await .with_context(|| format!("socket stalled for {} s", SEND_TIMEOUT.as_secs()))??; ``` On expiry the error propagates through the existing `.context("sending ...")?` sites, the connection is dropped, and the existing reconnect path redials with a fresh congestion window. Worst case silence falls from about two minutes to 15 s, inside the hub's 40 s presence window (10 s stale + 30 s grace). Queued payload is sampled metrics and flows, so nothing worth keeping is lost, and alerts that fail to send already behave this way on any socket error. ## Verification - `cargo check -p agent --locked`: clean. - `cargo test -p agent --locked`: 126 + 3 + 2 pass, 0 fail. - Reproduced the stall live by restarting the hub, with a probe sampling `ss` once a second and dumping thread state on silence; the socket counters above are from that capture. The transport itself (bursty loss and jitter on the IPv6 path between the Contabo agents and the hub) is outside this change; the agent now survives it instead of going quiet.

david added 1 commit

2026-11-01 15:06:44 +01:00

fix(agent): bound every send so a stalled socket redials instead of waiting

CI / fmt + clippy + test (push) Successful in 3m35s

Details

CI / fmt + clippy + test (pull_request) Successful in 3m39s

Details

acb1ee0538

The agent heartbeats and samples through one select! loop, and each message
awaits its write. When the TCP path stalls, the kernel backs the
retransmission timer off (0.2 s doubling to 51 s) and nothing leaves the
socket; the hub then sees a silent agent and pushes a false 'down'.

Seen on ewtg-prod while it was reported down: Send-Q 196854, notsent 192780,
cwnd:1, ssthresh:7, rto:51712, backoff:8, lastsnd:13420 ms, with the loop's
next metrics already queued behind the stalled write.

The queued data is sampled metrics and flows, so giving up on the connection
costs nothing that matters. Wrap the write in SEND_TIMEOUT (15 s): on expiry
the send returns an error, the connection is dropped, and the existing
reconnect path redials with a fresh congestion window. Worst case falls from
about two minutes of silence to 15 s, inside the hub's 40 s presence window.

CI / fmt + clippy + test (push) Successful in 3m35s

Details

CI / fmt + clippy + test (pull_request) Successful in 3m39s

Details

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/agent-send-timeout:fix/agent-send-timeout

git switch fix/agent-send-timeout

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

david/netwatch!34

No description provided.

Rows
Columns