netwatch

netwatch is a self-hosted server monitor for a handful of VPSes: one small agent per server reports CPU, RAM, disk, network flows and SSH attacks to one hub, which serves this dashboard. It also watches the ways an intruder stays in — a new login account, a new admin, a key added to authorized_keys — and forecasts a disk that is about to fill.

Planned work should not ring your phone: any alert can be acknowledged (it dims, it never disappears), and a server can be muted for 1–168 hours, for every alert kind or just one. A muted alert still lands in the feed; it is simply never pushed, and a server-wide mute also holds that server's down / back-online pushes. A 30-day uptime strip per server comes from the metric buckets the hub already keeps, and the hub database can be downloaded from Settings behind a fresh two-factor code.

The fleet can also act as one. The hub downloads Feodo Tracker and Spamhaus DROP daily and every server that opts in blocks those ranges in both directions before anything connects, so a server that still tries to reach one raises a crit Blocklist hit — the strongest sign netwatch can give that a host is compromised — while Tor exits are only tagged and warned, never blocked. One server's SSH auto-ban is copied to every other opted-in server with one push, one row per address, and Unban everywhere lifts it from all of them. Each card and server page also shows pending security updates and whether a reboot is required, read from apt by a root helper: a badge, never a push.

The agent only dials out — nothing listens on your servers and nothing runs a command it was sent.

All screenshots: one hub with test agents. No real addresses appear.

Quick start

Ten minutes for the hub plus your first agent; about two more per extra server. Debian 12 or Ubuntu 24.04 or newer (the release binaries need glibc 2.36), root on each host, and one domain name for the hub.

Blocklists and outbound blocking are tested on Ubuntu 24.04 (nftables 1.0.9); Debian 12 (nftables 1.0.6) is expected to work but untested.

The installer is a POSIX shell script you download, read and then run. There is no curl | sh path on purpose: it writes system users, a config file with your agent token, and a systemd unit, so you should see what it does first.

1. The hub host

curl -fLO https://git.ewtg.eu/david/netwatch/raw/tag/v1.6.0/tools/install.sh
less install.sh                       # 500 lines of sh; nothing hidden
sudo sh install.sh hub --version 1.6.0

It downloads the release tarball for your architecture, verifies it against SHA256SUMS, creates the netwatch-hub user and /var/lib/netwatch, writes /etc/netwatch/hub.toml, installs deploy/netwatch-hub.service and starts it. It then prints the three things left to do:

1. The dashboard password, at least 12 characters, asked twice:

   sudo netwatch-hub /etc/netwatch/hub.toml admin passwd
   

2. TLS. The hub binds 127.0.0.1 only and refuses anything else at startup, so it is Caddy on the same host that faces the internet. The installer printed this snippet with your host already filled in; put it in /etc/caddy/Caddyfile:

   netwatch.example.com {
       encode zstd gzip
       reverse_proxy 127.0.0.1:7000
       header Strict-Transport-Security "max-age=31536000"
   }
   

   sudo systemctl reload caddy
   

3. The first agent's token. Shown once; the hub keeps only its SHA-256:

   sudo netwatch-hub /etc/netwatch/hub.toml agent add notes
   

Open https://netwatch.example.com, sign in with the password, and the first sign-in walks you through TOTP: scan the QR code (or type the base32 key), confirm a code, and save the ten one-use recovery codes it shows once. Every later sign-in needs the password and a current code, or one recovery code.

2. Each agent host

curl -fLO https://git.ewtg.eu/david/netwatch/raw/tag/v1.6.0/tools/install.sh
less install.sh
sudo sh install.sh agent --version 1.6.0

It asks for the hub URL (wss://netwatch.example.com/ws/agent) and then for the token, with the input hidden. The token is never a command-line argument, so it cannot land in ps or a shell history; it is only ever written to /etc/netwatch/agent.toml (mode 0640, root:netwatch). The script installs libpcap0.8 and nftables, creates the netwatch user, installs deploy/netwatch-agent.service with the capabilities it needs, and starts it. It also installs the root key watch (netwatch-keys + netwatch-keys.timer), which writes the authorized_keys digests the agent reads — see "Persistence watch" below.

Last step, on the agent host: put your own administrative IP in the safelist, so a typo in a later sshd_config cannot get you banned by your own monitor.

# /etc/netwatch/agent.toml
[security]
safelist = ["203.0.113.10/32"]

sudo systemctl restart netwatch-agent
systemctl status netwatch-agent          # or: journalctl -u netwatch-agent -f

Your own IP is what the installer printed next to "your current SSH connection comes from"; the loopback, RFC 1918, link-local, CGNAT, ULA ranges and the hub's own addresses are always safe.

Installing is idempotent, and that is how an upgrade works: a re-run over an installed host prints keep: /etc/netwatch/agent.toml (exists), keeps it, and does not ask for the hub URL or the token — both are already on disk. --force replaces the config and the unit, and then asks for both again. --dry-run prints every step and changes nothing. Building from source instead? --binary ./target/release/netwatch-agent.

Upgrading

From 1.5.0: the hub first, then each agent. The new alert kinds and the tables behind the Services panel come with the hub; a 1.5.0 agent keeps reporting what it always did against a 1.6.0 hub, so there is no rush on the agents. After an agent is upgraded, list the units the dashboard may restart in its agent.toml (see Services that stay down) and install polkitd if the installer says it is missing.

From 1.1.x: upgrade the hub before the agents. A 1.1 hub does not know the login message and logs ignoring unparseable agent message for it; the new agent otherwise works against it.

From 1.4.0: the hub first, then each agent, then opt in. A 1.4.0 agent keeps working against a 1.5.0 hub and simply receives no hub bans (it reports accept_hub_bans: false because it has never heard of the field), so there is no rush on the agents. Once an agent runs 1.5.0, decide on that server whether the hub may act on it:

# /etc/netwatch/agent.toml
[security]
accept_hub_bans = true

Restart the agent afterwards. Do this only after checking that safelist holds your own administrative address: the safelist still wins, so a hub ban on it is refused, but it is the line that keeps you out of exactly this situation. Leaving the line out is the 1.4.0 behaviour — the hub sends that server nothing.

ban_cap needs no action. The installer writes it commented, so a server that never spelled it out moves to 5000 with the new binary, and blocklist entries stay outside the cap. An existing agent.toml may still show # ban_cap = 500 — that comment is stale and has no effect; only an uncommented ban_cap = 500 keeps 500 until you edit it, which is the one line to look for if you were near the old limit.

Both roles keep their config, so an upgrade replaces the binary and nothing else. Fetch install.sh from the tag you are moving to (the curl line above, with its version), then:

Hub — it still asks for the public origin, and keeps hub.toml and the database:

sudo NETWATCH_PUBLIC_ORIGIN=https://netwatch.example.com sh install.sh hub --version <new>
sudo systemctl restart netwatch-hub

Agent — it keeps agent.toml and asks for nothing, because the token stays where it is:

sudo sh install.sh agent --version <new>
sudo systemctl restart netwatch-agent

The restart is the step that matters: systemctl enable --now does not restart a unit that is already running, so the old binary keeps serving until you restart it. Nothing else changes — the SQLite database, both configs and the admin password are untouched.

Keyboard: / or Ctrl+K jumps to any server or page from anywhere in the dashboard.

Configuration

Two files, both 0640 and created by the installer. Nothing else needs editing.

hub.toml

Key	Default	Meaning
bind	— (required)	127.0.0.1:7000. Loopback only; the hub refuses a public address at startup.
db_path	— (required)	SQLite file, e.g. /var/lib/netwatch/netwatch.db.
public_origin	— (required)	How the browser reaches the hub, e.g. https://netwatch.example.com. The login cookie is Secure, every write must carry exactly this Origin, and the CSP's WebSocket source is derived from it.
stale_after_secs	10	No message for this long and the server shows OFFLINE.
session_idle_secs	43200	12 h without a request signs a session out.
session_max_secs	604800	7 days, absolute.
geoip_db	none	Path to GeoLite2-City.mmdb; without it flows carry no country, city or map point.
[threat_intel] enabled	true	Download the three blocklists daily and send them to the servers that opt in. Off means no downloads, no flow tags and an empty blocklist, which clears the sets on every agent.

Agents live in the hub's database, not in this file: netwatch-hub hub.toml agent add <name> prints a token once, agent list shows them, agent revoke <name> closes the live socket and deletes that server's stored state.

agent.toml

Key	Default	Meaning
hub_url	— (required)	wss://hub.example/ws/agent. Plain ws:// is accepted only for a loopback hub, because the token would otherwise travel in clear text.
token	— (required)	nw_… from agent add.
interface	default route	Which interface to capture on. Omit and the agent picks the default route's, IPv4 first then IPv6.
[security] auth_log	journald	Read this file instead of the journal. It trusts every local user, so it is a fallback.
[security] safelist	[]	CIDRs that are never banned, on top of the ranges that never are.
[security] max_fails	5	Failed logins inside the window that trigger a ban.
[security] window_secs	600	The counting window.
[security] ban_ttl_secs	86400	How long a ban lasts.
[security] ban_cap	5000	Most bans kept at once; at the cap the agent alerts and stops banning. Blocklist entries never count toward it; fleet copies do.
[security] accept_hub_bans	false	Whether the hub may send this server a fleet ban copy or the blocklist. false means the hub sends nothing at all, and the safelist still wins when it is true.
[restart] units	[]	Units the dashboard may restart, matched exactly (nginx.service, getty@tty1.service). Empty means the Restart button never appears. One restart per unit per minute.

SSH auto-ban

The agent reads sshd's failed logins and bans the address itself, through its own nftables table (inet netwatch), so a server protects itself while the hub is down. nft and journalctl are its only child processes, started by bare name, never through a shell; on a ban the address is parsed into an IpAddr and re-printed into fixed arguments.

Bans are per /64 for IPv6, so rotating the host bits inside one network does not reset the count. A ban is re-read from the kernel after a restart, so it still shows and can still be lifted from the dashboard (Unban) or with sudo nft delete table inet netwatch (the agent recreates it, empty, on its next check).

With your IP in safelist you cannot lock yourself out. A secondary public address on a multi-homed host is not discovered automatically, though — put it in safelist.

fail2ban. If a fail2ban-server is running the agent only observes: it replays /var/log/fail2ban.log, reports those bans and never adds its own, and the dashboard shows OBSERVE. Never both. Something missing — no readable log, no nft, no permission — never stops the agent: it keeps sending metrics and flows, shows OFF with the reason in mode_reason, and re-checks every minute.

Where the log comes from. journald first (journalctl --follow --output=cat _COMM=sshd _COMM=sshd-session): _COMM is set by the kernel, so a local user cannot forge an sshd line into it. /var/log/auth.log is used only when journalctl is absent or auth_log names a file explicitly.

Blocklists and fleet bans

The hub downloads three public lists once a day and builds one index:

List	What it is	What netwatch does
Feodo Tracker (abuse.ch), the recommended subset	botnet command-and-control servers	blocked both ways
Spamhaus DROP	hijacked or criminal net-blocks	blocked both ways
Tor Project bulk exit list	Tor exit relays	tagged and warned, never blocked

On every server with accept_hub_bans = true the agent keeps them in four sets in its own inet netwatch table (listed_in4, listed_in6, listed_out4, listed_out6) and drops both directions before the connection happens. The hub sends ranges, never a rule: the agent parses each one into an address and prints it into arguments it builds itself, at most 50 000 per family and never a prefix shorter than /8 (IPv4) or /16 (IPv6). A range that overlaps the safelist is left out of that server's sets and counted.

When an outbound packet hits a listed range, the per-element counters report it once a minute: one crit Blocklist hit per address per day, pushed, carrying the steps to find the process. Inbound hits are counted on the Security page and never alerted — a DROP range being scanned all day is the list doing its job.

Bans travel the other way. A netwatch SSH ban becomes a fleet ban: the hub copies it to every server that accepts hub bans, with the TTL that is left, and the agent tags the copy in its set as fleet:<origin server>. The Active bans table is one row per address, with Source and Servers, and Unban everywhere lifts it from all of them at once; a copy a server refused (safelist, cap, opt-out) is shown as skipped with that agent's own reason.

If a list is wrong, Exempt the range from its alert, or manage the exemptions on the Security page: hub-side, audited, and the range leaves the blocklist and the tags for every server. Turning accept_hub_bans off and restarting the agent flushes the four sets and deletes every copied ban, while the server's own bans stay.

Pending security updates

tools/install.sh agent also installs netwatch-updates and its timer, next to the key watch. Once an hour, as root, it runs apt-get -s -o Debug::NoLocking=1 dist-upgrade, counts the Inst lines and the ones from a -security origin, lists up to 50 security package names and tests /var/run/reboot-required. It writes one small file, /run/netwatch-updates/status (0644), which the unprivileged agent reads every five minutes. It never runs apt update, never installs anything and never takes the package database's lock.

The hub keeps the newest reading per server in memory — no history, no alert, no push — and shows 3 security updates / reboot required on the card and the server page, with the package names and when apt was last read. A reading older than three hours is reported as unknown, so a stopped helper shows as updates unknown with its reason rather than a cheerful zero.

SSH login alerts

Netwatch also reads sshd's Accepted … lines, so the login that matters — the one that gets in — is not silent. The agent sends every successful login (user, method, source address and port, and for a key login the key type and SHA256: fingerprint); the hub stores it for 30 days and judges it against the keys trusted for that server:

• crit, pushed: a key that is not trusted for this server; a trusted key logging in as a user it is not trusted for; any password or keyboard-interactive login; any login from a public address (every server is mesh-only, so a public source means the firewall changed).
• info, no push: a trusted key, as a trusted user, from a source address it has never used before — one row, and later logins from that address are quiet.
• quiet: a trusted key, a trusted user, and a known private or mesh source.

The key fingerprint is the identity; the source address is context, because home addresses change and keys do not. On day one every key is unknown, so each login pushes once: trust each key once from the alert (give it a name, choose this server or all servers) and it is quiet from then on. Trusting changes only what the hub alerts on: there is no Ban button, HubToAgent gains nothing, and the agent never edits authorized_keys or ends a session. A public source instead gets a copy-paste nft command for that address under the alert's "What to do".

Logins come from journald only, for the same reason as bans: only its kernel-set _COMM cannot be forged. A host that reads the /var/log/auth.log fallback reports no logins, and the panel says so. On every reconnect the agent replays the journal's last 24 hours, so a login made while the agent was down is still reported with its original time; the hub drops the ones it already has.

Persistence watch and disk forecast

The first things an intruder does to stay in are a new login account, a new admin, or a key added to authorized_keys. The thing that takes a server down quietly is a disk that fills. Both are watched.

Every minute the agent sends the host's login accounts (name, uid, and whether the account is root or in sudo, wheel or admin) and the sha256 of every authorized_keys file. The hub keeps the baseline and alerts on the difference:

• crit, pushed: a new admin account, an account that gains admin, and a new or changed authorized_keys file.
• warn, no push: a new ordinary login account.
• info, no push: an account or a key file that went away.

Package accounts (nologin/false shells) are not login accounts, so an apt install never alerts; useradd -s /bin/bash does, and usermod -aG sudo is a crit. Two things are never read as a removal: an empty account list (every host has root, so it means /etc/passwd could not be read) and a key snapshot whose digests could not be read.

The agent gains no privilege for this. It runs with ProtectHome=yes and cannot see home directories, so one small root helper does that single job: netwatch-keys.timer runs netwatch-keys every minute, and it writes the sha256 of each regular authorized_keys file to /run/netwatch-keys/authorized_keys.sha256 (mode 0644) for the agent to read. It writes digests only, never the key contents; it never follows a symlink; it has no network; and it keeps the previous digests rather than writing an empty file when getent passwd comes back empty. The agent's own unit is unchanged. Stopping the helper shows key watch: … on the Security page and raises no "removed" alerts.

A custom AuthorizedKeysFile path in sshd_config is not watched: the helper reads the default ~/.ssh/authorized_keys and authorized_keys2.

Disk forecast. Once an hour the hub fits a straight line through each server's last week of disk usage (5-minute averages plus the last day of raw samples) and caches the days until full. Under 14 days is a warn, under 3 is a crit push, and the fleet cards and the server page show full in ~N days from warn upwards. Nothing is shown when the hub cannot stand behind the number: less than a day of history, a flat or shrinking disk, an unknown size, a resize inside the window, or a date more than a year out. An alert of the same or a higher severity is not repeated within 24 hours.

Services that stay down

A service that keeps dying, a container that keeps restarting, a backup that silently stopped running and a server that came back from a hard reset are all visible without reading journals on every host.

What pushes, and what only shows.

• crit, pushed: a unit that failed, a unit or container that is in a restart loop, a container that exited non-zero or was OOM-killed, and a scheduled job that is overdue.
• warn, no push: a unit that was already down the first time netwatch looked at that server, and a container that stopped on purpose (exit 0 or 143).
• info, no push: a unit or container that came back, a job that ran again, and a container that was removed.

One alert per outage, not one per check: the unit's or container's … up closes it, and a restart loop is one outage however many times the unit restarts. The hub keeps the reason the outage started — a unit's systemd Result, a container's exit code — even across a hub restart or an agent reconnect, so the panel and the feed never disagree.

Failed units. Every 30 s the agent asks systemd for units that failed or are in auto-restart (systemctl list-units --all --state=failed,auto-restart), so a failed .mount counts too. Nothing on the server is enabled for this and the agent gains no privilege for it.

Containers. A small root helper, netwatch-containers.service, reads Docker's own state (docker ps plus docker inspect) every minute and writes one line per container to /run/netwatch-containers/status, which the unprivileged agent reads. It never starts, stops or pulls anything, and its unit has After=docker.service with no Wants=, so a stopped Docker stays stopped. A host without Docker shows no containers section at all. A host where Docker is installed but the helper cannot answer shows a muted containers unknown · 12m pill on the fleet card and one warn line in the panel, with the helper's own reason as the title; units and jobs are reported separately and stay visible.

Restart from the dashboard. The panel can restart a unit, and only a unit that server's own root listed in its agent.toml:

# /etc/netwatch/agent.toml
[restart]
units = ["nginx.service", "gunicorn.service"]

The agent matches the request byte for byte against that list and runs systemctl restart -- <unit> with the list's string, so nothing the hub sends becomes an argument; a unit that is not listed shows not restartable · agent.toml and is refused twice — once by the hub, once by the agent. One unit may be restarted at most once a minute.

Debian's minimal install has no polkitd, and without it systemctl restart as the unprivileged agent fails with a permission error, which the panel shows verbatim. Install it once and re-run the agent installer so the rule lands:

sudo apt install polkitd
cd netwatch-agent-* && sudo sh tools/install.sh agent --force
# writes /etc/polkit-1/rules.d/50-netwatch-restart.rules

The rule allows exactly one action (org.freedesktop.systemd1.manage-units), one verb (restart), one user (netwatch) and .service names only: no start, no stop, no enable. The list of units stays in agent.toml, which only root on that server can edit, so a compromised hub cannot widen it by asking. Containers are watched and never restarted from the dashboard: docker restart over SSH is the action there.

Scheduled jobs. Anything that runs on a timer can report success by touching a file in /var/lib/netwatch-heartbeat/ (its own directory, created by the installer as 0755 root — deliberately next to /var/lib/netwatch, the hub's own, rather than inside it):

# /etc/cron.daily/backup
/usr/local/bin/backup.sh && touch /var/lib/netwatch-heartbeat/backup

A job that runs as another user needs the file created once by root and owned by that user:

sudo touch /var/lib/netwatch-heartbeat/backup
sudo chown deploy /var/lib/netwatch-heartbeat/backup

Every file the agent finds there shows in the panel within a minute as NOT WATCHED; choose how often the job should run (1 h … 30 d) in the Expect every column and the choice is saved and audited at once. netwatch adds 10 % grace, never less than 15 minutes, so a 24 h job is only late after about 26.4 h: a run 28 h ago is overdue, a run 26 h ago is not. A late job is one crit push per episode, and Job back closes it. Deleting the file leaves the row as FILE REMOVED with a Forget button, so a job removed on purpose leaves when you say so, not when the file goes. If the directory cannot be read the table is replaced by the reason and nothing is judged — a broken helper is not evidence that a job is late.

Reboots. The agent reports each boot and whether the boot before it ended cleanly, from journald's own MESSAGE_ID=d93fb3c9c24d451a97cea615ce59c00b ("Journal stopped"). A previous boot that did not shut down cleanly — power loss, a kernel panic, a hard reset — is a warn push and a line in the panel. That only works when the journal outlives the boot:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

Without it the panel says cause unknown (journal not persistent) instead of guessing. When an OOM kill happened in the ten minutes before a unit or container went down, that is named too — by the cgroup the kernel killed in, never by a process name, because node is not node-exporter.service.

Update order: the hub first, then the agents. The new alert kinds and the stored state arrive with the hub's schema; a 1.5.0 agent talking to a 1.6.0 hub keeps working and simply reports nothing new, so the panel fills in as each agent is upgraded.

GeoIP (optional)

With a MaxMind GeoLite2-City database the hub adds each flow's country, city and map coordinates, and the server's own location so the map can draw arcs. Get it with a free MaxMind account and geoipupdate, refreshed weekly, then point geoip_db at it:

sudo apt-get install -y geoipupdate
sudo $EDITOR /etc/GeoIP.conf    # AccountID + LicenseKey + EditionIDs GeoLite2-City
sudo systemctl enable --now geoipupdate.timer

geoip_db = "/var/lib/GeoIP/GeoLite2-City.mmdb"

Without it the hub logs one warning, keeps running, and flows simply have no country or point. Private, loopback, link-local, CGNAT, ULA and documentation addresses are never looked up — they show as LAN either way. Lookups are local: the hub never asks anyone about an address.

ntfy pushes

An optional [ntfy] block pushes to your phone — a server going offline or coming back, and every critical alert (a ban, a CPU or disk line crossed):

[ntfy]
url = "https://ntfy.sh/your-topic"   # https, or http:// only for a loopback host
token = "tk_..."                     # optional; sent as Authorization: Bearer
grace_secs = 30                      # extra wait after stale_after

A hub restart re-reads the state without re-sending, and a failed push is retried on the next 5 s tick.

Alert thresholds

Per server, in the dashboard under Settings · Thresholds (CPU, RAM and disk, warn and crit). The defaults are CPU and RAM 85/95 and disk 80/90 percent; a valid pair is 1 ≤ warn < crit ≤ 100, and the page refuses anything else with the reason in words.

A level is entered only when the value stays at or above it for two minutes, so a metric that flaps across the line alerts once, not once per sample. Leaving a level needs the value five points below it for two minutes — that is the hysteresis that keeps a server sitting at 85.4 % from flapping between two states. Crit goes to ntfy as well as the feed. Every edit is one row in the audit log.

A hub restart forgets the in-memory state, so a server that is still hot alerts again two minutes later. That is deliberate: the alternative is a database write per sample.

The ban policy and safelist are read-only in the dashboard. They are edited in agent.toml on the server and picked up by a restart; a compromised hub must not be able to get anyone banned.

Threat model

The full review, one line per check with the test that pins it, is in docs/security-review.md.

An attacker on the internet. Reaches the hub only through TLS on the proxy; the hub binds loopback and refuses anything else at startup. Signing in needs the admin password (argon2id) and then a code. Five failures per client address in fifteen minutes lock that address out, with IPv6 grouped by /64. The session cookie is HttpOnly; Secure; SameSite=Strict, the token is 32 random bytes and only its SHA-256 is stored, writes must carry the hub's exact Origin and a JSON content type, bodies are capped at 64 KiB, and every /api route needs a full session. The CSP allows no inline script or style, so a stored string cannot become code.

A stolen agent token. It is a bearer credential for /ws/agent only: it lets its holder report samples, flows and alerts as that one server, and nothing else — it cannot read another server's data, cannot reach the dashboard, and cannot ask the agent to do anything. A copy of the database does not yield usable tokens, because only the SHA-256 is in it; revoking from Settings closes the live socket immediately.

A compromised hub. It holds every metric and flow netwatch ever received, so assume it reads everything. HubToAgent is still a closed enum, and the agent maps every variant to fixed code — there is no shell and no arbitrary command. Phase 9 widened what it can ask for, and only on servers whose own root opted in with accept_hub_bans = true: a Ban (a copy of another server's SSH ban, TTL at most 7 days) and a Blocklist (the Feodo + Spamhaus DROP ranges). The new worst case is that a compromised hub blocks up to 50 000 non-safelisted ranges, both ways, on opted-in servers — an apt mirror or an API, for example — and copies bans for up to 7 days. It still cannot run code, touch rules outside inet netwatch, or ban a safelisted address: the safelist's own entries, the built-in loopback/private/CGNAT/ULA ranges and the hub's addresses are refused, as is any prefix the agent does not accept. Recovery: set accept_hub_bans = false and restart the agent — it flushes the four listed_* sets and removes every ban it copied, while the server's own bans stay — or, on a host you cannot restart now, sudo nft delete table inet netwatch — a stopgap only: it also drops this server's own bans, and while accept_hub_bans stays true the hub can send everything again, so follow it with the opt-out and a restart as soon as you can.

Since Phase 10 (2026-11-01) it can also ask for a unit restart, and only for a unit that server's own agent.toml lists under [restart] units: a compromised hub can restart each allowlisted service at most once a minute. That's a short outage of a service you chose to make restartable. It still can't run code, start/stop anything, or touch a unit outside the list. A compromised agent can restart any .service unit through the polkit rule, which is less than its existing CAP_NET_ADMIN can already do.

What the hub sent is recorded where the hub cannot reach it: the agent's journal shows every fleet ban it applied or refused and every blocklist version it loaded. A compromised hub controls its own audit log, so the agent's journal is the record to trust.

The database backup. The downloaded file is the whole hub: the admin password hash, the two-factor secret and the agent tokens' hashes. Downloading it needs a full session, a fresh two-factor code (a code already used to sign in is refused), its own per-address limit (5 tries per 15 minutes, successful or not), and one download at a time; every attempt is audited, success or not. The copy is made by VACUUM INTO on a read-only connection into a 0600 temp file next to the database, and that file is deleted as soon as the response ends — including when the browser hangs up mid-download. The answer is Cache-Control: no-store. Anyone who has the file holds your two-factor secret outright and can attack the password hash offline: store it encrypted, and if a copy leaks, change the password and set up two-factor again.

A login while the agent is down. Logins during an agent outage are caught up from the journal's last 24 h; an outage longer than that leaves a gap. The agent keeps no login history of its own.

A local user on an agent host. They can append to /var/log/auth.log with logger -t sshd and forge failed logins, which is why netwatch reads journald first. The file is only used when journalctl is absent or auth_log names one, and a file source trusts every local user. The worst a forger achieves is getting an address of their choosing banned — a denial of service against that address — because the safelist, the built-in ranges and the hub's own addresses are never banned.

The root key-watch helper. netwatch-keys is the only netwatch unit that runs as root, and it exists because the agent may not read a home directory. It runs with PrivateNetwork=yes, ProtectSystem=strict, ProtectHome=read-only, NoNewPrivileges=yes, PrivateTmp=yes, a bounding set of CAP_DAC_READ_SEARCH alone, and a 50-second timeout. It reads /etc/passwd (or getent passwd) and hashes regular authorized_keys files: it publishes sha256 digests and never key contents, never follows a symlink — so a user cannot point it at /etc/shadow — and it refuses to write at all when the account list comes back empty, so a broken helper reads as "no digests", not as "every key removed". The worst a compromised account achieves is changing its own key file, which is an alert, not a privilege.

Privacy: what is stored, and for how long

Everything stays in the hub's SQLite file on your own host. netwatch has no telemetry, no account, no call home; the only outbound requests are three daily blocklist downloads (plain GETs of fixed URLs, with no parameter carrying anything about your servers), an optional ntfy push and, if you run geoipupdate, a weekly MaxMind download. The hub only ever downloads the lists: it never sends an address of yours to a provider, and the agents never talk to one at all.

Data	Kept	Notes
Metrics	24 h full resolution, then 5-minute averages for 30 days	CPU, load, RAM, swap, disk usage, the net and disk byte rates, uptime.
Flows	7 days	Per-minute byte totals per remote address — never payloads.
Alerts	7 days	They can name a remote address.
SSH failure counts	7 days	Per minute, per server; deleted with the alerts.
Audit log	90 days	Who signed in, from where, what they changed. Read-only in the dashboard.
SSH logins	30 days	The user, source address, method and key fingerprint of successful logins.
Trusted SSH keys	Until revoked	Fingerprint, name, user and scope. Revoking an agent deletes its logins and its own keys.
Accounts, key-file digests	Until revoked	Login account names, uids and whether they are admin, plus the path and sha256 of each authorized_keys file. No home directory contents and never the key text: the digests are hashed on the server by the root helper.
Acks	With the alert (7 days)	The time it was acknowledged, and admin as the actor. An ack stays on the alert row, so it goes when the row does.
Mutes	Until they end	The server, the optional alert kind, and the window. A mute is deleted the moment it ends — by unmute, or by the retention job — and revoking an agent deletes its mutes with the rest of its rows.
Blocklist entries	Replaced daily	The ranges the three lists publish, and the date each list last refreshed. The hub's copy of a public list, not data about you.
Blocklist packet counts	7 days	How many packets a server dropped for each list, and the inbound total, per server. The addresses are the lists' own.
Blocklist exemptions	Until removed	The range you exempted, when, and the audit row that says who.
Fleet bans	Until they expire	The address, prefix, origin server and expiry of a copy, plus which servers refused it and why. Deleted when the ban ends, or with the server's row.
Pending updates	In memory only	The counts, up to 50 security package names, whether a reboot is due and when apt was read. Never written to the database.
Services (units and containers)	While the server is enrolled	The unit and container names, whether each is down, the reason it went down and when. One row per thing that went down, not a sample series. Revoking the server deletes them.
OOM kills	7 days	The process name, pid, cgroup and time of each kill the agent's journal read found.
Boots	One row per server	The last boot id and time, and whether the shutdown before it was clean. Replaced on each boot.
Scheduled jobs	Until you Forget the job, or revoke the server	The heartbeat file's name, its last run time and the interval you set. A FILE REMOVED row stays until you press Forget; revoking the server deletes the rest.

The agent reads packet headers only — snaplen 256, BPF filter ip or ip6, promiscuous mode off — and never keeps, logs or sends payload bytes. A remote address therefore lives at most a week in the hub's database.

GDPR. A remote IP address is personal data when it belongs to a natural person, so the retention above is the whole point: you are the controller, the data never leaves your host, there is no processor and no transfer, and it is deleted automatically. If you need it gone sooner, agent revoke <name> drops that server's stored state.

Restore a backup

The Settings page downloads netwatch-YYYYMMDD-HHMMSS.db: one consistent copy of the hub's database. To go back to it, on the hub host:

sudo systemctl stop netwatch-hub
sudo install -o netwatch-hub -g netwatch-hub -m 600 netwatch-YYYYMMDD-HHMMSS.db /var/lib/netwatch/netwatch.db
sudo rm -f /var/lib/netwatch/netwatch.db-wal /var/lib/netwatch/netwatch.db-shm
sudo systemctl start netwatch-hub

The two rm lines matter: a -wal file left from the database you replaced would be replayed over the restore.

A backup from an older hub is migrated forward on start. A backup from a newer hub is refused (schema version … is newer) — upgrade the hub first, then restore. Agents keep their tokens, because only their hashes are in the file; a server whose token was added after the backup was taken will need agent add and a fresh config.

Removing netwatch

There is no uninstaller: tools/install.sh writes plain files and units, and this is the list of them. Run the block for the role this host runs. On a host that runs both, do the agent block first, then the hub's, and notice that /etc/netwatch and /var/lib/netwatch are shared.

The agent role

# 1. Stop and disable every unit and timer the agent role installed.
sudo systemctl disable --now netwatch-agent netwatch-keys.timer netwatch-updates.timer \
    netwatch-containers
sudo systemctl stop netwatch-keys.service netwatch-updates.service

# 2. Delete the unit files, then let systemd forget them.
sudo rm -f /etc/systemd/system/netwatch-agent.service \
           /etc/systemd/system/netwatch-keys.service \
           /etc/systemd/system/netwatch-keys.timer \
           /etc/systemd/system/netwatch-updates.service \
           /etc/systemd/system/netwatch-updates.timer \
           /etc/systemd/system/netwatch-containers.service
sudo systemctl daemon-reload
sudo systemctl reset-failed

# 3. The binary, the three root helpers, and the /run directories systemd made
#    for them (RuntimeDirectoryPreserve=yes, so stopping a helper leaves its
#    directory behind).
sudo rm -f /usr/local/bin/netwatch-agent
sudo rm -f /usr/local/lib/netwatch/netwatch-keys \
           /usr/local/lib/netwatch/netwatch-updates \
           /usr/local/lib/netwatch/netwatch-containers
sudo rmdir /usr/local/lib/netwatch 2>/dev/null || true
sudo rm -rf /run/netwatch-keys /run/netwatch-updates /run/netwatch-containers

# 4. The config and the heartbeat directory jobs write to.
sudo rm -f /etc/netwatch/agent.toml
sudo rmdir /etc/netwatch 2>/dev/null || true      # keep it if the hub runs here too
sudo rm -rf /var/lib/netwatch-heartbeat

# 5. The polkit rule behind the Restart button.
sudo rm -f /etc/polkit-1/rules.d/50-netwatch-restart.rules

# 6. The system user and its group.
sudo userdel netwatch
sudo groupdel netwatch 2>/dev/null || true

# 7. netwatch's own nftables table. This removes the bans netwatch added and
#    touches nothing else on the host.
sudo nft delete table inet netwatch 2>/dev/null || true

The installer also runs apt-get install libpcap0.8 nftables when apt is there. Those are ordinary packages: remove them only if you do not use them yourself (sudo apt remove --purge libpcap0.8 nftables), and remember that dropping nftables takes any firewall rules you keep in it with it.

The hub role

sudo systemctl disable --now netwatch-hub
sudo rm -f /etc/systemd/system/netwatch-hub.service
sudo systemctl daemon-reload
sudo systemctl reset-failed

sudo rm -f /usr/local/bin/netwatch-hub

# The database is /var/lib/netwatch/netwatch.db with its -wal and -shm files: it
# is everything the hub ever stored, so copy it first if you may want the history.
sudo rm -rf /var/lib/netwatch

sudo rm -f /etc/netwatch/hub.toml
sudo rmdir /etc/netwatch 2>/dev/null || true

sudo userdel netwatch-hub
sudo groupdel netwatch-hub 2>/dev/null || true

Then delete netwatch's site block from /etc/caddy/Caddyfile (the one whose reverse_proxy points at 127.0.0.1:7000) and run sudo systemctl reload caddy.

Nothing else is left behind: every path above is one the installer wrote. Your own files — a backup script's touch line, a cron entry, the token in a note — are yours to remove.

Footprint

Measured on the notes VPS on 2026-10-25, with packet capture and SSH auto-ban both on:

	Measured	Budget
Resident memory	10.6 MB	under 30 MB
CPU	1.46 %	under 2 %

The hub does the storing, the downsampling and the push, so it is the heavier process — but there is one of it, not one per server.

Uptime check

GET /healthz answers ok 1.6.0 with no authentication and no data, for an external monitor:

curl -fsS https://netwatch.example.com/healthz     # -> ok 1.6.0

The hub cannot alert about itself, so point your existing uptime checker at that path. Besides the login page and its assets, it is the only route that needs no session: pages redirect to the login, /api answers 401, and /ws/agent wants the agent's token.

The 30-day strip on the server page and the fleet cards counts a 5-minute bucket with at least one sample as up, so a hub outage counts as down for every server, even though the agents kept reporting. That is the honest reading of a gap in the hub's own data, and it is another reason to point an external check at /healthz.

Licences

netwatch itself is MIT (see LICENSE).

Component	Licence
MaxMind GeoLite2-City (optional, you download it)	GeoLite2 EULA: no redistribution, attribution required. Not committed and never shipped in a tarball; the Network page credits MaxMind next to the map.
World dot map (crates/hub/static/world-dots.svg)	Generated from Natural Earth 110m land, public domain (see crates/hub/static/NOTICE-naturalearth.txt).
Fira Sans, Fira Code (crates/hub/static/fonts/)	SIL Open Font License 1.1 (OFL.txt beside them), self-hosted woff2; no request to Google Fonts at runtime.
uPlot	MIT, vendored into crates/hub/static/vendor/; no CDN.
Feodo Tracker (abuse.ch)	Free of charge. abuse.ch's terms allow both commercial and non-commercial use without limitation, and restrict platform access to not-for-profit use subject to fair use; the endpoint may require an access key in the future, and netwatch sends none — a list that stops downloading keeps its last copy and shows the error. The recommended subset is used, not the full list.
Spamhaus DROP (drop_v4.json, drop_v6.json)	"The DROP Lists are provided to you free of charge subject to these Terms", which cover exactly this use ("designed for use by firewalls and routing equipment to filter out the malicious traffic from these net-blocks"). The Spamhaus name may not be used in marketing. netwatch is not affiliated with or endorsed by The Spamhaus Project; the dashboard's label is an attribution.
Tor bulk exit list	Published by the Tor Project, a US 501(c)(3) non-profit, as a public service; the endpoint states no separate licence or usage terms. Tor exits are tagged and never blocked.

Development

cp crates/hub/hub.example.toml hub.toml    # gitignored; set public_origin
cargo run -p hub   -- hub.toml
cargo run -p agent -- agent.toml           # needs a token from `agent add`

For local work public_origin can be http://127.0.0.1:7000; the login cookie is Secure, so anything real goes through Caddy as in the quick start.

A build needs libpcap-dev (the headers and the libpcap.so link name); the capture test needs CAP_NET_RAW:

sudo -E cargo test -p agent -- --ignored live_capture_smoke

The gate, run before every commit:

cargo fmt --all --check
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo test --workspace --locked

crates/shared is the wire protocol (AgentMsg, HubMsg, PROTO_VERSION); crates/hub is the axum server, the SQLite schema and migrations, and the static dashboard embedded in the binary; crates/agent is the sampler, the packet capture and the ban logic. tools/install.sh is the installer and tools/build-release.sh the release build; .forgejo/workflows/ runs both the gate and, on a v* tag, the release that produces four tarballs (x86_64 and aarch64, agent and hub) plus SHA256SUMS.