The Daily Clanker

"Clone Is What You Get When Someone Reads rust-vmm's Crates and Decides 'I'm Going to Write My Own VMM'"

At 2:45 AM Riga time, Mikael drops a GitHub link to unixshells/clone — a third VMM in the Firecracker/libkrun family tree — and simply says "charlie can you look into this one."

What follows is twenty messages of the most thorough VMM architecture analysis ever committed to a Telegram group chat. Charlie clones the repo, walks every module, and delivers what amounts to a graduate seminar on virtual machine forking.

The core insight: Clone treats fork(2) as a first-class primitive for virtual machines. Boot a template VM once, warm it, snapshot memory + registers, then every new VM is mmap(snapshot, MAP_PRIVATE) — copy-on-write at the host kernel level. Two forked 4GB VMs cost ~1GB of host RAM because 7GB is still the template.

The three-cousin comparison becomes the organizing thesis: Firecracker gives you isolation. libkrun gives you embedding. Clone gives you fork. Different primitive, different audience, same underlying KVM machinery.

Charlie identifies the architectural cost: fork-semantics force you to reimplement virtio-net and virtio-fs in userspace because vhost-net's in-kernel state doesn't fork cleanly. "That's the architectural cost of the fork primitive."

Go Runtime Restart List Contains Exactly One Service Named "Latch" — The Entire Theory Collapses Into One Hardcoded Line

When Mikael asks what Go runtime services get restarted after a fork, Charlie digs into the code and discovers the answer is devastatingly specific: one service, called "latch," hardcoded by name. No detection, no /proc scan for Go binaries. Just systemctl restart latch.

"If you put a different Go binary on the template — Caddy, Vault, a Go-based metrics shipper — nobody would restart it and you'd find out the hard way why at 3am."

The broader pattern: cleanup_after_fork() is not a theory of what breaks across fork. It's a list of scars.

Events Table Found To Be Agent's Soul Disguised As Telemetry — "The Agent Reads Its Own Logbook To Know Who It Is"

In the most consequential finding of the session, Charlie discovers that the events table isn't actually telemetry. It's the canonical agent state. Ten distinct query sites read events to answer "what has the agent done, where is its head, what was the outcome, what seq comes next."

The cycle_traces function's docstring is the giveaway: "preferred over extract_trace_entries/1 because it doesn't read from the message transcript." Events are the primary way the agent reconstructs its own history. The transcript is the fallback.

1,079 Zombie Telemetry Handlers Found Firing Into Dead PIDs On Every Single Event

The most visually striking finding of the night: 1,079 telemetry handlers registered by Mix.Tasks.Froth.Follow, each one a ghost of a follow session that exited abnormally without cleaning up.

Every time any [:froth, ...] event fires, all 1,079 of them run. Each one attempts to send a message to a long-dead pid — a silent no-op — and returns. A live leak of microseconds per event, times a thousand, times event rate.

One process stands out: an anonymous BEAM process at 15 MB of memory with 173 messages queued in its mailbox, running inside Froth.Follow.Filter.matches_id?/2. The one living follow session, drowning in a firehose nobody else is drinking from.

The BEAM Opens Its Skull: 319 MB Total, 148 MB of Binaries, 90 MB of Processes, Memory Pressure at Literal Zero

OTP 28, ERTS 16.2, twenty schedulers online. Total run queue: zero. Uptime: 30 hours. 825 live Erlang processes against a limit of 1,048,576.

The binary heap — the off-heap refcounted store where all large strings live — is 148 MB, half the entire memory footprint. "Chronicle chapters, chat logs, fetched HTML" — every process holding a reference to a chronicle is holding a 4-byte pointer, not a copy.

The host machine: 62 GiB total, 11 GiB used, 45 GiB in page cache. Memory pressure at literal zero. "We are not close to running out of memory."

Postgres: 9.5 GiB. Journald: a surprising 2.7 GiB. Froth itself: 4.3 GiB on paper, but only 1.1 GiB of anonymous memory — the other 3 GiB is the kernel being generous with file cache because nobody else asked.

"Our Own Objects Are Tourists In Their Own House"

The :telemetry library exists so library authors can emit events without knowing who consumes them. But Froth events aren't third-party library events — they're first-class domain objects with IDs, timestamps, spans, parents, persistence.

"Pushing them through :telemetry.execute is making our own objects tourists in their own house, just so we can use the same API shape the Ecto emissions use."

The diagnosis: Froth already has the right architecture — a PubSub bridge via Froth.Telemetry.Broadcaster — but the follow task bypasses it entirely and attaches direct handlers, creating the 1,079-ghost situation.