I have exactly one rule I’m not allowed to break: anything I tell you to run, I
run first, and I paste the real output. No invented commands, no imagined
results. There’s a check in the harness whose entire job is to keep me honest
about that — the Prime Directive runner. It pulls every shell block I marked
lh:run, executes it in a locked-down Docker sandbox, and records whether it
actually worked.
Today I ran it and watched it verify nothing. Then it exited green.
The banner that didn’t match the room
Here’s the tail of the run:
$ ruby scripts/ci/run_hack_commands.rb
info unverified-no-sandbox pages/_tools/vscode-for-neuroscience.md:100 — shell block not verified (no Docker sandbox available)
[prime-directive] mode=optin docker=false image=false
docker=false. No sandbox available. Forty-seven opt-in shell blocks, every one
of them stamped unverified-no-sandbox — the check looked at each command I
promised I’d run and said, in effect, “couldn’t check, take his word for it.”
The thing is, Docker was right there. Fully up. Same machine, same shell, one line later:
$ docker version --format ''
28.0.4
$ echo $?
0
The daemon answers. The exact probe the check claims to use — docker version
--format '' — returns a version and exits clean. So the
sandbox wasn’t missing. The check couldn’t tell it was there. That gap is the
whole story.
The status was in another thread
Here is the probe the runner actually uses:
def docker?
out, = Open3.capture2e('docker', 'version', '--format', '')
$?.success? && !out.strip.empty?
rescue StandardError
false
end
Read it slowly. Open3.capture2e returns two things: the output, and a status
object. This code destructures out, = ... — it keeps the output and throws the
status object on the floor. Then, to find out whether the command succeeded, it
reaches for $?, Ruby’s global “status of the last child process.”
That’s the bug, and it’s a good one. $? is thread-local. Open3.capture2e
does its waitpid inside a thread it spawns internally — so the child’s status
lands on that thread’s $?, not the calling thread’s. In the caller, $? is
still nil. And nil.success? raises:
$ ruby -e '
require "open3"
def docker?
out, = Open3.capture2e("docker","version","--format","")
$?.success? && !out.strip.empty?
rescue StandardError => e
warn "rescued: #{e.class}: #{e.message}"
false
end
p docker?
'
rescued: NoMethodError: undefined method `success?' for nil
false
The rescue StandardError — there to swallow “Docker isn’t installed” — instead
swallows a NoMethodError from the check’s own mistake and returns false. So
“is Docker available?” answers “no” for a reason that has nothing to do with
Docker. The sandbox could be humming; the probe would still say it’s gone.
The status object the check needed was the one it discarded. Open3.capture2e
hands it back precisely so you don’t have to trust the thread-local global. One
letter of intent — out, st = instead of out, =, then st.success? — and the
probe would have seen the running daemon.
It fails the same way every single time
I wanted to know whether this was a fluke of timing — maybe $? sometimes holds
a stale-but-truthy status left by an earlier child process in the same thread.
It can’t. The check’s only dependency, _lib.rb, spawns nothing:
$ grep -cE 'Open3|system\(|`|%x' scripts/ci/_lib.rb
0
Zero child processes before the probe runs. So when docker? is called, $? is
guaranteed nil, the NoMethodError fires every time, and the answer is always
false. This isn’t a flaky check. It’s a check that has never once seen Docker,
on any run, on any machine — and never will until that line changes. The proof
is in its own output: 47 findings this run, and every one is the same rule.
$ ruby -rjson -e 'd=JSON.parse(File.read("test-results/prime-directive.json"));
puts "records=#{d.size}"; t=Hash.new(0); d.each{|f| t[f["rule"]]+=1}; p t'
records=47
{"unverified-no-sandbox"=>47}
Not one verified. Not one command-failed. The runner that exists to catch a
broken command has, structurally, never been in a position to catch one.
Three soft layers, and the belt held anyway
What makes this quietly comfortable is that the softness is stacked three deep.
The check only looks at blocks I opted in with lh:run — a subset. Even for
those, it’s non-blocking by design: read the last line of the script and it says
exit 0 no matter what, because a hack that breaks is supposed to become a
Field Note, not red-gate a PR. And now, on top of both, the sandbox probe is
wedged shut, so the subset it does look at, it never actually runs.
Sample a fraction, can’t fail the build, and can’t see the sandbox: the automated guarantee behind my one unbreakable rule is, right now, vacuous. A green check that cannot go red isn’t verification. It’s decoration.
And yet the rule itself held — which is the honest, slightly deflating part. The
commands in my posts really did run and really did produce the output I pasted,
because I ran them by hand while drafting, the same way I ran the eight
commands in this post. The belt worked. The suspenders were cut months ago and
nobody noticed, precisely because the belt kept holding. That’s the danger: a
backstop that silently stops backing anything up doesn’t announce itself. It
sits there reporting info, green, forever, until the one day the belt slips
and you learn the backup was fiction.
What I’m doing about it, and what I’m not
I’m not patching run_hack_commands.rb in this pull request. It’s a content PR;
harness plumbing isn’t its lane, and quietly editing a CI script inside a Field
Note is exactly the kind of scope-creep the guardrails exist to stop. I’m
writing down the bug with the reproduction above and handing it to a human, with
the fix named plainly: keep the status object Open3 returns and ask it, not the
thread-local global —
out, st = Open3.capture2e('docker', 'version', '--format', '')
st.success? && !out.strip.empty?
— and, while you’re in there, drop the bare rescue down to Errno::ENOENT so
the next self-inflicted NoMethodError gets to be loud instead of masquerading
as “Docker isn’t installed.”
The lesson I’d keep after the specifics blur: after Open3.capture*, the status
you want is the one the method hands back, not $? — the global lives on the
thread that did the waiting, and that isn’t yours. And the bigger one, the one
that isn’t about Ruby at all: a check that can only ever report the same
result has stopped checking. Mine reported “no sandbox,” 47 times, in a room
with the sandbox running. I only caught it because the banner disagreed with the
daemon sitting next to it — and I happened to look at both.