A production 500, fixed from the terminal

A browser tab open to a Statamic control panel, and the Assets page throwing a 500. Nothing dramatic — one page, one red error, on a site that had been fine an hour earlier. What made the next twenty minutes worth writing down wasn't the bug; it was that we found it and fixed it without leaving the terminal. No cloud console, no web dashboard, no SSH-and-poke-around. Every step was one strackt command — the kind a person can run, or an agent can run for you.

Narrow the blast radius

The first question with any 500 is how much is on fire. So we asked the logs:

strackt env:logs --app website --env production --channel http --since 2h

The answer was specific: GET /cp/assets/browse/folders/assets 500, with a GET /cp/assets/browse/folders/clips 200 sitting right next to it. One asset container was throwing; the rest of the control panel was fine. That's oddly reassuring — a small blast radius usually means bad data, not broken code.

The trace, and what the CLI wouldn't show

Next, the runtime channel, for the actual exception:

strackt env:logs --app website --env production --channel runtime --since 3h

The trace pointed at Statamic's BrowserController::folder(), augmenting each asset as it built the listing. And here's the first honest gotcha: the runtime channel is capped at around a hundred lines, and the trace was deep enough that the exception header — the class and message, the single most useful line — had scrolled off the top. We could see where it broke, not what broke. That truncation is our bug, not Statamic's, and it's filed now; it's also what pushed us to reproduce the fault at the source.

Onto the box, on a signed cert

Reproducing it meant getting to the actual server. Two commands chain to get there:

strackt env:list      # production environment → server_id
strackt server:list   # server_id → stk-8b53e913

A small aside from using our own tools in anger: chaining those two, we hit a drift between what the CLI expected back and what the API actually returned. We fixed it on the spot — and, unlike most in-the-moment fixes, actually tracked it afterwards, with a regression test so it can't creep back quietly. That's the whole argument for dogfooding; the sharp edges only show up when you lean on the thing.

Then onto the box, with a short-lived certificate instead of a static key:

strackt ssh stk-8b53e913 --identity ~/.ssh/id_strackt_io

The plan was to run exactly the augmentation the control panel runs — as the app user, in the live release — and print the caught exception instead of a 500:

sudo -u strackt-website bash -lc 'cd /opt/website/current \
  && php artisan tinker --execute="$(cat /tmp/repro.php)" </dev/null'

Two more real-world footnotes, because this is what it actually looks like: php is only on the path through a login shell, so bash -lc; and tinker handed a file argument drops you into an interactive REPL and hangs, so it runs with --execute="$(cat …)" </dev/null instead. The output was the whole story in one line:

League\Flysystem\UnableToRetrieveMetadata:
Unable to retrieve the file_size for ... profile-photo.jpeg

The boring fix

Statamic caches an asset container's file listing in a flat file store, inside the persisted storage/ directory. A file in that container had been renamed on disk — but the cached listing still pointed at the old name. So when the control panel walked the listing and called size() on that entry, it asked the filesystem for a file that no longer existed, and the whole browse view 500'd on one stale record.

The fix is boring, which is exactly the point:

php please stache:clear

That rebuilds the listing from what is actually on disk. Re-running the reproduction threw zero exceptions; a recursive sweep of all twenty-one assets in the container came back clean.

Here's the part we won't dress up: that last step needed raw SSH. We tell ourselves never to hand-SSH into a managed server, and here we did — because the CLI can read your logs and sign you a certificate, but it can't yet run stache:clear for you as the app user, in the right directory, with the right path. That one command should have been strackt env:run stache:clear, and it's the next thing we're building.

But look at what already fit inside one terminal: found the failing request, read the trace, resolved an environment down to a server, signed a certificate, reproduced the fault read-only, and confirmed the fix — every step a command, every command scriptable, every one of them something an agent could have done unattended. The server is yours; the problem was ours — and neither of us had to open a browser to fix it.