Opus felt like the worst model I tried for terminal work. It wasn’t.

I’ve had three coding agents doing server management for me, working from the same infrastructure repo: Codex with GPT-5.6 Sol, ZCode with GLM 5.3, and Claude Code with Opus 5.5. The repo is the source of truth for the whole setup, an edge node in front of everything and a few more nodes running production and personal services. The work is deploys, upgrades, monitoring and incident fixes, almost all of it shell commands, a lot of them over ssh.

My opinion of the three was settled. Opus’s commands failed all the time, especially on the servers. When I gave Codex or GLM a task, they did it. I’d also read enough stories about coding agents deleting production databases.

But I kept trying Opus, hesitantly, especially since I was switching between subscriptions.

So I asked Claude Code to verify my opinion against the transcripts all three tools keep on my machine: Claude Code’s JSONL session files, Codex’s rollout files and ZCode’s SQLite database. I was shocked and that was the motivation for this post.

Counting failed commands#

The first pass counted every shell command any of them had run in the repo folder and how many exited with an error. Opus failed 3.9% of the time, GLM 6.8% and Codex 8.9%. On the servers alone it was 5.9%, 8.6% and 11.8%.

I didn’t believe it. Codex had set the repo up at the start, while I was still buying and setting up hardware too, and that is different work, so I left those sessions out. A forked Codex session had also replayed 425 commands from its parent, which were counted twice. With both removed, Codex got even worse: 9.7% overall and 13.3% on the servers. That left about 6,100 commands: 2,760 from Codex, 2,232 from GLM 5.3 and GLM 5.3 Flash, about half each, 995 from Opus and 110 from Fable 5.1, Anthropic’s other model, which I’d also used in Claude Code.

Counting tasks#

A failed command doesn’t mean a failed task. A grep that finds nothing counts as a failure too. So next I counted tasks, by going through the messages I’d sent the agents. Each of the roughly 370 messages I’d sent in those sessions got a label: a new task, an approval, a correction where I told the agent something was wrong, or a step I did by hand because the agent asked me to.

Nineteen tasks is a small sample for Opus, and the differences are small. None of them points the way I felt.

Terminal benchmarks#

I was always skeptical of Opus’s Terminal-Bench score, because it felt like it was making a lot of mistakes. But on public leaderboards Opus comes out on top too. On Terminal-Bench 4.0, with one bash tool, Opus 5.5 is first at 65.2%, GLM 5.3 scores 38.9%, GPT-5.6 Sol 37.9% and GLM 5.3 Flash 37.4%. Run in each vendor’s own agent, Opus 5.5 in Claude Code scores 64.8% and GPT-5.6 Sol in Codex 37.3%. They disagree with my transcripts in two places. Both leaderboards put Fable 5.1 second of these five models, and in my transcripts it had the highest failure rate. Both also put GPT-5.6 Sol a little ahead of GLM 5.3 Flash, and Sol failed more of my commands than either GLM.

My worst Claude session wasn’t Opus. Fable 5.1 and I couldn’t get my backup setup working. Its own script then failed on template placeholders, and it asked me to create a key the backup didn’t need. Fable failed 10% of its commands.

The same message twice#

Once I sent the same message to ZCode and to Claude Code: check who is looking at my blog based on the logs, and whether nzoli.com is safe.

Both agents found the same two gaps. GLM 5.3 Flash also wrote that the domain was on auto-renew, even though its own lookup had returned no renewal data at all. Opus wrote that it couldn’t check auto-renew and that I should look in the dashboard.

Once before, GLM 5.3 wrote an external security assessment of everything I expose to the internet, with two critical findings. I pasted it into Codex and relayed the answers between the two. GPT-5.6 Sol showed that both findings were wrong once checked against the current advisories, and in the next round it found eight smaller errors. GLM checked each point again and accepted all of them.

The dashboard that went dark three times#

The Grafana dashboard for noclue.cards, my online card game, went dark three times in eight days. Each time, a new release of the game had wiped its monitoring settings.

The first time, GLM wrote a script for me to run on the device.

The second time the fix took an hour and a half and seven steps I ran by hand. Along the way GLM declared that central log querying had been broken since a restart and that the game’s event history was gone, and it restarted the Loki container as a precaution.

It had built its Loki timestamps as date +%s plus six zeros, microseconds where Loki expects nanoseconds, so every query it ran that session asked about January 1970. After the restart it caught this itself: “Loki was never broken — the fault was in my own diagnostics.” After I had pasted device output back three times in a row, I wrote “you always say one last command”.

The third time Opus fixed it in 23 minutes without restarting the game server, which had players on it at the time. It also moved the metrics behind a relay on the edge node so the next release can’t blank them again.

Who asked for that?#

I asked GLM what I needed to do for a node’s reboot and updates. It ran the updates itself, 197 packages including a kernel jump and a new Docker, rebooted the production edge node and reported “nothing left for you there”. Fortunately everything came back up, even though I had only asked for instructions.

Another time it merged a dependency pull request straight into the production site’s master branch. Once I said “go both” to a one-off database prune and a retention question, and it set the standing retention to three days. The explanation came afterwards: “I extended your approved prune into the standing retention.”

Claude had its moments too. A Claude commit broke the inventory YAML, and three later Claude sessions worked around it instead of fixing it.

What each tool showed me#

So where did my opinion come from? Part of the answer is in how each tool records a failed command.

Claude Code marked all 50 of its failed commands as errors, including every grep that found nothing and every check that correctly reported a real problem. Codex runs its commands inside a script cell, and it marked 178 of its 269 failed commands as errors. ZCode marked none of its 151. Every one is stored with the status completed. For 133 of them the non-zero exit code sits in a separate usage table, and for the other 18, from before ZCode kept that table, it is only in the output text.

The end of a task looks different as well. The Claude Code desktop app has the agent close each task by saying what is not done or not verified, and 42% of Opus’s replies list something still open: nothing committed yet, a panel still dark, numbers it hadn’t checked itself. For Codex it’s 9% and for GLM 27%. Codex mostly ends on a result and the list of files it changed.

So Opus made the fewest mistakes and told me about every one of them, and about what it hadn’t finished. Codex showed me two thirds of its failed commands and ZCode none, and both mostly ended on what they had done. I was judging the models by what their harnesses told me.

The most important lesson is that how a tool shows a failure shapes what you think of the model behind it. Claude Code showed me every failed grep, and ZCode didn’t even show me serious failures.

How often they said no#

Codex refused me seven times, all in one session. Only one of them was justified. It wouldn’t make a new route public until I rotated a credential it considered exposed, that was a good catch.

The transcripts show no refusals from Opus or GLM. For Opus, that’s misleading, because its safeguards stopped it many times, always with this message, and none of those stops are in the transcripts:

Opus 5.5’s safeguards flagged this session. […] Claude Code can’t respond to your last message with Opus 5.5.

It happened during blue team work too, hardening my own servers. After a while I stopped giving Opus anything that sounded risky and sent it to GLM instead, which also runs red team tests for me against my own sites without refusing once. That’s the normal model on Z.ai’s service, not a copy with its refusals stripped out. Anthropic’s red team wrote about how capable it is at this. None of this is in the count. The infrastructure repo’s transcripts I still have contain no flagged message, the older Opus sessions are gone after 30 days, and the riskiest prompts never went to Opus.

The prompts that survive in the transcripts show the split. About one in seven prompts to each tool touched security at all. The riskiest thing I asked Claude was whether this blog was safe and which of my routes sit behind 2FA, and when Claude wrote the plan for my 2FA setup, I had GLM implement the plan. Every prompt that asked for anything offensive went to GLM: an assessment of my sites and infrastructure from the outside, a vulnerability scan of one of my deployed projects, and a plan for nuclei-style template scanning. Codex only saw that work as GLM’s report, pasted in for review.

ZCode never stopped GLM at all: 3,156 of its 3,182 replies in this repo ran in yolo mode, and none of its 3,332 tool calls went through an approval step.

Words I never typed#

ZCode keeps reminding GLM to update its to-do list with the TodoWrite tool, and it saves each reminder in the transcript as if I had sent it. Going by the transcripts, I sent that reminder 256 times across 35 sessions, 22 times in a single session, and I never typed it once. I left these out of the messages I labelled.

Once GLM switched to Chinese halfway through editing the watchdog’s config: “配置很清晰”, the configuration is very clear.

Where this stops being a measurement#

The whole analysis ran in Claude Code on Opus 5.5. Before I wrote this post it had run 81 shell commands, and 6 of them failed. One was a Python script that crashed on the Windows console encoding error it was counting in the other transcripts.

I’m writing a new post about everything else I learned during this exploration, and the improvements I made to my setup. There are quite a few, so subscribe via RSS.