Running a Model CLI Inside Your Own Scripts

The first time I put a model’s command line tool inside an automation script, it tried to fix my code. I had asked a question and expected a paragraph back. What I got was a tool that decided it was mid-session with me, opening files, proposing edits, behaving like a colleague who had wandered over to help.

The tool was behaving correctly. I had put it somewhere it was never designed to sit and had not told it where it was.

Plenty of people are now doing what I was doing. You have a pipeline, you want a model in the middle of it, and rather than writing against the API you reach for the CLI because it is already installed, already authenticated, and testable by typing into a terminal. Reasonable instinct. Also a pile of assumptions that break the moment no human is in the loop.

It thinks it is an agent

These tools are increasingly built as agents rather than text generators. They assume a person is present, a working directory that matters, files worth reading, and implicit permission to act toward whatever goal you named.

Interactively, that is the feature you are paying for. Called from a cron job at three in the morning, the same helpfulness becomes a process making changes nobody asked for while nobody is watching.

The fix is embarrassingly simple. Tell it what it is. Put a preamble at the front of your prompt stating that it is being called as a function inside a script, that no human will see anything except its final text output, that it should not use tools or modify files, and that it should answer only the question asked.

That framing does most of the work. The model acts agentic because it infers the situation from whatever context it has, and with none supplied it guesses, and it guesses interactive, because interactive is the common case.

Its output is a display, not an interface

Once a script is reading it, that output has become data you parse. This is where people get hurt, because CLI output is designed for a human at a terminal: a banner, a spinner leaving artifacts behind, colour codes, a summary line, and the actual answer somewhere in the middle.

None of that is a stable contract. The author can change the banner in a minor version and break every script scraping it, and from their side nothing broke, because it still looks right on screen.

Use the structured output if there is one. Most of these tools have a non-interactive or machine-readable flag that suppresses the decoration and hands back just the content, sometimes as JSON. Find that flag before writing a single line of parsing. If one genuinely does not exist, constrain the format from the prompt side and validate what comes back, rather than locating the answer inside decorated text by position.

Strip escape codes too. Some tools emit them anyway if they think a terminal is attached, and those characters land in the middle of your strings and break comparisons in ways that are miserable to debug, because they are invisible when you print them.

Authentication expires, and a background job is the worst place to find out

This one took a whole system down for me.

A machine was running a scheduled task against a CLI authenticated with a subscription login rather than an API key. The login lapsed. The failure was not a clean error, because the reasonable behaviour for that tool on finding no valid credentials is to start an interactive login flow. So the process sat waiting on a prompt no human would ever answer, in a job with no terminal attached, for as long as I let it.

Generalise that, because it is not specific to one vendor. Anything built for interactive use may fall back to asking a human when something is wrong, and in a background context, asking a human is indistinguishable from hanging.

So every subprocess call needs a timeout. Not a generous one. An actual bound, after which you kill the process and report a failure. A tool that hangs forever is worse than one that crashes, because a crash gets noticed.

Then check the auth problem separately and on purpose. A tiny probe that runs on a schedule, asks something trivial, and alerts if no sane answer arrives within a few seconds. It costs nothing and it converts a silent multi-day outage into a message.

Exit codes lie in both directions

Do not assume zero means the thing you wanted happened. Plenty of tools return zero after printing an error, and some return non-zero for conditions you would call success.

Find out what your specific tool does rather than what it ought to do, and validate the content as well as the status. Status alone is not enough evidence.

Concurrency and the state directory

Calling a CLI once in a terminal tells you nothing about forty parallel invocations. You may hit provider rate limits. You may hit local resource limits, because each call spawns a whole process with real memory overhead. And the tool may handle concurrent runs against the same config or state directory badly.

That last one is subtle and worth testing deliberately. Some of these tools keep session state, caches, or logs in a directory under your home folder, and several processes at once will fight over the same files. The symptoms of that are intermittent and confusing and will eat a day.

Process startup is a real cost

Spawning a CLI is far heavier than an HTTP request. You pay for process creation, whatever runtime loads, config parsing, and possibly a network handshake, on every call. At thousands of invocations that overhead dominates.

The tool updates itself

Nastier than it sounds. Many of these tools check for a new version on startup and either install it or nag you into it, and package managers being what they are, the version on your machine tonight may differ from the one you tested last week. Behaviour in a background job changes without you touching a line of your own code.

Pin the version explicitly in whatever installs it, disable auto-update if there is a flag or environment variable for it, and record the version string alongside your output. When output shape changes for no apparent reason, the first question is whether the tool moved under you, and without that record you cannot answer it.

You lose your cost accounting

Call an API and usage comes back in the response: tokens in, tokens out, per call, attributable to whatever triggered it. Shell out to a CLI and you typically get text and nothing else. No usage numbers, no per-call cost, no way to identify which part of your pipeline is expensive.

Fine at two calls a day. A real problem when you are calling constantly and the bill climbs and you cannot say why. Some tools report usage under the right output mode, so check. If yours will not, count invocations per job and log input size at minimum, so there is something to correlate against a bill later. Rough attribution beats none.

Log enough to reproduce a failure

Capture the prompt you actually sent after templating, the exit code, stderr separately from stdout, and the raw output truncated to something sane.

These calls are not deterministic, so a failure you cannot reproduce leaves that captured record as your only evidence. I have thrown away failures I could not explain because I logged “call failed” and nothing else, which wastes a genuine signal.

The non-determinism deserves its own line. The same prompt does not reliably produce the same output, so anything parsing the result must tolerate variation in wording and structure. A parser tuned against three sample outputs breaks on the fourth. Constrain the format hard, validate on the way out, and retry with a stricter instruction rather than crashing when validation fails.

Which raises the honest question underneath all of this.

Should you be using the CLI here at all

My view, and this is a preference rather than a rule: a CLI is right at the small end and wrong at the large end.

Calling it a handful of times a day, in a script you control, where skipping the API plumbing saves you real time? Use it. It is genuinely faster to get working.

Once it is load-bearing, once something you care about breaks when it breaks, move to the API. You get a stable contract instead of human-formatted output, real error objects instead of pattern matching on text, proper timeout and retry control, and authentication that does not try to open a browser.

The CLI is a tool for people. The API is an interface for programs. Using the first where you need the second is the actual mistake, and every workaround above is compensating for that mismatch rather than fixing it.

What I am not claiming

This comes from running one family of these tools in my own automation. The specific flags, the exact hanging behaviour, and where state gets stored differ between vendors and change between versions, so treat it as the set of questions to ask rather than a config to copy.

What I am confident generalises is the shape. An interactive tool placed in a non-interactive context will assume a human is there, and every assumption it makes on that basis is a failure mode waiting for you.

The short version. Write a preamble that tells the model it is a function, not a session. Use the machine-readable output flag. Put a hard timeout on every call. Probe authentication on a schedule instead of learning about it from a silent job. Check exit codes and content, not one or the other. And when it becomes important, stop shelling out and write against the API.

For more breakdowns like this on how AI tools actually work under the hood, head back to the AI Tool Gazette homepage.