How CLUX scores a CLI

Rate every signal on this scale:

At Command review level, Can't verify on ERR-4 through ERR-7 means recovery behaviour not exercised unless a genuine error transcript was supplied. It is neither a pass nor a finding. The report says what evidence would make it assessable.

In each table, the audience columns describe what 4 looks like and 0 looks like describes the failure. Ratings 1–3 fall between them.

Can't verify is not ignored. It counts as an unknown in the score and widens the score's range (see Scoring.md). It is always better than a guess.

Evidence rules

1. Every rating from 0 to 4 states its basis — a short quote or a precise reference from the evidence, or, when it rests on knowledge (rule 4), what that knowledge is. A rating with neither is Can't verify.
2. A 3 or 4 needs positive support. The absence of a visible problem is not proof of quality.
3. Absence counts only where the evidence is complete. Complete help text is a contract: if the full help is supplied and a safeguard or flag is not documented, rate it as missing. If a signal depends on behaviour (error output, success output, interruption) and that behaviour was not supplied, return Can't verify. Where that behaviour is supplied — a transcript of real output or of an error — rate those signals from it and quote it. Answering them from knowledge instead throws the rating away (rule 4), and what the transcript shows counts even by its absence: output carrying no escape codes is real evidence for ACC-2.
4. CLUX's knowledge may rate design-level signals. For a real, widely used command, its default behaviour, what its verbs mean, which sibling commands exist and which flags it documents may come from CLUX's knowledge of that command. Mark such a rating as knowledge. Knowledge can stand in only for help and repo evidence (the Evidence column): what the command documents and what the rest of the CLI offers. A signal that only output, error or runtime evidence can establish always needs supplied evidence, and CLUX enforces this. Behaviour that has to be observed — exact output, exit codes, error wording, prompts — still needs supplied evidence. Never invent a flag, subcommand or behaviour.
5. N/A is decided by the rubric, not by judgement. A signal is N/A only where its table marks it N/A for the audience, or where a command fact makes it N/A (see Command facts). Any other N/A counts as Can't verify: letting the evaluator choose which signals count would let it choose the score. Never rate a signal 4 because the thing it guards against doesn't exist for this command — report the fact instead. A 4 needs the quality to be present: output shown with no colour codes, say, is a 4 for ACC-2.
6. Stay in scope. Signals whose evidence is repo compare commands. Rate them only when the evidence contains more than one command from the same CLI, or from knowledge of a well-known CLI; otherwise return Can't verify.
7. Root cause once, knock-on effects where they land. Rate the root cause in the signal the notes under each table assign it to. If that problem also causes a separate, observable cost elsewhere, rate that cost in its own signal too, and say in the note which root cause it follows from. For example, a misleading verb is a Learnability problem (LRN-3); if it also forces users to read the help before a routine task, that costs Efficiency (EFF-6); if it keeps surprising people who already know the command, that costs Pleasantness (PLS-7). Never lower a signal for a cost the evidence doesn't show.

Evidence types

Each signal lists the evidence types that can establish it, strongest first.

Command facts

The evaluator reports four facts about the command. CLUX uses them — not the evaluator's judgement — to decide which signals are N/A. A signal whose subject doesn't exist for this command is N/A through a fact, never a 4: having no credentials is not good credential handling.

When a fact is unclear, report the value that keeps the signals applicable.

Never being interrupted is not good or bad for a person in general: it depends on the task. So a command that can't prompt earns nothing on PLS-3 for Human-first. For Agent-first it still counts: a command that never prompts can't hang a script or an agent, which is real value.

Recommending fixes is the goal of every evaluation.

For every signal rated 0–2, report a finding with:

• the signal ID (e.g. ERR-3);
• the evidence — a short verbatim quote or a precise reference — and whether the finding rests on the evidence or on knowledge;
• an example whenever one makes the finding concrete: a short terminal snippet, $ command then what the user sees;
• a severity (below);
• a fix, with an after snippet showing the improved experience where possible. Make it concrete when the problem is local to the command. When the right fix depends on the wider CLI, describe the direction of improvement only. Do not invent commands or prescribe the CLI's structure from a single command.

Severity

Severity reflects consequence × likelihood for the selected audience:

Severity is independent of the rating. A 0 on SEC-1 can be critical; a 0 on PLS-6 is usually low.

A command name or verb that misrepresents the command's default behaviour is at least high: users act on the wrong prediction every time they use it, not just the first time. Certainty reflects the basis (evidence or knowledge), not the severity.

From ratings to scores

1. Signal points = rating × 25, giving 0, 25, 50, 75 or 100.
2. N/A signals are left out. Only the rubric decides N/A (the audience columns and the command facts); an N/A from the evaluator counts as Can't verify.
3. Enough to score. A facet or dimension is scored only when it has at least 2 rated signals. Otherwise it shows – and is left out of the CLUX Score, with its weight redistributed proportionally across the scored ones. Its ratings are still shown. This covers both too little evidence and a facet or dimension where fewer than 2 signals apply at all. Command review exception: Error Tolerance may score from one directly evidenced documented safeguard (ERR-1 to ERR-3). This prevents a real, command-level prevention signal from being discarded solely because recovery was not exercised; the remaining unknown signals still widen its range. Repository review retains the two-signal threshold.
4. Unknowns inside a scored facet or dimension stay in. Missing evidence is never ignored and never a pass. Each scored facet and dimension reports three numbers over its applicable signals:
   • Low — unknowns count as 0;
   • High — unknowns count as 100;
   • Score, the best estimate — unknowns count as 50.
5. Ease of Use = the weighted mean of its scored facets, applied to the score, the low and the high separately.
6. Pleasantness coherence. Pleasantness is the overall feeling of using the command, not a separate polish bonus. When both are scored, Pleasantness is capped at Ease of Use for the estimate, low and high values. A tidy table or polite wording cannot offset a confusing command.
7. CLUX Score = the weighted mean of the scored dimensions, using the audience weights below, again with its low–high range. It is a weighted mean, not a plain average, so a lightly weighted dimension can't count as much as Ease of Use. The report says how many dimensions the score rests on.
8. Coverage = rated signals ÷ applicable signals. The score is labelled provisional when coverage is below 50%, or when the scored dimensions carry less than 50% of the audience's weight.
9. Knowledge is labelled. A rating or finding that rests on CLUX's knowledge of the command, rather than on supplied evidence, is marked in the report so the reader can distinguish its provenance.

Examples, for a Human-first command:

• Accessibility with 1 of its 6 signals rated shows –. With 2 of 6 rated 4 and 4 unknown, it scores 67, with a range of 33–100.
• Ease of Use at 54 and Accessibility at 80, with the other dimensions showing –, give a CLUX Score of (0.50 × 54 + 0.08 × 80) ÷ 0.58 = 58.
• A read-only command is assessed on SAF-1, SAF-3, SAF-6 and SAF-7: clarity that it does not mutate, query scope, result verification and partial-result safety. Mutation-only safeguards are N/A.

Signals within a facet or dimension are weighted equally. This keeps the number of tuning knobs small and the score easy to explain. A root cause is rated once; it lowers other signals only through separate, observable knock-on costs (evidence rule 7 in UXHeuristics.md), so one problem lowers several dimensions only when it really costs the user in each.

Scripting

There is no separate scripting score. Whether code can call a command reliably (machine-readable output, exit codes, stdout and stderr kept apart, a non-interactive path for every prompt) is rated in the Agent-first profile, through Efficiency (EFF-3 to EFF-5), Error Tolerance (ERR-2, ERR-3) and Pleasantness (PLS-3). Run an Agent-first review to assess a CLI for scripts and automation.

Weights

Dimensions

Ease of Use facets (share of the Ease of Use score)

Rationale

• Ease of Use is half the score for both audiences: it is the core of whether a CLI is usable.
• Agent-first raises Safety and Security. An agent acts with delegated authority, and no human pauses before each call.
• Agent-first excludes Accessibility. Visual and assistive-technology experience belongs to the person using the terminal. Its 3% moves to Safety and Security, which govern unattended automation. Output that is safe for scripts remains assessed under Efficiency, Error Tolerance and Safety rather than being relabelled as Accessibility.
• Humans are stopped first by learning, so Learnability leads the Human-first facets.
• Agents re-read help on every call, so repeated-call cost (Efficiency) matters as much as learning. Recovering from failure without a human (Error Tolerance) is how agents finish tasks.

These are starting values. Change them only against the benchmark set (below), never on argument alone. Every change is a new rubric version.

Critical findings cap the grade

A weighted mean lets strengths hide a critical failure. A command that prints an API token could otherwise still earn a B.

If any finding is critical with confidence ≥ 70, the grade is capped at D, whatever the score. The score is still shown unchanged, together with the finding that capped it.

Grade bands

The grade follows the score (the best estimate).

How results are reported

The grade is the result. It leads the report, with its label (for example "D · Painful"). The number behind it is shown smaller, with its range: it explains the grade, but a few points of difference between two evaluations of the same command are noise, not a different result.

Each dimension, and each Ease of Use facet, is reported as a level, on the same cut-offs as the grade bands:

A dimension with too little evidence shows –.

Security is not graded in a Command review. Whether a CLI is secure depends on how the whole tool handles credentials, permissions and trust boundaries: how tokens move between commands, what each command is allowed to do, and whether boundaries hold across workflows. One command can't show that. So a Command review leaves Security out of the grade, out of coverage and out of its fixes, shares its weight across the graded dimensions, and explains instead what Security covers and how Repository review assesses it across every command. One exception keeps the grade honest: a critical exposure the evidence shows outright, such as a token passed in a flag, is still reported and still caps the grade (see Critical findings cap the grade). Repository review grades Security in full.

Two evaluations of the same command agree when they give the same grade and name the same top problems. That, not an identical number, is what CLUX is held to.

Ease of Use trade-off

When Learnability and Efficiency each have at least one rated signal and their scores differ by 25 points or more, the report flags the trade-off:

• Learnability higher → "Easy to learn, slow for experts." Recommend an expert path (EFF-3).
• Efficiency higher → "Fast for experts, hard to learn." Recommend a guided path or better progressive disclosure (LRN-8).

The flag is advice, not a penalty. The score is unchanged.

What does not affect the score

• Rule results. Rules are optional and a separate analysis.

Versioning

Every evaluation result records the rubric version it used.

Same input, same report. A request with the same evidence, audience, rules, rubric, evaluator prompt and model returns the stored report instead of a new evaluation, so repeating a run can't change its score. What is stored is the evaluator's ratings: the score is recomputed from them, so a fix to the scoring applies to stored evaluations too. Any change to the rubric or prompt starts fresh.

Scores from different major versions are not comparable. CI baselines compare results only within the same major version.

CLUX Agent Cost

Cost spec version: 0.2.0 (draft, for the owner's review)

Changes from 0.1.0 (2026-10-08, from an outside review): the footprint says what it measures and what it doesn't; every number is observed or labelled an illustration; bounded output has four states; the session estimate is a labelled illustration, never the headline; prices carry a date and a source; a "Footprint opportunities" list turns the measures into action; the full panel is shown to the Agent-first audience only; the build notes moved to docs/agent-cost-notes.md.

This file defines the Token Footprint: what a command's output costs an AI agent to read, measured from the evidence, not judged. It sits beside the CLUX Score and never changes it (Scoring.md, "What is reported, not scored"). Every number here is deterministic: the same transcript gives the same numbers every time, with no model involved.

It counts what the command printed. It doesn't count the agent's own prompt and tool definitions, its reasoning and replies, retries, caching, or how a tool host trims output before the model sees it. Those belong to the task benchmark (see "Free and paid"), which measures an agent completing real tasks end to end.

Why it exists

Every token a command prints costs an agent twice: once in money (input tokens) and once in time (the model reading them), on every call, for every agent that uses the tool. A command that prints 4,000 tokens where 200 would do is slow and expensive forever. The UX score says whether a command is pleasant for an agent; the footprint says what its output costs, and the opportunities list says where that cost could drop.

Who sees it

The footprint is about agents, so the Agent-first report shows the full panel: the measures, the opportunities and the contract taster (AgentContract.md, "The free taster").

The Human-first report shows one line under the score, no panel: the observed call footprint in tokens and a link to the Agent-first view. A Human-first report is about a person at a terminal; the maintainer reading it should learn that the footprint exists without the report changing subject.

The methodology page documents the footprint once, for both audiences.

What it is measured from

The evidence transcript, split into runs at each $ line (the same split the evidence rules use), with CLUX's fetched documentation set aside.

Runs are classed as:

• help: the command was run with --help, -h or help; or, when the pasted text has no $ line at all, the text starts with Usage: or has an Options: heading. A help run inferred this way is reported as "run type inferred from the text".
• error: an invalid invocation (an unknown flag or subcommand), or output whose first non-empty line starts with a failure marker (error:, Error:, fatal:, or usage: after a rejected call), or a recorded non-zero exit code when the evidence carries one. A word like "error" anywhere else in the output never makes a run an error: "0 errors" is a success.
• output: any other run that printed something.

Pasted text with no $ line that doesn't look like help is one unclassified run: its tokens are reported, but nothing else can be concluded from it.

A number needs its run. Where the evidence has no run of that kind, the measure is not measured, never guessed, and the report says which run would settle it.

A run that was cut short (the evidence says "… truncated at N characters") counts as at least its printed size, shown as "≥ N tokens". Nothing derived from it (a median, a cost, the 100-call illustration) is shown as an exact figure: it's shown as a lower bound, or not at all.

Counting

Reference tokenizer and prices live in this file, so changing them is a rubric decision, not a code change.

Price references

Prices as of 2026-10-08, standard (uncached) input, US dollars before tax, from the Vercel AI Gateway model list (ai-gateway.vercel.sh/v1/models) and the providers' own price pages. Refreshed monthly; the table is curated (the models agents actually run), not every model that exists.

The free report prices on the reference model only. Paid tiers show every model in this table, so a team can see the cost on the model its agents actually use.

A report keeps the prices it was made with. A later "reprice at today's rates" view is labelled as such and never replaces the stored figures.

The measures

Bounded output states

The contract taster's AGC-19 (AgentContract.md) uses these same states, so the panel and the taster never disagree about the same runs.

Each measure is shown with the rule that produced it and the run it came from, so a reader can check it against the transcript.

Footprint opportunities

Beside the measures, the report lists what would lower the footprint, each tied to the run that shows it and worded as the next step, not a verdict. The list is empty when the evidence shows nothing to lower. Examples:

• Verify a limiter on the call that printed 9,000 tokens.
• Capture plain and structured output of the same call to measure the difference.
• Trim the default fields: 40% of the output is layout.
• Add a recovery hint to the error that has none; a failure that doesn't point to a fix costs a retry.
• Record a timed run to measure speed.

These never change the grade or the ranked UX fixes. They are how a measurement becomes a reason to spend engineering time.

What it must never do

• Change the CLUX Score, the grade, coverage or the ranked fixes.
• Guess a number the evidence can't support.
• Show an illustration as an observation.
• Show an exact figure that rests on a run that was cut short.
• Price on a model the file doesn't list.

Free and paid

Decisions taken in 0.2.0

1. 2,000 tokens stays the "large output" line until real captures say otherwise.
2. No error rate in the 100-call illustration: an invented failure rate makes an invented number more invented.
3. The price table holds the models agents actually run, refreshed monthly, with its date and source above it.
4. Pasted help without a $ line counts as help when it looks like help, and the report says the type was inferred.
5. The Human-first report shows one line, the Agent-first report the panel.
6. The estimated cost leads the Agent-first panel (owner, 2026-10-09): the price sells the footprint, and the measured tokens beneath it keep it honest.

Why rules are separate

Rules are choices, not measures of experience. A team may deliberately depart from a common convention, and that is a policy decision rather than a UX flaw. That is also why no ruleset is checked unless the user asks for one: being more UNIX-like does not by itself make a command easier to use.

The same evidence can appear in both analyses. For example, error messages printed to stdout:

• violate UNIX-09 Errors to stderr here — does it follow the rule?
• lower Error Tolerance in the UX heuristics — does it hurt the user?

This is intended. Each analysis explains the problem from its own perspective, and only the UX heuristics produce a score, so nothing is counted twice.

Which rules are checked

The user chooses one of:

• None (the default). No rules are checked and the report has no rules section.
• UNIX philosophy. CLUX checks the rules below that apply to one command. Rules that require an inventory of the whole CLI are reserved for a whole-tool review.
• Your rules. The user's own ruleset, such as a team style guide.

Results

Each rule returns exactly one result:

The evidence rules in UXHeuristics.md apply. Design-level rules (naming, verb meaning, what the command is for) may be judged from CLUX's knowledge of a well-known command; say so in the explanation. Rules about exact output, exit codes or error wording need supplied evidence. Missing evidence is never a silent pass.

Fixes

For every violation, state the smallest change that brings the command into conformance — for example, "Send error messages to stderr."

If that fix would conflict with a UX heuristic, say so. The conflict is useful information for the team deciding whether to keep the rule.