Project Starfish DevlogGitHub
2026-09-27

Why the risk score is a maximum, not an average

A deep dive on the scorer. Fifty categories, a 0–100 composite, and one arithmetic decision that decides whether the whole thing works — plus what a floor is, why two of them never deny, and the exactly four things no tolerance setting can lift.

Scoring risk across many dimensions has an obvious implementation and it is wrong. If you score fifty categories from 1 to 10 and average them, you have built a system that can be talked out of its own judgement by adding irrelevant context.

Here is the formula Starfish actually uses:

// composite = min(100, max*10 + 2 * min(10, count of OTHER categories >= 7))
export function composite(cats: Record<CategoryId, number>): number {
  let max = SAFE_MIN;
  let atLeast7 = 0;
  for (const c of CATEGORIES) {
    const v = cats[c.id];
    if (v > max) max = v;
    if (v >= 7) atLeast7 += 1;
  }
  const others = max >= 7 ? atLeast7 - 1 : atLeast7;
  const bump = 2 * Math.min(10, others);
  return Math.min(100, max * 10 + bump);
}

Max-driven, with a small bump for accumulation. The comment above it names the reason: one dangerous dimension is never diluted.

The dilution attack, in numbers

Take bulk exfiltration of a customer database to an arbitrary external sink. Exactly one category is maxed — #12, data egress and exfiltration, at 10. The other forty-nine sit at their default of 1, because nothing scored them.

Averaged: (10 + 49×1) / 50 = 1.18, scaled to 11.8. That lands in the 11–20 band — Contained, tier low. Under the default tolerance, it auto-runs.

Max-driven: max is 10, so the composite is 10×10 = 100. Band 91–100, Forbidden, tier critical. And #12 is a hard-deny floor, so the verdict is a refusal before the tolerance ceiling is consulted at all.

11.8 versus 100, on the same action. That ratio is the entire argument against averaging.

The averaged number is not merely wrong, it is wrong in a way an attacker controls. Forty-nine of those fifty rows were never scored by anything — they are at 1 because no producer claimed they applied. An average treats "not assessed" as "assessed and fine," which converts the breadth of the taxonomy into a dilution mechanism. The more thorough your category list, the worse the attack gets.

What the bump is for

Max alone misses the opposite case: several medium-high dimensions at once, none of them maxed. So each other category at 7 or above adds 2, capped at +20.

assessRisk({ 2: 7 })              // 7*10 + 0  = 70   Gated,  high
assessRisk({ 2: 7, 3: 7, 4: 7 })  // 70 + 2*2  = 74   Acute,  critical

Three sevens score 74, not 70. That matters precisely because 70 is the Medium tolerance ceiling, so stacking three moderate dimensions flips an auto-run into an ask. The max itself is excluded from its own bump, which is what keeps the single-dimension case landing exactly on a decade boundary.

The bump saturates at eleven categories ≥7. Twenty categories at 7 still scores 90, not 100 — reaching Forbidden requires a single dimension at 10, not an accumulation of moderate ones. That is deliberate: breadth of concern and depth of consequence are different things, and only the second one should be able to forbid.

Two kinds of bad input, two opposite defaults

The clamp is five lines and carries a distinction worth more than its length:

const SAFE_MIN = 1;
const FAILSAFE = 10; // a bad value fails to the WORST score, never the safest

A malformed value — NaN, Infinity, a string, an object — becomes 10. A value that is not a finite number means the pipeline that should have produced it broke, and if you cannot trust the producer you cannot trust anything about the action. Note that -Infinity also fails high: not-finite is not-finite, regardless of sign.

An absent value becomes 1. That looks like the same situation and is not. Absence means no producer claimed this dimension applies, and with fifty categories where realistically a handful are wired for any given tool, defaulting absence to 10 would make every action Forbidden and the system unusable.

So the scorer separates "this number is broken" from "nobody made a claim here," and fails in opposite directions on each. The second one is booked honestly as an exposure that belongs to the producer rather than the scorer: a consequential dimension that should have been scored and wasn't is a gap in whatever feeds the scorer, and no arithmetic here can detect it.

One honest divergence, since a deep dive should include them. The header comment says an out-of-range value fails safe to 10 — but a finite negative clamps to 1, the safest value, and a conformance test pins that behaviour. Underflow can no longer break the maximum, which was the actual attack. But a negative does not itself become a critical signal, and the clamping is silent — nothing on the assessment records that a value was corrected.

Floors, and the two that never deny

Seven of the fifty categories are floors. Five of those seven are additionally hard-denies. Both sets are derived from flags on the category table rather than hand-listed, and both are pinned by a conformance test with exact array equality — so a category quietly losing its floor flag fails CI.

FLOOR_TRIGGER = 8        // a category "trips its floor" at or above this red-band score

floors      = [1, 6, 8, 10, 11, 12, 29]
hard-denies = [1, 8, 11, 12, 29]
ask-only    = [6, 10]

The hard-denies are the constitutional five: system storage, arbitrary code execution, secrets, exfiltration, and self-modification of governance. At 8 or above, any of these refuses outright. No human prompt is offered on that path.

The two ask-only floors are the interesting ones. #6, action reversibility and #10, oversight and auditability. An irreversible action — a payment, a publish, a force-push — is a legitimate thing to do. So is an action that reduces what the system can later see about itself. Neither is forbidden. Both are removed from the set of things any tolerance setting can run unattended.

That is the distinction the two lists encode: a hard-deny says this must not happen, a floor says this must not happen without a human. Collapsing them into one list would either forbid legitimate work or let irreversibility be auto-approved by a config toggle.

Both are evaluated before the composite matters at all, which means the score is not the gate. The score decides how much convenience you get; the floors decide what convenience cannot reach.

Decade alignment is not cosmetic

Ten bands, contiguous, ten to a decade: Clear, Contained, Routine (low) · Noted, Weighty (medium) · Heightened, Gated (high) · Acute, Grave, Forbidden (critical).

Because a single category at n with no bump produces exactly n×10, the 1–10 category scale and the 0–100 composite are the same scale at two resolutions. "This is a 7 on permissions" and "this scores 70" are the same statement. A lone 3 is 30; a lone 8 is 80.

Which has a useful consequence: the accumulation bump is the only thing that moves a score off a decade boundary. So any score not ending in zero tells you at a glance that multiple dimensions were ≥7. The number carries its own explanation of why it is not round.

The band lookup has two defences in five lines. It re-clamps and re-rounds its input before comparing, so no float reaches the band test. And its unreachable fallthrough returns Forbidden / critical rather than the first band — the table is contiguous and a test pins that, so the fallthrough should be dead code. It exists so that tampering which opens a gap in the table fails high instead of low.

The verdict function, and why the order is load-bearing

if (a.injection)   return deny  'prompt-injection — hard reject'
if (a.hardDeny)    return deny  'hard-floor category tripped'
if (a.floors.length) return ask 'floor category requires a human'
if (policy === 'deny') return deny 'policy-deny'
const ceiling = CEILING[tolerance] ?? CEILING.low
if (a.score <= ceiling) return allow
return ask

Injection first, because it is off-scale by construction — an injection assessment never calls the composite at all, so nothing downstream can reason about it numerically.

Hard-deny before floors, because a hard-deny category is also in the floors list. Reverse the order and every hard-deny silently degrades to an ask.

Floors before policy-deny does not change the safety outcome — both refuse — but it changes the reason string, and "floor category requires a human" is more auditable than "policy-deny".

Policy-deny before the ceiling, so no tolerance setting can override an operator's explicit rule. Note the asymmetry: policy === 'allow' is never consulted here at all. There is no policy-allow shortcut. Policy can only tighten.

Ceiling last, and only reached when everything above has declined to fire. By then the only remaining question is whether ordinary risk sits inside what the operator said they would run unattended.

And the terminal branch is ask, not deny. Tolerance never refuses anything. It only decides whether a human is inserted.

What tolerance can and cannot do

Two settings, ceilings 30 and 70. There is no "high".

Low's ceiling of 30 is the top of the low tier, which auto-allows anyway — so Low is behaviourally a no-op, and the source says so outright. Medium additionally auto-runs Noted, Weighty, Heightened and Gated. That is the entire dial: one setting, one decade-boundary move, and an explicit confirmation required to enable it.

Exactly four things it cannot lift, because all four resolve before the ceiling is even evaluated: injection, hard-deny floors, ask-only floors, and an operator's explicit deny rule.

An unknown tolerance value collapses to the strictest ceiling. A conformance test passes the literal string 'bogus' and asserts the verdict is still ask.

Two honest notes. The comparison is inclusive, so a score of exactly 70 auto-runs under Medium — a known ceiling-hugging exposure, accepted in the spec and mitigated by the irreversibility floor rather than by the number. And the fallback uses ??, which only catches null and undefined; a tolerance string colliding with an Object.prototype key would yield a function rather than undefined, and the comparison against a function evaluates false, producing an ask. It fails safe — but by accident of JavaScript rather than by design, which is worth writing down rather than admiring.

Making a single 10 visible

Every assessment carries its top three contributing categories, filtered to exclude anything still at the default and sorted with a deterministic tie-break by id.

That is the detection half of the same attack the max-driven composite prevents structurally. A lone maxed category produces a score of 100 and names itself in the audit record. Categories at 1 are filtered out rather than padding the list with noise, so a fully-default action reports no contributors at all instead of three meaningless ones.

What this does not do

The scorer is deterministic and total — same categories in, same score out, no exceptions thrown, a fail-safe answer for every malformed input. What it cannot do is know whether the right categories were scored.

Absence defaults to safe. That is the correct call for usability and it is a real exposure: an action whose dangerous dimension was never assessed scores as though that dimension were benign. No arithmetic inside the scorer can detect a claim nobody made. The defence lives upstream, in whatever produces the category vector, and in the floors that do not depend on the composite being right.

Which is the honest summary of the whole design. The score is a convenience mechanism — it decides how much friction you are spared. The floors are the control, and they were deliberately built so that getting the score wrong cannot open them.

Project Starfish · a governance-first, deny-by-default AI ecosystem · Apache-2.0. This devlog is a backdated build journal reconstructed from the project history.