Harmonia

Evaluation pipeline

Each LLM is scored from a corpus of its public artefacts — model and system cards, licence text, technical reports, safety policies — passed through a transformer encoder with five regression heads, one per governance dimension.

Layers

Functions

• GELU — activation inside the encoder feed-forward blocks; smoother than ReLU for language features.
• Softmax — over attention logits, so each token attends across the document.
• Sigmoid — squashes each head to (0,1), then ×100 for the dimension score.
• Huber loss — per-head regression against human-rated labels; robust to outlier ratings.
• AdamW — optimiser with weight decay; cosine learning-rate schedule, warmup 500 steps.
• Layer normalisation — stabilises each residual block during training.

Labels would come from human raters scoring artefacts against the same rubric, so the network learns to reproduce a transparent standard rather than an opaque one. Until that training set exists, the rubric in Method is the ground truth and the live scores follow it directly.

At a glance

By jurisdiction

Ethical-AI roadmap by country

Illustrated guide — how a score forms

Read left to right: a prompt or model is scored across six dimensions; those scores then feed two readouts at once — their weighted magnitude is quantised onto the Fibonacci scale (the band), while their balance is mapped to musical intervals that stack into a chord. Magnitude + profile = the combined signature.

For a model the six dimensions are different (openness, licence, documentation, safety, data — five of them), but the mechanism is identical: dimensions in, a Fibonacci band and a chord out.

Why a Fibonacci scale

Scores land on 1 · 2 · 3 · 5 · 8 · 13 · 21. The numbers are ordinal labels, chosen for the same reason agile teams estimate in Fibonacci: they discourage false precision. They are not claimed to derive from φ, and the thresholds are not φ-spaced. Saying so plainly matters, because v1 of this page asserted a golden scale while the arithmetic underneath used near-uniform bands — and drew a dial whose arcs were sized by the Fibonacci values instead, so band 21 covered eight score-points but forty per cent of the ring.

There is now one source of truth. The band thresholds and the arc widths both come from a single table, and every arc is as wide as the slice of the 0–100 composite that band actually covers. The dial can no longer contradict the number it is displaying.

The weakest-link gate

A weighted mean is compensatory: five strong dimensions quietly absorb one catastrophic one. Under the earlier aggregation a prompt scoring 5 on harm avoidance and 100 on everything else returned a composite of 81 and read as band 13, Strong — a false negative on precisely the case the index exists to catch. The mean now sets only the ceiling. A second, non-compensatory rule sets the floor: the band may sit at most one step above the band the weakest single dimension earns on its own, and any dimension below 25 forfeits even that step. A score with a hole in it now reads as a score with a hole in it.

The gate is scoped, and the scope is part of the rubric. For a prompt all six dimensions are gated, because any one of them can make a request unsafe by itself. For the provider atlas only the safety-framework dimension is: closed weights and a restrictive licence are business choices rather than failures, and gating on them would collapse every lab into the same band and assert something this index does not mean.

The six dimensions

A prompt is read against harm avoidance, fairness & non-discrimination, privacy & consent, honesty & transparency, autonomy & non-manipulation, and legality & rights — a plain-language distillation of widely used responsible-AI frameworks. Each is scored 0–100, then the set is mapped to a single Fibonacci band.

What the music actually is

The sound is a representation layer, not the measurement — there is no scientific link between harmony and ethics, and HARMONIA does not pretend otherwise. Each dimension becomes a note; a high score sits at a consonant interval to the root, a low one at a rough interval. So a well-aligned prompt stacks into a stable chord and a problematic one grinds. The music encodes the profile; the Fibonacci number encodes the magnitude.

Three things were wrong with how that was done and are now fixed. The interval above 80 was chosen from the parity of the score — odd gave an octave, even a fifth — so 84 and 85 sounded different for no reason at all; the mapping is now a pure function of the score. The ladder also placed the tritone below the minor second, which made the worst band audibly less rough than the band above it; intervals now follow the standard Plomp–Levelt roughness ordering — octave, fifth, major third, minor third, tritone, major seventh, minor second — so roughness rises strictly as the score falls. And the six voices each played at a fixed gain that summed past clipping; they are now equal-power scaled by voice count and routed through a limiter.

On the page, notes that share a staff step used to be drawn at identical coordinates and vanish into one another; colliding heads are now displaced by a notehead width, as engraving requires, and every chord carries a legend naming which dimension produced which note. Six dimensions collapsing into a handful of pitches is still lossy — that is inherent to the metaphor — but the loss is now legible rather than hidden.

Reproducibility, provenance and error

A live reading is the median of three independent judgements and reports the spread as a confidence interval, so a composite sitting on a threshold is shown as provisional rather than printed as a fact. The artefact under audit is passed between random nonce delimiters and the judge is told, in the system message, that everything between them is data — a prompt can no longer type its way out of a fixed quote and instruct its own auditor. Every returned field is schema-checked before it touches the arithmetic; previously a malformed response produced a silent NaN that rendered as a confident Critical. Every reading is stamped with the rubric version, engine, transport, sample size, timestamp and run id, and every provider score carries the date it was last reviewed and the class of public artefact it was read from.

The Eval tab is the part that matters most. It holds a small, versioned golden set with human-consensus labels and reports the index’s own band accuracy, composite error, and recall and precision on the unsafe cases. Thirty-six items make it directional, not validated — but an index that publishes its error rate is a different kind of object from one that does not.

On evaluating providers

You cannot “read the source code” of the frontier models — Claude, GPT, and Gemini are closed. What you can read is published: licences, model and system cards, technical reports, and safety/responsible-scaling policies, plus open weights for models like Llama, Qwen, DeepSeek, Mistral, and Gemma. The atlas scores exactly those public artefacts. It is one reading, compiled as a snapshot, and meant to be checked — not an authority.

Batch scoring

The offline screen runs instantly and locally. The judge makes one call per prompt at a concurrency of three and needs a proxy endpoint.

Golden set & self-accuracy

The offline run is instant and needs nothing. The judge run needs a proxy endpoint and makes one call per item.