Translation metrics cannot judge what their tokeniser deletes
Why “palimpsest”
TL;DR. Some invisible Unicode corruptions never reach a neural translation metric: the tokeniser erases them first, so a corrupted translation and a clean one arrive as the same token sequence and receive the same score. We measure that across XLM-R and mT5, build LIGATUR, a 171-item challenge set that separates byte-level corruption from semantic error, and build AEGIS, a metric that scores a cleaned hypothesis and charges the corruption through transparent penalties.
A palimpsest is a manuscript scraped clean and written over, with the old text still faintly there. That is the effect studied here: a translation is corrupted at the byte level (a no-break space for a space, a byte-order mark in front, a zero-width character inside a word), the tokeniser scrapes the corruption off, and the metric reads the surface underneath. A human looking at the string sees nothing; the metric sees nothing; the pipeline downstream sees a perfectly clean sentence.
Where the information disappears
Neural metrics such as COMET-22 and MetricX-24 do not read text; they read token ids. COMET-22 is built on XLM-R and MetricX-24 on mT5, and both tokenisers are SentencePiece models. Before SentencePiece cuts a string into pieces, a normaliser rewrites it, and that rewrite is where the palimpsest effect lives.
The question is empirical: for each kind of invisible corruption, does the token sequence change? The probe answers it for every clean sentence in the challenge set (252 distinct German and Hindi target strings), under both encoders.
The probe: twelve classes, a clean split
For each class the probe applies the perturbation to every applicable sentence and checks tok(clean)["input_ids"] == tok(perturbed)["input_ids"]. A sentence is “applicable” when the perturbation exists for it (space substitution needs a space; intra-word insertions need a word of at least four letters; the homoglyph needs a Latin “o”; the conjunct class needs a virama).
Four classes are erased: a no-break space, a narrow no-break space or an ideographic space substituted for every space, and a byte-order mark at the very start of the string. Eight survive. The result is identical for XLM-R and mT5, which looks like a coincidence until you see that the two tokenisers ship the same normalisation rules.
Try it yourself
The widget below models exactly that one step, using the rules measured from the real tokenisers (listed under it). Pick a class, type any sentence, and see what the encoder receives.
The mechanism, from the tokenisers’ own config
Both xlm-roberta-large and google/mt5-large are SentencePiece models with a 250,000-piece vocabulary and the same nmt_nfkc normaliser: an identical 237,539-byte precompiled character map, plus remove_extra_whitespaces and add_dummy_prefix. Hugging Face exposes it as a Precompiled normaliser, so you can feed it single characters and read the output. This check is not in the repository; it is three lines to reproduce:
from transformers import AutoTokenizer
for name in ("xlm-roberta-large", "google/mt5-large"):
tok = AutoTokenizer.from_pretrained(name)
for c in ("\u00a0", "\u202f", "\u3000", "\ufeff", "\u200b", "\u2060", "\u00ad"):
print(name, f"U+{ord(c):04X}", repr(tok.backend_tokenizer.normalizer.normalize_str("x" + c + "y")))| Code point | Name | After the normaliser | Result |
|---|---|---|---|
| Rewritten to an ASCII space (identical for XLM-R and mT5) | |||
| U+00A0 | no-break space | ASCII space | same ids as clean (when it replaces a space) |
| U+202F | narrow no-break space | ASCII space | same ids as clean |
| U+3000 | ideographic space | ASCII space | same ids as clean |
| U+FEFF | byte-order mark | ASCII space | leading: stripped, same ids · inside a word: a real word break, ids differ |
| U+200B / 200C / 200D | zero-width space, ZWNJ, ZWJ | ASCII space | inside a word: splits it, ids differ |
| U+200E / 200F | LRM, RLM | ASCII space | inside a word: splits it, ids differ |
| Passed through unchanged | |||
| U+2060 | word joiner | kept as a piece | ids differ |
| U+00AD | soft hyphen | kept as a piece | ids differ |
| U+034F | combining grapheme joiner | kept as a piece | ids differ |
| U+2063 · U+E0101 | invisible separator · variation selector 18 | kept as a piece | ids differ |
| U+043E | Cyrillic small о | kept as a letter | ids differ (a different character) |
normalizer_spec and precompiled character map). Both tokenisers use the same 237,539-byte map, which is why the two columns of the probe are identical. Source: measured from xlm-roberta-large and google/mt5-large; not part of the repositoryThis gives an operational rule: a corruption that is ordinary whitespace in disguise is invisible to these metrics; a corruption that is a different character is visible as a token, even though a human cannot see it. The practical consequence is uneven: the metric can react to a zero-width space but cannot react to a no-break space.
LIGATUR: a challenge set built to separate the two failure modes
A metric that fails on an invisible corruption is not necessarily bad at translation. LIGATUR is built to tell the two apart: the same metric is tested on byte-level corruption (meaning and rendering unchanged), on visible script errors that a competent metric should catch, and on semantic errors that confirm the set does not simply penalise all variation.
Each item is a contrastive pair over one English source: the reference, a correct hypothesis and a minimally corrupted one in randomised slots. The corrupted variants come from deterministic injectors (for the script-form and Unicode phenomena) or are authored as minimal pairs (for the semantic ones). A fixed seed (13) makes the generator reproducible.
{"segment_id": "en-de_#_LIGATUR_#_num-de-01_#_00001",
"doc_id": "en-de_#_LIGATUR_#_num-de-01",
"src_lang": "en", "tgt_lang": "de",
"src": "The company was founded in 1997 and now employs 250 people.",
"tgt": {"LIGATUR_hyp1": "Das Unternehmen wurde 1997 gegruendet und beschaeftigt heute 2500 Mitarbeiter.",
"LIGATUR_hyp2": "Das Unternehmen wurde 1997 gegruendet und beschaeftigt heute 250 Mitarbeiter."},
"ref": {"refA": "Das Unternehmen wurde 1997 gegruendet und beschaeftigt heute 250 Mitarbeiter."}}| Class | Clean | Corrupted | Code point |
|---|---|---|---|
| Leading BOM | Mitarbeiter muessen vor Ort einen Helm tragen. | ⟦FEFF⟧Mitarbeiter muessen… | U+FEFF at index 0 |
| No-break space | Der Vertrag wird automatisch verlängert. | …automatisch⟦00A0⟧verlängert. | U+00A0 replaces one space |
| Soft hyphen | Die Bewerbung muss heute eingereicht werden. | …heute e⟦00AD⟧ingereicht… | U+00AD |
| Zero-width pepper | Bitte reichen Sie das Formular heute ein. | Bitte r⟦200B⟧e⟦200B⟧i⟦200B⟧c⟦200B⟧h⟦200B⟧e⟦200B⟧n Sie… | U+200B between letters |
| Cyrillic homoglyph | …Antibiotika gegen die Infektion. | …Antibi⟦043E⟧tika gegen die Infekti⟦043E⟧n. | U+043E for Latin o |
| Homoglyph key words | Der Patient ist allergisch gegen Penicillin. | Der Patient ist ⟦0430⟧llergisch gegen ⟦0420⟧enicillin. | U+0430, U+0420 |
| Default-ignorable | Bitte reichen Sie den Antrag heute ein. | …heute⟦E0101⟧ ein. | U+E0101 variation selector |
| Devanagari conjunct | यह कार्य महत्वपूर्ण है। | …महत्वपूर्⟦200C⟧ण… | ZWNJ after virama U+094D |
| Conjunct, ZWJ variant | यह ज्ञान सफलता के लिए आवश्यक है। | …ज्⟦200D⟧ञ… | ZWJ after virama |
The full list of corruption injectors
| Injector | Behaviour |
|---|---|
| ghost / ghost_zw | 1–3 insertions of zero-width or bidi characters (U+200B–U+200F, U+2060) at random positions; ghost also swaps in a Cyrillic homoglyph 30% of the time |
| ghost_homo / homo_key | swap 1–2 eligible Latin letters for Cyrillic look-alikes (a, e, o, c; homo_key adds p, x, y) |
| conjunct / conjunct_zwj / …_cgj / …_wj | insert ZWNJ, ZWJ, CGJ or WJ immediately after a random virama U+094D |
| lig_zwnj_all | ZWNJ after every virama |
| soft_hyphen | 1–2 soft hyphens inside word interiors |
| bom · prefix_spam | BOM + text, or BOM + 3 × ZWSP as a prefix |
| nbsp · nbsp_all | one ASCII space → U+00A0, or all of them |
| zwsp_pepper · shy_pepper · wj_pepper · cgj_pepper | the character between all letters of one long word |
| bidi_sprinkle · cgj_mid | LRM and RLM in the first ~14 characters · one CGJ at a random interior position |
| (default) corrupt_number | one number changed (year digits, count ±, ×2, ×5, ×10) |
| authored bad | semantic controls written by hand as minimal pairs |
perturb.py. Corrupted hypotheses are generated from correct translations by rule; semantic controls are authored. Source: code/perturb.pyFormat details follow the official shared-task rules: a set name of at most seven letters (LIGATUR is seven), one language pair per file, <src>.<tgt>.<Name>.jsonl file names, the same number of hypotheses and references per file and a 1,000,000-token cap (the set is about 4.7k whitespace tokens).
AEGIS: score the cleaned text, charge for the corruption
If the neural metrics cannot see invisible corruption, a metric built from them needs another channel. AEGIS makes that channel explicit rather than hoping the encoders notice. Neural metrics score a cleaned hypothesis; a deterministic hygiene shield turns byte-level corruption into a transparent penalty in the same 0–100 points.
Fusion
Three neural signals are blended: COMET-22 and MetricX-24 in reference mode (averaged into an ensemble), and MetricX-24 in reference-free mode as a floor that is always available. How much the blend leans on the reference depends on how much the reference can be trusted.
| Reference type | Base weight w | If refQE > median hypothesis error + 3 | Reasoning |
|---|---|---|---|
| human | 0.85 | 0.425 | trusted reference: lean on the reference-based ensemble |
| postedit | 0.85 | 0.425 | same |
| pseudo (synthetic) | 0.70 | 0.35 | a machine-made reference deserves less weight |
| undefined / other (e.g. challenge-set rows) | 0.70 | 0.35 | default |
| no reference | 0 | – | pure reference-free MetricX-24 QE |
The hygiene shield
| Flag | Fires when… | Per hit | Cap |
|---|---|---|---|
invisible | a character from a 26-code-point default-ignorable set (ZWSP, WJ, BOM, CGJ, LRM/RLM, bidi embeddings and isolates, invisible operators, Hangul fillers, Mongolian selectors) appears in the hypothesis and not in the source or reference | 2.0 | 8.0 |
soft_hyphen | U+00AD in the hypothesis, unless it occurs in the source or reference | 1.0 | 3.0 |
ws_variant | a non-ASCII space (NBSP, narrow NBSP, U+2000–200A, U+205F, U+3000, U+1680) not present in the trusted texts | 0.5 | 2.0 |
zw_joiner | ZWNJ/ZWJ in the hypothesis that is absent from the trusted text, or present but not directly after a virama U+094D | 1.5 | 6.0 |
homoglyph_cloak | a word that mixes a Cyrillic/Greek look-alike with Latin letters, and whose look-alike-folded form occurs in the trusted text | 2.0 | 8.0 |
nfc_mismatch | the hypothesis is not NFC-normalised while the source and reference are | 0.5 | 0.5 |
def _clean(text):
# what the neural metrics are shown
text = "".join(c for c in text if c not in INVISIBLE and c != SOFT_HYPHEN)
text = "".join(" " if c in WS_VARIANTS else c for c in text)
return unicodedata.normalize("NFC", text)
# a penalty only counts characters absent from BOTH trusted texts
trusted = set(src + (ref or ""))
bad = sum(1 for c in hyp if c in INVISIBLE and c not in trusted)
penalty += min(bad * 2.0, 8.0)The degeneracy guard
| Flag | Trigger | Penalty |
|---|---|---|
empty | the hypothesis is blank | 60 (flat) |
runaway_length | source ≥ 10 words and the hypothesis is more than 3× as long | min(40, 10·(ratio − 3) + 10) |
repetition_loop | a hypothesis of ≥ 30 words where the most frequent 3-, 4- or 5-gram occurs ≥ 8 times and covers more than half of the words | min(40, 4·count) |
One score, assembled
| Quantity | Value | How |
|---|---|---|
| COMET-22 · MetricX-24 · QE (illustrative inputs) | 0.86 · 2.5 · 4.0 | chosen for the example, not measured |
| ens | 88.0 | 0.5·86 + 0.5·(100 − 10) |
| q100 | 84.0 | 100 − 16 |
| blend, w = 0.85 (human reference) | 87.4 | 0.85·88 + 0.15·84 |
| shield: one invisible character + one NBSP | 2.5 | 2.0 + 0.5 |
| AEGIS score of the corrupted hypothesis | 84.9 | 87.4 − 2.5 |
| AEGIS score of its clean twin | 87.4 | neural metrics saw the same cleaned text, shield = 0 |
What the numbers say
The repository contains one set of published metric numbers: COMET-22’s failures on seven LIGATUR phenomena, stored as constants in the CPU re-computation script so the author’s run can be diffed against them. They line up with the tokeniser probe in the way the hypothesis predicts.
| Phenomenon | Items | Fails | Fail rate | Mean margin | Token ids |
|---|---|---|---|---|---|
| Reference-copy | 5 | 5 | 100% | −0.075 | semantic |
| Byte-order mark | 6 | 6 | 100% | 0.0000 | identical |
| Non-breaking space | 6 | 5 | 83% | 0.0000 | identical |
| ZW joiner (en–de) | 12 | 1 | 8% | 0.107 | differ |
| Digit swap | 5 | 0 | 0% | 0.013 | semantic |
| Soft hyphen | 8 | 0 | 0% | 0.036 | differ |
| Conjunct (en–hi) | 23 | 0 | 0% | 0.074 | differ |
Unbabel/wmt22-comet-da, reference mode) on LIGATUR. The fail rate is derived from the counts; the margin is the mean of score(good) − score(bad). “Token ids” says whether the corruption changes the encoder input (from the probe). Source: code/mac_ligatur_comet.py; fail rate computed from fails / itemsRead left to right, the table is the argument in miniature. COMET-22 does well on everything the tokeniser does not erase (soft hyphens, conjuncts, digit swaps) and fails completely on what it does, with a margin of precisely zero. The one NBSP item that COMET-22 passed is not explained in the repository, so it should not be over-interpreted. Reference-copy is a different failure again: the reference-mirroring candidate is preferred even though it flips the meaning.
The AEGIS submission includes a self-check against LIGATUR: validate_submission.py counts the pairs where the submitted score ranks the corrupted hypothesis at or above the correct one and prints an expected baseline of about 6 failures in 159 pairs. The committed code also contains the scripts for the WMT24 ESA correlation analysis (segment-level Pearson with bootstrap confidence intervals, Kendall τ, pairwise ranking accuracy, a sensitivity grid over w), but the inputs are distributed elsewhere, so those numbers are not reproduced here.
Auditing the shield on the released set
Because the released items nearly always contain the reference as one of the two hypotheses, anyone can run the shield over LIGATUR and see which corruptions it catches.
refA as correct and the other as corrupted (162 of 171 items are usable), and count items where the shield penalises only the corrupted one. It fires on 101 of 162, on every invisible-character and look-alike family except the “default-ignorable” items, and never on semantic controls, as intended. The 12 default-ignorable items use U+E0101 (ten items) and U+2063 (two); the 26-point INVISIBLE set does not include U+E0101, so only the U+2063 pair is caught. Source: derived from the released data and code/aegis_shield.py; not a number from the paperWhat this does and does not show
- It shows that four classes of invisible corruption are token-identical to clean text on every applicable sentence, under XLM-R and mT5, and that the other eight are not.
- It shows that for those four classes, an XLM-R-based reference metric has a score margin of exactly zero (COMET-22: BOM and NBSP).
- It does not show that every neural metric is blind: a metric built on a different tokeniser, or an LLM judge that reads raw characters, behaves differently. The probe covers the two encoders behind COMET and MetricX only.
- It does not show how often these corruptions occur in real MT output; LIGATUR measures sensitivity, not frequency.
- LIGATUR is small (171 items, two language pairs), and its Devanagari items are byte-level correct but not yet confirmed by native readers.
Reproduce the probe
It runs end to end from the repository alone, needs only the two tokenisers, and finishes in seconds on a CPU.
git clone https://github.com/amethystani/palimpsest && cd palimpsest
uv run --with transformers --with sentencepiece --with protobuf \
python code/tokenizer_probe_broad.py
# diff the output against the committed results:
# data/probe_broad_results.jsonThe repository’s probe_broad_results.json is committed so you can diff your run against it; re-running the probe gives an identical file. The shared-task data lives in wmt-conference/wmt26-mteval. Models: xlm-roberta-large, google/mt5-large, Unbabel/wmt22-comet-da.


