Translation metrics cannot judge what their tokeniser deletes

Listen
Share

Why “palimpsest”

TL;DR. Some invisible Unicode corruptions never reach a neural translation metric: the tokeniser erases them first, so a corrupted translation and a clean one arrive as the same token sequence and receive the same score. We measure that across XLM-R and mT5, build LIGATUR, a 171-item challenge set that separates byte-level corruption from semantic error, and build AEGIS, a metric that scores a cleaned hypothesis and charges the corruption through transparent penalties.

A palimpsest is a manuscript scraped clean and written over, with the old text still faintly there. That is the effect studied here: a translation is corrupted at the byte level (a no-break space for a space, a byte-order mark in front, a zero-width character inside a word), the tokeniser scrapes the corruption off, and the metric reads the surface underneath. A human looking at the string sees nothing; the metric sees nothing; the pipeline downstream sees a perfectly clean sentence.

171items in LIGATUR (95 en–de, 76 en–hi)
12corruption classes probed
4classes erased by the tokeniser
251 / 251sentences identical under both XLM-R and mT5
Two different strings, one token sequenceA clean sentence and a copy with a no-break space both pass through the tokeniser and come out as the same token ids, so the metric returns the same score.CleanDer Vertrag wird automatisch verlängert.Corrupted…automatisch⟦00A0⟧verlängert.what the human readsTokenisernmt_nfkcnormaliserOne token sequenceXLM-R and mT5: identical ids251 of 251 sentencesscore(clean) = score(corrupted)The corruption is a real change to the bytes and a no-op for the model.Margin between the two translations: exactly 0.
Figure 1 The whole idea. A metric cannot rank what its tokeniser merges: if the corrupted and the clean translation reach the encoder as the same token sequence, any encoder-based score is identical, and the corruption is invisible however good the model is. Source: github.com/amethystani/palimpsest · README.md; data/probe_broad_results.json

Where the information disappears

Neural metrics such as COMET-22 and MetricX-24 do not read text; they read token ids. COMET-22 is built on XLM-R and MetricX-24 on mT5, and both tokenisers are SentencePiece models. Before SentencePiece cuts a string into pieces, a normaliser rewrites it, and that rewrite is where the palimpsest effect lives.

Where in a metric the information is lostA pipeline from raw text through the normaliser, the SentencePiece model, token ids and the encoder to a score, marking the normaliser as the step that erases some invisible characters.Raw textbytes as writtenNormalisernmt_nfkc charsmapSentencePiecepieces and idsEncoderXLM-R · mT5Scoremetric outputinvisible charactersRewritten to a space, then trimmed:U+00A0 · U+202F · U+3000 · leading U+FEFFSurvive as odd pieces (ids differ):U+2060 · U+00AD · U+034F · Cyrillic оEverything to the right of the normaliser neversees the difference. A better encoder cannot fixwhat was already removed upstream.
Figure 2 The pipeline behind every XLM-R or mT5 metric. The highlighted step is where the corrupted and clean strings stop being different, before any learned weight is involved. This is why scaling the model does not help. Source: github.com/amethystani/palimpsest · code/tokenizer_probe_broad.py (probe); mechanism checked from the tokenisers’ own configuration, see below

The question is empirical: for each kind of invisible corruption, does the token sequence change? The probe answers it for every clean sentence in the challenge set (252 distinct German and Hindi target strings), under both encoders.

The probe: twelve classes, a clean split

For each class the probe applies the perturbation to every applicable sentence and checks tok(clean)["input_ids"] == tok(perturbed)["input_ids"]. A sentence is “applicable” when the perturbation exists for it (space substitution needs a space; intra-word insertions need a word of at least four letters; the homoglyph needs a Latin “o”; the conjunct class needs a virama).

Tokeniser probe over 252 clean sentencesFor each of twelve corruption classes, the number of sentences whose token ids equal the clean text and the number whose ids differ, identical for XLM-R and mT5.token ids identical to clean texttoken ids differsame / applicableNBSP for every spaceU+00A0251 / 251Narrow NBSP for every spaceU+202F251 / 251Ideographic spaceU+3000251 / 251BOM at string startU+FEFF252 / 252BOM inside a wordU+FEFF0 / 250Zero-width spaceU+200B0 / 250Word joinerU+20600 / 250Soft hyphenU+00AD0 / 250Combining grapheme joinerU+034F0 / 250Left-to-right markU+200E0 / 250Cyrillic homoglyph (o → о)U+043E0 / 86Conjunct + ZWNJ after viramaU+200C0 / 93
Figure 3 The probe, in one picture. Four of twelve corruption classes are token-identical to clean text on every applicable sentence, under both XLM-R and mT5; the other eight are different on every applicable sentence. There is no middle: no class is “sometimes” invisible. Bar length is proportional to the number of applicable sentences (252 distinct clean target strings: 132 German, 120 Hindi). Source: github.com/amethystani/palimpsest · data/probe_broad_results.json (re-run: identical)

Four classes are erased: a no-break space, a narrow no-break space or an ideographic space substituted for every space, and a byte-order mark at the very start of the string. Eight survive. The result is identical for XLM-R and mT5, which looks like a coincidence until you see that the two tokenisers ship the same normalisation rules.

Try it yourself

The widget below models exactly that one step, using the rules measured from the real tokenisers (listed under it). Pick a class, type any sentence, and see what the encoder receives.

Unicode lensneeds JavaScript

An interactive demo of the normaliser: U+00A0, U+202F, U+3000 and a leading U+FEFF are rewritten to a space and stripped, so those corruptions leave the token sequence unchanged; U+2060, U+00AD, U+034F and Cyrillic о pass through and change it.

Token count for one German sentence under XLM-RToken count for one German sentence under XLM-Rclean19NBSP for every space19Narrow NBSP for every space19BOM at string start19BOM inside a word22Zero-width space22Word joiner22Soft hyphen22Combining grapheme joiner22filled = ids differ from clean
Figure 4 The single-sentence version of the probe: “Das Unternehmen wurde 1997 gegruendet und beschaeftigt heute 250 Mitarbeiter.” stays at 19 tokens with the first four corruptions and grows to 22 with the others. A Devanagari conjunct (क्ष) is one token intact and four tokens once a ZWNJ is inserted after the halant. Source: github.com/amethystani/palimpsest · code/tokenizer_probe.py (re-run: matches)

The mechanism, from the tokenisers’ own config

Both xlm-roberta-large and google/mt5-large are SentencePiece models with a 250,000-piece vocabulary and the same nmt_nfkc normaliser: an identical 237,539-byte precompiled character map, plus remove_extra_whitespaces and add_dummy_prefix. Hugging Face exposes it as a Precompiled normaliser, so you can feed it single characters and read the output. This check is not in the repository; it is three lines to reproduce:

python
from transformers import AutoTokenizer
for name in ("xlm-roberta-large", "google/mt5-large"):
    tok = AutoTokenizer.from_pretrained(name)
    for c in ("\u00a0", "\u202f", "\u3000", "\ufeff", "\u200b", "\u2060", "\u00ad"):
        print(name, f"U+{ord(c):04X}", repr(tok.backend_tokenizer.normalizer.normalize_str("x" + c + "y")))
Code pointNameAfter the normaliserResult
Rewritten to an ASCII space (identical for XLM-R and mT5)
U+00A0no-break spaceASCII spacesame ids as clean (when it replaces a space)
U+202Fnarrow no-break spaceASCII spacesame ids as clean
U+3000ideographic spaceASCII spacesame ids as clean
U+FEFFbyte-order markASCII spaceleading: stripped, same ids · inside a word: a real word break, ids differ
U+200B / 200C / 200Dzero-width space, ZWNJ, ZWJASCII spaceinside a word: splits it, ids differ
U+200E / 200FLRM, RLMASCII spaceinside a word: splits it, ids differ
Passed through unchanged
U+2060word joinerkept as a pieceids differ
U+00ADsoft hyphenkept as a pieceids differ
U+034Fcombining grapheme joinerkept as a pieceids differ
U+2063 · U+E0101invisible separator · variation selector 18kept as a pieceids differ
U+043ECyrillic small оkept as a letterids differ (a different character)
What the shared SentencePiece normaliser does to each code point (from the models’ own normalizer_spec and precompiled character map). Both tokenisers use the same 237,539-byte map, which is why the two columns of the probe are identical. Source: measured from xlm-roberta-large and google/mt5-large; not part of the repository

This gives an operational rule: a corruption that is ordinary whitespace in disguise is invisible to these metrics; a corruption that is a different character is visible as a token, even though a human cannot see it. The practical consequence is uneven: the metric can react to a zero-width space but cannot react to a no-break space.

LIGATUR: a challenge set built to separate the two failure modes

A metric that fails on an invisible corruption is not necessarily bad at translation. LIGATUR is built to tell the two apart: the same metric is tested on byte-level corruption (meaning and rendering unchanged), on visible script errors that a competent metric should catch, and on semantic errors that confirm the set does not simply penalise all variation.

The LIGATUR contrastive itemA source sentence and a reference feed two hypotheses, one correct and one minimally corrupted; the gold key is withheld; a metric must score the correct one higher.srcEnglish sourceref (refA)one reference translationLIGATUR_hyp1one slot…LIGATUR_hyp2…the other slotcorrect translationusually equal to the refminimally corruptedinjector or authored pairorder randomised per itemTask for the metricscore(correct) > score(corrupted)Fail when score(bad) ≥ score(good): a tie counts as a failure. The key is private.
Figure 5 One LIGATUR item. The set is a collection of contrastive pairs: same meaning, one corrupted. Gold labels are withheld and the order varies, so a metric cannot win by position. Source: github.com/amethystani/palimpsest · data/LIGATUR_README.md; code/perturb.py
LIGATUR items by corruption family (171 total)LIGATUR items by corruption family (171 total)Devanagari ZW joinersconjunct 23 · czwj 8 · zwnjnoise 334Zero-width / bidi mixesghost 13 · ghostzw 1225Cyrillic homoglyphsghosthomo 12 · homok 416Default-ignorablesU+E0101 ×10 · U+2063 ×212Soft hyphensofth8Byte-order markbom-de6No-break spacenbsp 4 de · 2 hi6Zero-width space pepperzwsp-de4Visible script controlshalant · matra · nukta · digits7Semantic controlsnumbers, units, negation, register, …53
Figure 6 What is in the 171 items. The filled bars are the 111 items that corrupt only invisible or look-alike characters; the last two bars are controls that confirm the set is not simply penalising all variation. Source: github.com/amethystani/palimpsest · tally of doc_id prefixes in data/en.de.LIGATUR.jsonl and data/en.hi.LIGATUR.jsonl

Each item is a contrastive pair over one English source: the reference, a correct hypothesis and a minimally corrupted one in randomised slots. The corrupted variants come from deterministic injectors (for the script-form and Unicode phenomena) or are authored as minimal pairs (for the semantic ones). A fixed seed (13) makes the generator reproducible.

data/en.de.LIGATUR.jsonl (first line)
{"segment_id": "en-de_#_LIGATUR_#_num-de-01_#_00001",
 "doc_id": "en-de_#_LIGATUR_#_num-de-01",
 "src_lang": "en", "tgt_lang": "de",
 "src": "The company was founded in 1997 and now employs 250 people.",
 "tgt": {"LIGATUR_hyp1": "Das Unternehmen wurde 1997 gegruendet und beschaeftigt heute 2500 Mitarbeiter.",
         "LIGATUR_hyp2": "Das Unternehmen wurde 1997 gegruendet und beschaeftigt heute 250 Mitarbeiter."},
 "ref": {"refA": "Das Unternehmen wurde 1997 gegruendet und beschaeftigt heute 250 Mitarbeiter."}}
ClassCleanCorruptedCode point
Leading BOMMitarbeiter muessen vor Ort einen Helm tragen.⟦FEFF⟧Mitarbeiter muessen…U+FEFF at index 0
No-break spaceDer Vertrag wird automatisch verlängert.…automatisch⟦00A0⟧verlängert.U+00A0 replaces one space
Soft hyphenDie Bewerbung muss heute eingereicht werden.…heute e⟦00AD⟧ingereicht…U+00AD
Zero-width pepperBitte reichen Sie das Formular heute ein.Bitte r⟦200B⟧e⟦200B⟧i⟦200B⟧c⟦200B⟧h⟦200B⟧e⟦200B⟧n Sie…U+200B between letters
Cyrillic homoglyph…Antibiotika gegen die Infektion.…Antibi⟦043E⟧tika gegen die Infekti⟦043E⟧n.U+043E for Latin o
Homoglyph key wordsDer Patient ist allergisch gegen Penicillin.Der Patient ist ⟦0430⟧llergisch gegen ⟦0420⟧enicillin.U+0430, U+0420
Default-ignorableBitte reichen Sie den Antrag heute ein.…heute⟦E0101⟧ ein.U+E0101 variation selector
Devanagari conjunctयह कार्य महत्वपूर्ण है।…महत्वपूर्⟦200C⟧ण…ZWNJ after virama U+094D
Conjunct, ZWJ variantयह ज्ञान सफलता के लिए आवश्यक है।…ज्⟦200D⟧ञ…ZWJ after virama
Real items from the set. The corrupted hypotheses are shown with each invisible character written as a tag; on screen they look identical to the clean text. Source: data/en.de.LIGATUR.jsonl, data/en.hi.LIGATUR.jsonl
The full list of corruption injectors
InjectorBehaviour
ghost / ghost_zw1–3 insertions of zero-width or bidi characters (U+200B–U+200F, U+2060) at random positions; ghost also swaps in a Cyrillic homoglyph 30% of the time
ghost_homo / homo_keyswap 1–2 eligible Latin letters for Cyrillic look-alikes (a, e, o, c; homo_key adds p, x, y)
conjunct / conjunct_zwj / …_cgj / …_wjinsert ZWNJ, ZWJ, CGJ or WJ immediately after a random virama U+094D
lig_zwnj_allZWNJ after every virama
soft_hyphen1–2 soft hyphens inside word interiors
bom · prefix_spamBOM + text, or BOM + 3 × ZWSP as a prefix
nbsp · nbsp_allone ASCII space → U+00A0, or all of them
zwsp_pepper · shy_pepper · wj_pepper · cgj_pepperthe character between all letters of one long word
bidi_sprinkle · cgj_midLRM and RLM in the first ~14 characters · one CGJ at a random interior position
(default) corrupt_numberone number changed (year digits, count ±, ×2, ×5, ×10)
authored badsemantic controls written by hand as minimal pairs
The deterministic injectors in perturb.py. Corrupted hypotheses are generated from correct translations by rule; semantic controls are authored. Source: code/perturb.py

Format details follow the official shared-task rules: a set name of at most seven letters (LIGATUR is seven), one language pair per file, <src>.<tgt>.<Name>.jsonl file names, the same number of hypotheses and references per file and a 1,000,000-token cap (the set is about 4.7k whitespace tokens).

AEGIS: score the cleaned text, charge for the corruption

If the neural metrics cannot see invisible corruption, a metric built from them needs another channel. AEGIS makes that channel explicit rather than hoping the encoders notice. Neural metrics score a cleaned hypothesis; a deterministic hygiene shield turns byte-level corruption into a transparent penalty in the same 0–100 points.

AEGIS two-stage pipelineStage one cleans each hypothesis and computes deterministic penalties; a GPU pass scores the cleaned text with COMET-22 and MetricX-24; stage two fuses scores and subtracts penalties.src · hyp · refofficial test setSTAGE 1 · gen_neural_inputs.pyhygiene shield6 deterministic rulesdegeneracy guardempty · runaway · loopscleaned hypothesisinvisibles droppedspaces normalised, NFCGPU PASSCOMET-22reference modeMetricX-24reference modeMetricX-24 QEno referenceMetricX-24 refQEreference as hypothesisSTAGE 2 · aegis_fuse.pyfuseweightstrust guardscore 0–100per segmentpenalties subtracted at the end
Figure 7 AEGIS, end to end. The key move is the split: the neural metrics see a cleaned hypothesis (so they judge meaning without being distracted by encoding noise), and the corruption is charged through a transparent, rule-based penalty that is subtracted at the end. Source: github.com/amethystani/palimpsest · code/gen_neural_inputs.py, code/aegis_fuse.py

Fusion

Three neural signals are blended: COMET-22 and MetricX-24 in reference mode (averaged into an ensemble), and MetricX-24 in reference-free mode as a floor that is always available. How much the blend leans on the reference depends on how much the reference can be trusted.

The AEGIS fusion formulaThe score is a weighted blend of a reference-based ensemble and a reference-free quality estimate, minus the shield and degeneracy penalties, clipped to 0 to 100.ens = 0.5·(100·COMET) + 0.5·(100 − 4·MetricX)q100 = 100 − 4·MetricX_QEw = 0.85 | 0.85 | 0.70 | 0.70 | 0human · postedit · pseudo · other · no referencehalved when the reference is itself far worse than the hypothesesscore = clip( w·ens + (1−w)·q100 − shield − degeneracy, 0, 100 )sub-scale: MetricX error is on [0, 25], so 100 − 4xCOMET-22 is on about [0, 1], so ×100
Figure 8 The locked fusion rule. Everything on the left is a neural quantity re-scaled to a 0–100 range; the shield and degeneracy penalties are in the same “points” so they can be subtracted directly. Source: github.com/amethystani/palimpsest · code/aegis_fuse.py (docstring and main)
Reference typeBase weight wIf refQE > median hypothesis error + 3Reasoning
human0.850.425trusted reference: lean on the reference-based ensemble
postedit0.850.425same
pseudo (synthetic)0.700.35a machine-made reference deserves less weight
undefined / other (e.g. challenge-set rows)0.700.35default
no reference0–pure reference-free MetricX-24 QE
How much AEGIS trusts the reference. The “trust guard” scores the reference itself as if it were a hypothesis (refQE) and halves w when it looks catastrophically bad relative to the hypotheses. Source: code/aegis_fuse.py

The hygiene shield

FlagFires when…Per hitCap
invisiblea character from a 26-code-point default-ignorable set (ZWSP, WJ, BOM, CGJ, LRM/RLM, bidi embeddings and isolates, invisible operators, Hangul fillers, Mongolian selectors) appears in the hypothesis and not in the source or reference2.08.0
soft_hyphenU+00AD in the hypothesis, unless it occurs in the source or reference1.03.0
ws_varianta non-ASCII space (NBSP, narrow NBSP, U+2000–200A, U+205F, U+3000, U+1680) not present in the trusted texts0.52.0
zw_joinerZWNJ/ZWJ in the hypothesis that is absent from the trusted text, or present but not directly after a virama U+094D1.56.0
homoglyph_cloaka word that mixes a Cyrillic/Greek look-alike with Latin letters, and whose look-alike-folded form occurs in the trusted text2.08.0
nfc_mismatchthe hypothesis is not NFC-normalised while the source and reference are0.50.5
The hygiene shield. Penalties only fire for characters that are absent from both the source and the reference, so legitimate typography (NBSP in numbers, ZWNJ in Indic and Persian orthography) is not punished when the trusted texts use it. Categories are capped independently; the theoretical maximum is 27.5 points. Source: code/aegis_shield.py
Maximum penalty per shield category (points)Maximum penalty per shield category (points)invisible8homoglyph cloak8zero-width joiner6soft hyphen3whitespace variant2NFC mismatch0.5
Figure 9 Per-category caps add up to 27.5 points on the 0–100 scale. A single stray NBSP costs 0.5; a word peppered with zero-width spaces hits the 8-point cap. Source: github.com/amethystani/palimpsest · code/aegis_shield.py
code/aegis_shield.py (abridged)
def _clean(text):
    # what the neural metrics are shown
    text = "".join(c for c in text if c not in INVISIBLE and c != SOFT_HYPHEN)
    text = "".join(" " if c in WS_VARIANTS else c for c in text)
    return unicodedata.normalize("NFC", text)

# a penalty only counts characters absent from BOTH trusted texts
trusted = set(src + (ref or ""))
bad = sum(1 for c in hyp if c in INVISIBLE and c not in trusted)
penalty += min(bad * 2.0, 8.0)

The degeneracy guard

FlagTriggerPenalty
emptythe hypothesis is blank60 (flat)
runaway_lengthsource ≥ 10 words and the hypothesis is more than 3× as longmin(40, 10·(ratio − 3) + 10)
repetition_loopa hypothesis of ≥ 30 words where the most frequent 3-, 4- or 5-gram occurs ≥ 8 times and covers more than half of the wordsmin(40, 4·count)
The degeneracy guard catches broken outputs that neural metrics can occasionally rate well. Source: code/aegis_shield.py

One score, assembled

QuantityValueHow
COMET-22 · MetricX-24 · QE (illustrative inputs)0.86 · 2.5 · 4.0chosen for the example, not measured
ens88.00.5·86 + 0.5·(100 − 10)
q10084.0100 − 16
blend, w = 0.85 (human reference)87.40.85·88 + 0.15·84
shield: one invisible character + one NBSP2.52.0 + 0.5
AEGIS score of the corrupted hypothesis84.987.4 − 2.5
AEGIS score of its clean twin87.4neural metrics saw the same cleaned text, shield = 0
A worked example with made-up inputs, to show how the pieces compose. The neural metrics score both twins identically (they receive the same cleaned text), so the 2.5-point gap is entirely the shield. Source: derived from the formula; inputs illustrative

What the numbers say

The repository contains one set of published metric numbers: COMET-22’s failures on seven LIGATUR phenomena, stored as constants in the CPU re-computation script so the author’s run can be diffed against them. They line up with the tokeniser probe in the way the hypothesis predicts.

COMET-22 failure rate on LIGATUR pairs (score(bad) ≥ score(good))COMET-22 failure rate on LIGATUR pairs (score(bad) ≥ score(good))Reference-copy5 / 5 · margin −0.075100%Byte-order mark6 / 6 · margin 0.0000100%Non-breaking space5 / 6 · margin 0.000083%ZW joiner (en–de)1 / 12 · margin 0.1078%Digit swap0 / 5 · margin 0.0130%Soft hyphen0 / 8 · margin 0.0360%Conjunct (en–hi)0 / 23 · margin 0.0740%
Figure 10 COMET-22 on seven LIGATUR phenomena. The two filled bars are the classes the tokeniser merges: a leading BOM and a no-break space produce a margin of exactly 0.0000, and since a tie is a failure, COMET-22 fails all six BOM items and five of six NBSP items. Reference-copy is a different failure: it prefers the reference-mirroring candidate (negative margin) in all five items. Source: github.com/amethystani/palimpsest · code/mac_ligatur_comet.py (published-values table, “Table 2”)
PhenomenonItemsFailsFail rateMean marginToken ids
Reference-copy55100%−0.075semantic
Byte-order mark66100%0.0000identical
Non-breaking space6583%0.0000identical
ZW joiner (en–de)1218%0.107differ
Digit swap500%0.013semantic
Soft hyphen800%0.036differ
Conjunct (en–hi)2300%0.074differ
COMET-22 (Unbabel/wmt22-comet-da, reference mode) on LIGATUR. The fail rate is derived from the counts; the margin is the mean of score(good) − score(bad). “Token ids” says whether the corruption changes the encoder input (from the probe). Source: code/mac_ligatur_comet.py; fail rate computed from fails / items

Read left to right, the table is the argument in miniature. COMET-22 does well on everything the tokeniser does not erase (soft hyphens, conjuncts, digit swaps) and fails completely on what it does, with a margin of precisely zero. The one NBSP item that COMET-22 passed is not explained in the repository, so it should not be over-interpreted. Reference-copy is a different failure again: the reference-mirroring candidate is preferred even though it flips the meaning.

The AEGIS submission includes a self-check against LIGATUR: validate_submission.py counts the pairs where the submitted score ranks the corrupted hypothesis at or above the correct one and prints an expected baseline of about 6 failures in 159 pairs. The committed code also contains the scripts for the WMT24 ESA correlation analysis (segment-level Pearson with bootstrap confidence intervals, Kendall τ, pairwise ranking accuracy, a sensitivity grid over w), but the inputs are distributed elsewhere, so those numbers are not reproduced here.

Auditing the shield on the released set

Because the released items nearly always contain the reference as one of the two hypotheses, anyone can run the shield over LIGATUR and see which corruptions it catches.

Does the released shield rank the correct hypothesis higher? (162 usable pairs)Does the released shield rank the correct hypothesis higher? (162 usable pairs)Devanagari ZW joiners34 / 34100%Zero-width / bidi mixes25 / 25100%Cyrillic homoglyphs16 / 16100%Soft hyphen8 / 8100%No-break space6 / 6100%Byte-order mark6 / 6100%Zero-width space pepper4 / 4100%Default-ignorables2 / 1217%Visible and semantic controls0 / 510%
Figure 11 A check anyone can re-run on the released files: treat the hypothesis equal to refA as correct and the other as corrupted (162 of 171 items are usable), and count items where the shield penalises only the corrupted one. It fires on 101 of 162, on every invisible-character and look-alike family except the “default-ignorable” items, and never on semantic controls, as intended. The 12 default-ignorable items use U+E0101 (ten items) and U+2063 (two); the 26-point INVISIBLE set does not include U+E0101, so only the U+2063 pair is caught. Source: derived from the released data and code/aegis_shield.py; not a number from the paper

What this does and does not show

  • It shows that four classes of invisible corruption are token-identical to clean text on every applicable sentence, under XLM-R and mT5, and that the other eight are not.
  • It shows that for those four classes, an XLM-R-based reference metric has a score margin of exactly zero (COMET-22: BOM and NBSP).
  • It does not show that every neural metric is blind: a metric built on a different tokeniser, or an LLM judge that reads raw characters, behaves differently. The probe covers the two encoders behind COMET and MetricX only.
  • It does not show how often these corruptions occur in real MT output; LIGATUR measures sensitivity, not frequency.
  • LIGATUR is small (171 items, two language pairs), and its Devanagari items are byte-level correct but not yet confirmed by native readers.

Reproduce the probe

It runs end to end from the repository alone, needs only the two tokenisers, and finishes in seconds on a CPU.

shell
git clone https://github.com/amethystani/palimpsest && cd palimpsest
uv run --with transformers --with sentencepiece --with protobuf \
    python code/tokenizer_probe_broad.py
# diff the output against the committed results:
#   data/probe_broad_results.json

The repository’s probe_broad_results.json is committed so you can diff your run against it; re-running the probe gives an identical file. The shared-task data lives in wmt-conference/wmt26-mteval. Models: xlm-roberta-large, google/mt5-large, Unbabel/wmt22-comet-da.

All articles

Research and building cool stuff

© 2026, Animesh Mishra

GitHub|LinkedIn

New Delhi, India