# Proxy BLEU, semantic thresholds, multilingual PPL

## 1. Proxy BLEU calculation (exact method used)

**Name:** `char_bleu4_proxy` (diagnostic only)

1. **Preprocess:** remove all whitespace (no language-specific tokenizer — keeps JA/ZH/AR comparable to Latin).
2. **n-gram precision (n = 1..4):**  
   \( p_n = \frac{\sum_g \min(c_{hyp}(g), c_{ref}(g))}{\sum_g c_{hyp}(g)} \)  
   character n-grams \(g\); clipping as in Papineni et al. BLEU.
3. **Geometric mean:** \( \exp(\frac{1}{4}\sum_{n=1}^{4} \log p_n) \)  
   (floor \(p_n\) at \(10^{-9}\) so a single zero order does not hard-zero the diagnostic).
4. **Brevity penalty:**  
   \( BP = 1 \) if \(|hyp| \ge |ref|\), else \( \exp(1 - |ref|/|hyp|) \)  
   lengths counted **without** spaces.
5. **Score:** \( 100 \times BP \times \exp(\overline{\log p_n}) \).

**Not** sacreBLEU (no `intl`/`13a` tokenization, no corpus smoothing as in mteval).  
Use **sacreBLEU / COMET / BLEURT** on `pairs_complete.jsonl` for paper numbers.

### Truncation effect on this proxy

Because BP penalizes short hyp and higher-n precisions collapse when the tail is missing, truncated Latin lines lose **~15–30 points** vs self-complete. CJK lines that already finished all anchors score **~100**.

---

## 2. Semantic truncation thresholds

**Coverage** = fraction of fixed meaning anchors present (tired/sister/bank/book/pictures/conversations/use…).

| Band | Coverage | Operational meaning | Training use |
|------|----------|---------------------|--------------|
| **T0_critical** | < 0.40 | Most slots missing | Do **not** use as positive labels |
| **T1_partial** | 0.40–0.69 | Setup OK, punchline often gone | OK for partial-gen aux loss only |
| **T2_near** | 0.70–0.89 | Minor tail loss | Prefer complete-up before distill |
| **T3_complete** | ≥ 0.90 | Practical semantic completion | Safe for supervised pairs |

**Char-ratio thresholds (complementary, density-aware):**

| Script class | Min trunc_ratio vs *that language's complete* | Notes |
|--------------|-----------------------------------------------|-------|
| Latin / Greek / Cyrillic | ≥ 0.95 | Else likely T0–T1 |
| Arabic / Devanagari | ≥ 0.92 | |
| JA / KO / ZH | ≥ 0.98 | Density already low; if incomplete, missing final 用/소용/役 |

**Combined gate for live fanout “complete” flag:**

```
semantic_band == T3_complete
AND trunc_ratio >= script_minimum
AND text ends with terminal punctuation (。．.?？!！»』'"…)
```

---

## 3. Multilingual perplexity metrics

### What we compute (no external LM required)

| Metric | Definition | Use |
|--------|------------|-----|
| **H_complete** | Char unigram entropy of complete text (bits/char) | Language complexity baseline |
| **CE_trunc** | Cross-entropy of truncated chars under complete unigram dist | How “in-distribution” the prefix is |
| **PPL_trunc** | \(2^{CE_{trunc}}\) | Same scale as LM perplexity, unigram only |
| **PPL_ratio** | PPL_trunc / PPL_complete_self | ≈1 for pure prefixes; spikes if trunc is noisy |

### How to use neural multilingual PPL (recommended next step)

| Model class | Metric | Notes |
|-------------|--------|-------|
| mT5 / mGPT / XLM-R MLM | Token PPL or pseudo-PPL | Compare trunc vs complete *same model* |
| NLLB / M2M teacher | Teacher CE on hyp | Distillation quality |
| COMET-QE / Kiwi | Reference-free quality | Orthogonal to PPL |

**Within-language only:** do not rank ZH vs DE by unigram PPL — alphabets differ. Rank **ΔPPL = PPL(trunc)−PPL(complete)** or coverage bands instead.

### Training tensor step suggestion

```
for each pair in batch:
  if semantic_band(trunc) <= T1:
    mask as incomplete; optional reconstruction loss toward complete
  else:
    standard CE / preference against complete ref
  log: proxy_bleu4, coverage, ppl_ratio  → early-stop fanout length policy
```