# Changelog

## Score

- CAI 71 → 69 (-2.4)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

## Lenses

- Code Health 91 → 89 (-1.7)
- Architecture 97 → 98 (+0.1)
- Maturity 69 → 70 (+0.9)
- Readiness 66 → 66 (+0.1)
- Security 71 → 63 (-7.8)
- Performance 91 → 91 (+0.0)

## Resolved (81)

- Change coupling: wordlevel.rs ↔ wordpiece.rs (tokenizers/tk-train/src/trainers/wordlevel.rs)
- Coverage not measured — JavaScript/TypeScript suite
- Duplicated block (10 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10 lines × 5) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- Duplicated block (10–11 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10–11 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10–11 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10–11 lines × 3) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (10–15 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (11 lines × 3) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (11–12 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (11–12 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (11–12 lines × 3) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (12–13 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (12–17 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (13 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (15 lines × 2) (tokenizers/tk-encode/src/pre_tokenizers/unicode_scripts/scripts.rs)
- Duplicated block (6 lines × 2) (tokenizers/bitcannon/src/models/cl100k.rs)
- Duplicated block (6 lines × 2) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- …and 61 more

## New (50)

- Duplicated block (12 lines × 5) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- Duplicated block (14 lines × 2) (tokenizers/tk-train/src/trainers/unigram.rs)
- Duplicated block (15 lines × 3) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- Duplicated block (6 lines × 2) (tokenizers/bitcannon/src/models/cl100k.rs)
- Duplicated block (6 lines × 2) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- Duplicated block (7 lines × 2) (tokenizers/tk-encode/src/models/unigram/lattice.rs)
- Edited copies of one member (3 members, 12+ corresponding lines) (tokenizers/tk-encode/src/pre_tokenizers/bert.rs)
- Edited copies of one member (3 members, 13+ corresponding lines) (tokenizers/tk-train/src/trainers/bpe/mod.rs)
- Edited copies of one member (3 members, 19+ corresponding lines) (tokenizers/tk-encode/src/tokenizer/pattern.rs)
- Edited copies of one member (4 members, 11+ corresponding lines) (tokenizers/tk-encode/src/tokenizer/pipeline/mod.rs)
- Edited copy of a member (1 corresponding lines) (bindings/python/src/components/decoders.rs)
- Edited copy of a member (10 corresponding lines) (tokenizers/tk-encode/src/vocab/bucket_vocab_store.rs)
- Edited copy of a member (11 corresponding lines) (tokenizers/bitcannon/src/simd/x86.rs)
- Edited copy of a member (12 corresponding lines) (tokenizers/bitcannon/src/classify/avx.rs)
- Edited copy of a member (12 corresponding lines) (tokenizers/tk-encode/src/utils/unicode.rs)
- Edited copy of a member (12 corresponding lines) (tokenizers/tk-train/src/trainers/mod.rs)
- Edited copy of a member (14 corresponding lines) (tokenizers/tk-encode/src/models/wordlevel/mod.rs)
- Edited copy of a member (15 corresponding lines) (bindings/node/src/pipeline.rs)
- Edited copy of a member (19 corresponding lines) (tokenizers/bitcannon/src/classify/wasm.rs)
- Edited copy of a member (19 corresponding lines) (tokenizers/bitcannon/src/literal.rs)
- …and 30 more

## Changes since last survey

- 33 commits — 22 feature/other, 11 fixes

## By area

- tokenizers/tk-encode — 12 commits
- bindings/python — 10 commits
- .github/workflows — 5 commits
- (root) — 2 commits
- tokenizers/bitcannon — 2 commits
- bindings/node — 1 commit
- tokenizers/tk-convert — 1 commit

## Notable commits

- fix: Fix Kimi pretokenization for overlapping Han categories (#2567)
- fix: fix(added-vocab): fix panic in single_word check (#2666)
- fix: fix(added-vocab): use splitmix64 as hash function for Mphf (#2667)
- fix: fix(bpe): fuse unk tokens on models with affixes (#2676)
- fix: fix(byte-level): drop unknown bytes + warn (#2550)
- fix: fix(convert): canonicalize singleton Metaspace sequences (#2591)
- fix: fix(parallelism): don't build a pool when the lock is contended (#2633)
- fix: fix(parallelism): register the fork handler with `Once` (#2632)
- fix: fix(tk-encode): update ptr_hash to 2.1.3, fix forked Tokenizer build (#2581)
- fix: fix(unigram): drop long lattice paths iteratively (#2658)
- fix: fix(wordpiece): encode vocab tokens that contain null byte (#2661)
- change: Load byte-fallback BPE vocabs that lack some <0xNN> codes (#2461)
- change: Mark generated tokenizers.pyi as linguist-generated (#2606)
- change: chore(deps): bump the actions group across 1 directory with 6 updates (#2659)
- change: chore(deps): bump the actions group with 5 updates (#2587)
- change: chore(python): drop support for python 3.10 (#2668)
- change: chroe: bump deps, centralize audit allowlist (#2609)
- change: ci(stale): never auto-close PRs (#2621)
- change: feat(composition): ComponentSequence (#2577)
- change: feat(composition): TokenizerSpec: JSON config as source of truth (#2562)
- …and 13 more
