Apply Qwen NFC normalization before token splitting

This commit is contained in:
Georg Bauer committed 2026-10-04 17:14:00 +02:00
1 parent 5129c450ee
commit 6507a494bc
36 files changed
+1831 -39

No files matched your search

+5
View File
@@ -0,0 +1,5 @@
Compiling ds4-server v0.1.0 (/Users/gb/Projects/DS4Server)
Checking unicode-normalization-alignments v0.1.12
Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.51s
warning: the following packages contain code that will be rejected by a future version of Rust: block v0.1.6
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`