Apply Qwen NFC normalization before token splitting
This commit is contained in:
1 parent
5129c450ee
commit
6507a494bc
36 files changed
+1831
-39
No files matched your search
@@ -0,0 +1,5 @@
|
||||
Compiling ds4-server v0.1.0 (/Users/gb/Projects/DS4Server)
|
||||
Checking unicode-normalization-alignments v0.1.12
|
||||
Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.51s
|
||||
warning: the following packages contain code that will be rejected by a future version of Rust: block v0.1.6
|
||||
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
|
||||
Reference in new issue
Block a user