Apply Qwen NFC normalization before token splitting
This commit is contained in:
1 parent
5129c450ee
commit
6507a494bc
36 files changed
+1831
-39
No files matched your search
@@ -0,0 +1,4 @@
|
||||
[transformers] You are using a model of type `qwen4_exp` to instantiate a model of type ``. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating.
|
||||
[tokenizer] restored the pre-tokenizer regex from tokenizer.json: the loaded tokenizer split combining marks off their letters (Devanagari, Thai, vowelled Arabic tokenized 25-90% longer than the model was trained on)
|
||||
[mtplx] chat turn segmentation {"enabled": true, "reason": "structure holds and all 12 probe renders give the single-call ids when cut", "tokenizer": "TokenizerWrapper"}
|
||||
{"cases": 13281, "requests": 8, "scalar_sha256": "1905b5f60f629850aa35dfc8ccb11fa6ae4d1fb0c79efd6774d36ca8e3e64eab", "tokenizer_sha256": "0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3", "tokenizers": "0.22.2"}
|
||||
Reference in new issue
Block a user