AI chat hides oMLX provider error and fails for model without chat template #39
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Reproduction: use online endpoint http://127.0.0.1:9000/v1 with model mlx-community--gemma-4-12B-8bit and send a chat message. bDS shows only a parse error for HTTP 400 at /v1/chat/completions. A minimal OpenAI-compatible request outside bDS returns the same 400 from oMLX with the actionable provider detail that tokenizer.chat_template is not set and no template argument was passed. MLX Server is not involved and must not be used. Evaluate both: determine whether this model can be served for chat by oMLX with a correct chat template/model configuration or whether bDS should reject it as non-chat-capable; preserve and surface provider response bodies for non-2xx chat responses instead of discarding them through error_for_status, with localized actionable UI treatment. Acceptance: root cause and supported configuration documented; chat succeeds for a supported oMLX model/configuration or the unsupported model is rejected with clear guidance; provider error bodies retained and tested; no MLX Server testing.
Evaluation complete without MLX Server. oMLX model catalog contains mlx-community--Qwen3.5-9B-4bit. A minimal /v1/chat/completions request with that model returned HTTP 200, and the rebuilt bDS GUI returned OK to the same prompt. The same minimal request with mlx-community--gemma-4-12B-8bit returned HTTP 400 because tokenizer.chat_template is not set, proving the chat failure is specific to that served model artifact/configuration rather than bDS endpoint persistence. bDS also discarded the oMLX response body through error_for_status. That is fixed: non-2xx chat responses now retain the provider message, and the UI presents it with a localized provider-error prefix. The packaged German UI now shows the full chat-template explanation for Gemma, while Qwen chat succeeds. Regression tests cover provider body preservation and all locale keys. cargo test --workspace, cargo build --workspace, formatting, and diff checks pass. The original Gemma endpoint, default model, and title model settings were restored after testing.