add support for a separate vision model (small) to add eyes #71
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
a gemma 4b or qwen vl 4b model with temperature 0 and thinking off that is used to describe image content so the deepseek model can then use that to reason over the image. if the deepseek model needs something from the image, it is asking the small model to do that as deterministic as possible.
This requires probably a separate use of mlx directly to run that model the normal way, not a custom inference engine as the main model uses, this is just to add eyes to the better model.
If I go for the gemma 4b model, I could do the same with audio attachments to listen into them, despite deepseek also having no audio understanding. Also video is another viable input. Also images give more information from gemma, because it is. vision understanding that goes further than normal image descriptions. but gemma might require more recent mlx versions or require me to implement a custom inferencer.
The model integration would be using internal tools, not a chat integration, those model runs are one-shot invocations, so no context handling - the only context is the actual file.
One important thing might be the need for offloading of expert banks from ds4, so that I can bring in the vision model - can this be done? this would sequentializie model use, but that is the case anyway, because ds4 has to wait for the vision model description anyway, so during that time the experts could be put away and then reloaded - the general expert unloading/loading is already part of the ds4 code based. Reason for this: tight memory situations.
The handling could even be done by the agent itself, doing the vision model run right before handing the chat enriched with the visual description to the reconstructed ds4 model. this will add some overhead to the timing, but could allow the ds4 context staying high.
If we go with the temporary offloading of experts to run vision, the server itself would probably have to be disabled, because it will just not work during that time - or we can just accept long timeouts there.