A 'free' multi-modal 'embedding' model (already) included with your LLM: vLLM plugin + upstream PR

Disclaimer: I work for NuMind, the company behind this plugin. TLDR: Text/image (simili-)embeddings from the LLM you already have loaded :-) I've been working on making it easier to obtain embeddings from a LLM (preferably multi-modal) that is already being served by vLLM. The idea is simple (and not mine as we'll discuss): the model already computes internal representations (1 hidden state per decoder layer) of your prompt to predict the first output token. We expose the last hidden state vector through the chat API (compatible with the openai-api and openai client even though it's not standard) There are some interesting papers on this: PromptEOL, PromptReps and E5-V. To briefly summarize the E5-V work (which is the paper I based most of my work on): They use this prompt Summary above image in one word: And use the last hidden state (the hidden state of the last decoder block) as an embedding. And it works quite well. The intuition is that asking for a one token/word summary encourages the model to concentrate relevant information at that position. At NuMind, we build NuExtract3, an open-weight 4B model under Apache-2.0, for extracting structured data and converting an image/document into Markdown. We also have a NuExtract platform and API for using document extraction with a much larger (pro) model. (see my NuExtract 3 anouncement post for more detailed information). Since our model mainly outputs json (except in the Markdown mode), the prompt we use is {"short description": "string"} This plug-in could be useful for local RAG, semantic search, clustering documents or images, or training a small classifier on top of the vectors. Of course, a hidden state isn't automatically a good embedding. Quality depends on the model, prompt, and task, and a dedicated embedding model may be faster or better. The plugin just makes these experiments easier (and faster than the manual PyTorch + transformers set-up we used to determine these are viable embeddings) I'd really like to get this merged into vLLM: PR #57185. If you have any review/opinion on my code, know anyone that I could talk to to get it merged, it would be wonderful. I know it's kind of an edge case but I would still say it's useful. In the meantime, we have an Apache-2.0 plugin for vLLM. You opt in on a one-token, non-streaming chat request and get the vector in the response. Ordinary chat and streaming still work on the same server. The README has examples and the current support/validation limits. Would there be interest in me porting this workflow to llama.cpp? I know most people here use llama.cpp rather than vLLM so I might try to push a PR there too. No idea whether it's easier to have code accepted in vLLM or in llama.cpp. If you have any feedback, please share :-)Disclaimer: I work for NuMind, the company behind this plugin.TLDR: Text/image (simili-)embeddings from the LLM you already have loaded :-) I've been working on making it easier to obtain embeddings from a LLM (preferably multi-modal) that is already being served by vLLM. The idea is simple (and not mine as we'll discuss): the model already computes internal representations (1 hidden state per decoder layer) of your prompt to predict the first output token. We expose the last hidden state vector through the chat API (compatible with the openai-api and openai client even though it's not standard) There are some interesting papers on this: PromptEOL, PromptReps and E5-V. To briefly summarize the E5-V work (which is the paper I based most of my work on): They use this prompt Summary above image in one word And use the last hidden state (the hidden state of the last decoder block) as an embedding. It works fairly well. The intuition is that asking for a one token/word summary encourages the model to concentrate relevant information at that position. At NuMind, we build NuExtract3, an open-weight 4B model under Apache-2.0, for extracting structured data and converting an image/document into Markdown. We also have the NuExtract platform and API for using document extraction with a much larger (pro) model. (see my NuExtract 3 anouncement post for more detailed information). Since our model mainly outputs json (except in the Markdown mode), the prompt we use is {"short description": "string"} This plug-in could be useful for local RAG, semantic search, clustering documents or images, or training a small classifier on top of the vectors. Quality will depend on the model, prompt, and task, and a dedicated embedding model may be faster or better. The plugin just makes these experiments easier (and faster than the manual PyTorch + transformers set-up we used to determine these are viable embeddings) I'd really like to get this merged into vLLM: PR #57185. If you have any review/opinion on my code, know anyone that I could talk to to get it merged, it would be wonderful. I know it's kind of an edge case but I would still say it's useful. In the meantime, we have an Apache-2.0 plugin for vLLM. There is a extra argument to add, max_tokens=1 and the embeddings will be in the additional response arguments. Ordinary chat and streaming still work on the same server. The README has examples and the current support/validation limits. Would there be interest in me porting this workflow to llama.cpp? I know that's what most people here use so I might try to push a PR there too. No idea whether it's easier to have code accepted in vLLM or in llama.cpp (seems to be quite long for vLLM). If you have any feedback, please share :-) submitted by /u/Gailenstorm [link] [comments]

Source: r/LocalLLaMA | 2026-10-12

Related