嵌入模型更省

Papers arxiv:2608.12875 Copy markdown The Embedder's Dilemma: LLMs Are Better, but at What Cost?

Published on Aug 13 · Submitted by Niklas Muennighoff on Aug 21 Upvote 11 +3 Authors: Adnan El Assadi ,Niklas Muennighoff ,Jinhyuk Lee Abstract Large language models and dedicated embedding models achieve nearly identical aggregate performance across diverse tasks, but embedding models are far cheaper and faster, supporting a division of labor by task type.

Generated by thinkingmachines/Inkling-Small Should you replace your text-embedding pipeline with a large language model?

We answer this with a controlled, cost-aware comparison of ten LLMs across six families and 26 embedding models (118M to 14B parameters) on 37 tasks spanning classification, semantic textual similarity (STS), clustering, pair classification, and retrieval.

In aggregate the two paradigms are effectively tied: the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (77.2) differ by 0.4 points.

Their strengths differ by task: LLMs lead on reasoning-heavy retrieval, embedding models lead on classification, and the two match on clustering, STS, and pair classification.Reaching that parity is expensive.An LLM costs up to 1,431x more than an embedding model of comparable quality (USD 154 vs.

USD 0.11 per benchmark pass), and the open LLMs tested process tokens 2.5 to 736x more slowly on the same GPU.Reasoning tokens account for 28 to 81% of LLM inference cost; lower reasoning budgets preserve or improve retrieval quality for most models in our ablation.

The Pareto frontier contains the leading embedding models and one LLM, Gemini 3.1 Pro.These results support a division of labour: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning-intensive retrieval.

Our code, datasets, and results are publicly available at https://github.com/embeddings-benchmark/embedders-dilemma.View arXiv page View PDF Add to collection Community Muennighoff Paper submitter 2 days ago Embeddings vs LLMs ...

Reply librarian-bot 2 days ago This is an automated message from the Librarian Bot.I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders (2026) DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillation (2026) Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking (2026) Can Frontier LLMs Match Natively Multimodal Embeddings?

A Comparison on Hard-Negative Text-to-Image Retrieval (2026) Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrieval (2026) Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders (2026) GEM: A Generative Embedding Model Bridging Reasoning and Retrieval (2026) Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend Reply O96a 1 day ago The gap on a table is noise once you multiply it by a million queries.

The real decision is cost per query at scale, and that's where a dedicated embedding model still wins by an order of magnitude on the GPU bill.I'd spend that saved budget on reranking or a better retriever instead — both move the needle more than the embedder gap.

The dilemma only bites if your task sits right on the quality cliff, and most pipelines don't.Until then, the math does the deciding.Reply pszemraj about 20 hours ago Do you plan to run this for OpenAI and anthropic models, which are also on openrouter?

Would love to hear if and how this changes the Pareto frontier Reply Muennighoff Paper submitter about 9 hours ago @pszemraj i think its a bit expensive to run so not planned atm i think; but happy to help if sb wants to run it!

Reply EditPreview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.Tap or paste here to upload images Comment · Sign up or log in to comment Upvote 11 Get this paper in your agent: hf papers read 2608.12875 Don't have the latest CLI?curl -LsSf https://hf.

co/cli/install.sh | bash Models citing this paper 0 No model linking this paper Cite arxiv.org/abs/2608.12875 in a model README.md to link it from this page.Datasets citing this paper 0 No dataset linking this paper Cite arxiv.org/abs/2608.12875 in a dataset README.md to link it from this page.

Spaces citing this paper 0 No Space linking this paper Cite arxiv.org/abs/2608.12875 in a Space README.md to link it from this page.Collections including this paper 1