In 2023, "open source AI" mostly meant Stable Diffusion and a couple of LLaMA-derived chat models. By 2026, the open ecosystem covers nearly every modality and is genuinely competitive on the frontier — sometimes ahead, sometimes a quarter behind. This is a working list of what we'd actually use.
Why open weights matter in 2026
Three reasons people switch (or add) open-weights tools to their stack:
- Cost at scale: running your own inference on a $10,000 server beats $20,000/year of API spend at a few million requests/month.
- Privacy and control: regulated industries (legal, medical, defence, anything with data sovereignty rules) cannot send prompts to a third-party API.
- Fine-tuning: if you need a model that knows your codebase, your medical taxonomy, or your brand voice intimately, you need weights you can adapt.
For most individual users, frontier APIs are still the right default. Open-source is the right answer when one of those three factors dominates.
Large language models
Llama 3.x (Meta)
The 70B and 405B variants are the closest open weights have come to frontier quality. Llama 3 ships with permissive (though not fully open) licensing, runs on commodity GPUs at the smaller sizes, and has the largest fine-tuning ecosystem.
Practical setup: 8B / 70B on consumer hardware (single 24GB GPU runs 8B comfortably, 70B with quantisation); 405B needs serious infrastructure or a hosted endpoint (Together, Fireworks, Groq).
Mistral / Mixtral
The Mistral series remains the best for size-to-performance ratio. Mixtral 8x22B and the dense variants compete with much larger models. Apache 2.0 licensed (some variants), commercially usable.
DeepSeek
DeepSeek's models were the surprise of 2024-2025. The R1 reasoning model approaches o1-class performance with open weights and an order of magnitude cheaper training cost. Strong for code, mathematics, and structured reasoning.
Qwen (Alibaba)
The strongest multilingual open-weights family, especially for Chinese, Japanese, and Korean. Permissive license, mature ecosystem in China, increasing adoption in the West.
What we'd actually deploy
| Job | Model |
|---|---|
| General chat, mid-budget | Llama 3 70B or Mistral Large |
| Reasoning / code on a budget | DeepSeek Coder / R1 |
| Local on a single GPU | Llama 3 8B or Qwen 7B |
| Multilingual | Qwen 72B |
| Frontier-class, can pay for inference | Llama 3 405B (Together / Fireworks) |
Image generation
Flux (Black Forest Labs)
Already covered in our image generators guide. The current state of the art for open-weights image generation, with the Schnell variant under Apache 2.0 (commercial-friendly) and Dev for personal use.
Stable Diffusion 3.x
Stability AI's continuation of the SD line. Still widely used because of the LoRA / fine-tuning ecosystem, even though Flux often matches or beats it on raw quality.
SDXL Turbo / SD3 Turbo
For real-time / iterative use cases — sketch → image, animation frames, live coaching tools.
Speech and audio
Whisper (OpenAI, open source)
Still the default for speech-to-text. Whisper v3 / v3-turbo run locally with whisper.cpp. Quality is genuinely competitive with paid APIs for English.
XTTS / Coqui
Open-source text-to-speech with voice cloning. Lower quality ceiling than ElevenLabs but free and local.
Bark / Tortoise
Older but still useful for specific styles; superseded for most use cases.
Vector databases and RAG infrastructure
- Qdrant — fast, lean, written in Rust, great default for self-hosted vector DBs.
- Weaviate — slightly heavier but with built-in modules for hybrid search.
- Chroma — the easiest to start with for prototyping; not as scaling-friendly.
- pgvector — if you already have Postgres, you have a vector DB. Often the right answer for small/medium projects.
- Milvus — for serious scale.
Agentic frameworks
- LangChain — the most-used but increasingly criticised for abstraction overhead.
- LlamaIndex — better-focused on retrieval-augmented generation specifically.
- Haystack — by deepset, production-oriented, less hype.
- DSPy — the most academically rigorous; treats prompts as compilable programs.
- Smol Agents / Open Interpreter / Aider — simpler agents that you can read end-to-end.
Increasingly, people are skipping the frameworks and writing direct API calls or thin wrappers — for production, the abstractions add complexity without much benefit at the scales most teams operate.
Local inference runtimes
- llama.cpp / ollama — the easiest path to running an LLM on a Mac or a Linux box.
- vLLM — the highest-throughput open inference server for GPU servers.
- TGI (Text Generation Inference) by Hugging Face — production-grade alternative.
- ExLlamaV2 / TabbyAPI — for enthusiasts squeezing the last bit of performance out of consumer GPUs.
Hosting open-weights models in production
You usually don't want to run your own GPUs. The hosted-open-weights tier is its own category:
- Together AI — broad model selection, fair pricing.
- Fireworks — focus on low latency.
- Groq — astonishingly fast inference for supported models.
- Anyscale / Modal / RunPod — for serverless GPU when you need a custom model.
- HuggingFace Inference Endpoints — convenient but pricier; good for niche models.
For most production use cases, Together or Fireworks is the right starting point — you get open-weights flexibility with API-level convenience.
What we'd actually pick if starting from zero
- Llama 3 70B via Together / Fireworks for general-purpose generation.
- DeepSeek Coder for code-heavy work.
- Flux Schnell for images (or Flux Pro via API for quality).
- Whisper v3 local for transcription.
- Postgres + pgvector for RAG, until you outgrow it.
- Direct API calls + thin Python wrappers, skip the framework abstractions.
The open ecosystem genuinely competes now. Use it where cost, privacy, or control matter — and use frontier APIs where the marginal capability is worth the marginal cost.