llmops

BerriAI/litellm

LiteLLM moves to a Rust core for a leaner AI gateway — LiteLLM now ships a Rust core under its Python SDK, aiming to be a faster, lighter self-hosted AI gateway. It unifies 100+ LLM APIs (Bedrock, Azure, OpenAI, Anthropic, VertexAI, vLLM, Nvidia NIM) in OpenAI format, adding cost tracking, guardrails, load balancing and logging for teams running LLMOps at scale.

The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]

langfuse/langfuse

Open-source tracing and evals for LLM apps and agents — Langfuse is a self-hostable platform to trace, evaluate, and debug LLM applications, with integrations for LangChain, LlamaIndex, and AutoGen plus a prompt playground and management tools. Backed by Y Combinator, it suits teams needing open, self-hosted observability instead of closed SaaS-only tools.

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

ComposioHQ/composio

Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench to help you build AI agents that turn intent into action.

promptfoo/promptfoo

Promptfoo's LLM Evals and Red Teaming, Now Backed by OpenAI — Promptfoo is a CLI and library for evaluating and red-teaming LLM apps, agents, and RAGs, with declarative configs that plug into CI/CD. It recently became part of OpenAI while staying open source and MIT licensed, and is used by OpenAI and Anthropic to catch vulnerabilities before shipping.

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

MakazhanAlpamys/Soup

One-command LLM fine-tuning that fits an 8B model into 4GB VRAM — Soup is a CLI that fine-tunes and post-trains LLMs from a single YAML config, using layer streaming to run 8B-parameter training on a consumer laptop GPU with just 4GB VRAM. It targets developers who want LoRA/QLoRA/DPO/SFT workflows without SSH setup or config complexity, with exports to GGUF and Ollama.

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

maximhq/bifrost

A zero-config gateway unifying 23+ LLM providers behind one API — Bifrost is a Go-based AI gateway giving one OpenAI-compatible API across 23+ providers, with automatic failover, load balancing, semantic caching, virtual keys and budgets. It deploys in seconds via npx or Docker, includes a web UI for config and monitoring, and adds under 200µs overhead at 5k RPS—aimed at teams building reliable, multi-provider AI apps and agents.

Open-source, high-performance AI gateway that works as your LLM gateway, MCP gateway and agent gateway. One OpenAI-compatible API for 10,000+ models, with failover, load balancing, semantic caching, virtual keys, budgets and MCP Code Mode. Works with any SDK or agent and adds under 200µs of overhead at 5k RPS.

lmnr-ai/lmnr

Observability for AI agents, built on OpenTelemetry tracing — Laminar (YC S24) traces agent runs with one-line OpenTelemetry SDK support for Vercel AI SDK, LangChain, Browser Use, Stagehand and major LLM providers. It adds plain-English 'signals' that Slack-alert on bad behavior, an evals SDK/CLI, SQL-queryable traces via MCP/CLI, and custom dashboards—useful for teams debugging and monitoring agent pipelines.

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

jwuthri-cloud/Tracely-ai

Turn real agent failures into CI gates, not hand-written test sets — Tracely grades live AI agent traces, clusters failures into issues, and freezes the actual bad runs into hermetic regression tests that replay on every PR, blocking regressions automatically. Self-hostable stack (API, worker, UI, Postgres, ClickHouse, Redis, MinIO) aimed at teams tired of hand-authoring eval datasets.

Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.

mozilla-ai/otari

A self-hosted gateway that unifies 40+ LLM providers behind one API — Otari is a Mozilla AI project that lets you run your own OpenAI/Anthropic-compatible gateway in front of 40+ model providers. It adds virtual API keys, pre-spend budget checks, usage tracking, and local routing policies for failover, useful for teams wanting to own their AI infrastructure instead of relying on a hosted proxy.

Open-source, OpenAI-compatible LLM gateway you run yourself. One endpoint for 40+ providers, with virtual keys, budgets, and usage tracking.

respanai/respan-gateway

A Single Rust Binary That Speaks OpenAI, Anthropic and Gemini — Respan Gateway lets any OpenAI, Anthropic or Gemini SDK call any model by translating between their APIs on the fly. It covers chat, realtime voice, images, embeddings and video, adds routing/fallback/hedging, fronts MCP servers, and ships as a single extensible Rust binary.

One endpoint for every LLM vendor. Any SDK, any model, any tool. One Rust binary.