vllm

12 repositories

datalab-to/surya

OCR, layout analysis, reading order, table recognition in 90+ languages

Python🇺🇸 North AmericaAttention 25
★21,606
datalab-to/chandra

Chandra OCR 2 turns documents into structured HTML, Markdown or JSON — Chandra OCR 2 is an open-weights OCR model that converts images and PDFs into structured HTML/Markdown/JSON while preserving layout, handling complex tables, forms and handwriting. Useful for developers digitizing documents who need layout-aware output rather than plain text; a managed, higher-accuracy version with batch processing is also offered by Datalab.

Python🇺🇸 North AmericaAttention 35
★12,738
ai-dynamo/dynamo

An orchestration layer that turns single-node LLM engines into a cluster — Dynamo coordinates SGLang, TensorRT-LLM, and vLLM across multiple nodes rather than replacing them, adding disaggregated serving, smart routing, multi-tier KV caching, and autoscaling. Built in Rust with a Python extension layer, it targets teams running datacenter-scale LLM, reasoning, multimodal, and video generation inference.

Rust🇺🇸 North AmericaAttention 47
★8,530
vllm-project/semantic-router

A programmable routing layer that picks the right LLM per call — vLLM Semantic Router gives agent harnesses one API across many models and backends, choosing or combining them per request based on policy for quality, latency, cost, and location. Useful for teams running mixed local, private, and cloud inference who want a single decision layer instead of hardcoding model choices.

GoAttention 48
★6,511
mostlygeek/llama-swap

A lightweight proxy that hot-swaps local AI models on the fly — llama-swap is a single Go binary that sits in front of llama.cpp, vllm, stable-diffusion.cpp and similar servers, switching models on demand via one config file. It supports both OpenAI and Anthropic API endpoints, including chat, embeddings, images, audio and rerank, making it useful for anyone running multiple local models without juggling separate server instances.

Go🇺🇸 North AmericaAttention 39
★6,274
kserve/kserve

CNCF platform unifying generative and predictive AI serving on K8s — KServe is a CNCF incubating project that unifies LLM and traditional ML model serving on Kubernetes. It adds vLLM/llm-d backends, OpenAI-compatible APIs, KV cache offloading and model caching for generative workloads, alongside multi-framework support, canary rollouts and explainability for predictive AI. Useful for MLOps teams running both LLMs and classic models at scale.

GoAttention 37
★6,247
vllm-project/vllm-ascend

vLLM Plugin Brings LLM Serving to Huawei Ascend NPUs — This community-maintained plugin lets vLLM run inference on Huawei Ascend hardware instead of GPUs, following vLLM's plugin architecture. It's for teams deploying LLMs on Ascend NPUs who want vLLM's serving features without vendor lock-in to NVIDIA. Active releases (v0.30.0 as of this writing) show ongoing support for new vLLM versions.

PythonAttention 51
★3,122
SemiAnalysisAI/InferenceX

A standardized benchmark suite for GPU inference stacks — InferenceX from SemiAnalysis bundles end-to-end serving, networking/collective, and kernel-level benchmarks across vLLM, SGLang, and TensorRT-LLM on NVIDIA and AMD GPUs. It targets researchers and MLOps teams comparing inference hardware and software, with results published on a public performance dashboard and new agentic long-context benchmarks like AgentX.

Python🇺🇸 North AmericaAttention 51
★2,069
eagle-hq/EAGLE-4

Official Inference Code of EAGLE-4

PythonAttention 0
★1,214
smg-project/smg

A Rust gateway that unifies vLLM, TRT-LLM, SGLang and more — SMG is a Rust-built gateway offering full OpenAI and Anthropic API compatibility across vLLM, TRT-LLM, SGLang, TokenSpeed, Gemini and others. It adds KV cache-aware routing, a gRPC pipeline, tokenization caching, WASM plugins and multi-tenant auth, aimed at teams running large-scale, multi-engine LLM deployments.

Rust🇺🇸 North AmericaAttention 43
★733
adventists-ai/sophon-realtime

Open full-duplex voice models that hear your tone and speak with one. One GPU with vLLM-Omni, or fully offline on a Mac with MLX. Zero-shot voice cloning, OpenAI Realtime / GPT-Live compatible, free hosted API trial. Apache-2.0.

Python🇨🇳 Greater ChinaAttention 86
★482
huawei-csl/MACaQ

Huawei's model-agnostic quantization toolkit beats GPTQ-class methods on speed — MACaQ quantizes text, image, video and audio models for llama.cpp, vLLM, Diffusers and PyTorch, backed by a published paper. It reportedly quantizes a 122B model in 64 minutes on 32.5GB memory, and beats MR-GPTQ/DuQuant++ perplexity or matches it 11x faster on Llama-3-8B. Useful for anyone running local inference across multiple backends and modalities.

C++🇪🇺 EuropeAttention 88
★378