llm-inference

gitleaks/gitleaks

Gitleaks enters maintenance mode, focus shifts to Betterleaks — Gitleaks scans git repos, files, and stdin for secrets like passwords and API keys using regex-based detection. The maintainer has declared it feature-complete, limiting future releases to security patches while development moves to a new project called Betterleaks. Useful for DevSecOps teams running CI/CD secret scanning today.

Find secrets with Gitleaks 🔑

ai-dynamo/dynamo

An orchestration layer that turns single-node LLM engines into a cluster — Dynamo coordinates SGLang, TensorRT-LLM, and vLLM across multiple nodes rather than replacing them, adding disaggregated serving, smart routing, multi-tier KV caching, and autoscaling. Built in Rust with a Python extension layer, it targets teams running datacenter-scale LLM, reasoning, multimodal, and video generation inference.

A Datacenter Scale Distributed Inference Serving Framework

kserve/kserve

CNCF platform unifying generative and predictive AI serving on K8s — KServe is a CNCF incubating project that unifies LLM and traditional ML model serving on Kubernetes. It adds vLLM/llm-d backends, OpenAI-compatible APIs, KV cache offloading and model caching for generative workloads, alongside multi-framework support, canary rollouts and explainability for predictive AI. Useful for MLOps teams running both LLMs and classic models at scale.

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

raullenchai/Rapid-MLX

A faster local inference server for coding agents on Apple Silicon — Rapid-MLX is an Apache 2.0, OpenAI/Anthropic-compatible inference server built on MLX, aimed at coding agents needing reliable tool calling. Benchmarks show it decoding faster than Apple's mlx-lm, oMLX, and LM Studio on the same Qwen weights, making it worth a look for local-LLM Mac users.

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

youssofal/MTPLX

Native MTP speculative decoding speeds up local LLMs on Apple Silicon — MTPLX runs Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on Macs using the models' own multi-token prediction heads for exact, batched speculative decoding. It reports roughly 2x decode speed over plain decoding, with a 27B model fitting on 16 GB Macs, via an OpenAI/Anthropic-compatible local server for tools like OpenCode.

The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

mw00/project-maya

Self-hosted inference stack for massive MoE models like GLM-5.3-Flash and DeepSeek-V4.1 — Project Maya lets you run huge mixture-of-experts models (up to 748B parameters) privately on your own GPUs, from a single card to 16, on NVIDIA or AMD, Linux or Windows. It bundles an inference engine, server, dashboard, OpenAI/Anthropic-compatible API, and quantization tools for self-hosters wanting frontier-scale LLMs without cloud dependence.

Project Maya - run GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE) and MiniMax-M3.1-Flash (428B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.

SemiAnalysisAI/InferenceX

A standardized benchmark suite for GPU inference stacks — InferenceX from SemiAnalysis bundles end-to-end serving, networking/collective, and kernel-level benchmarks across vLLM, SGLang, and TensorRT-LLM on NVIDIA and AMD GPUs. It targets researchers and MLOps teams comparing inference hardware and software, with results published on a public performance dashboard and new agentic long-context benchmarks like AgentX.

Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

AtomicBot-ai/Atomic-Chat

A Tauri-based desktop app for running open-weight LLMs offline — Atomic Chat is a desktop inference engine that runs open-weight models like Qwen, Gemma, DeepSeek and GPT-OSS fully offline, with Apple Silicon (MLX) and GGUF/llama.cpp support. Built with Tauri and Node.js, it targets users who want private, self-hosted AI chat and agent tooling without cloud dependency.

Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V

ashhart/TensorFold

A Zig binary that serves LLMs directly on Metal or CUDA — TensorFold is a Zig-based inference engine that runs language models on Apple Silicon or NVIDIA GPUs without needing Python or MLX. It reads checkpoints and runs Metal/CUDA kernels directly, serving models like Qwen3.5-2B via a simple CLI. Useful for developers wanting a lightweight, dependency-free local LLM server.

LLM Inference Engine for Metal, CUDA and Vulkan.

StayLameBro/backburner

Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable

eagle-hq/EAGLE-4

Official Inference Code of EAGLE-4

amurgola/LumaBrowser

Local AI desktop app: a Chromium browser with a GPU-aware llama.cpp runtime, OpenAI-compatible API, AI chat and agents, Stable Diffusion image/video, TTS and music, plus an MCP server so Claude Code, Codex and Cline can drive your browser. Windows, macOS, Linux.