mlops

10 repositories

apache/airflow

Apache Airflow: Workflow Orchestration as Python DAGs — Airflow lets teams define, schedule, and monitor data pipelines as Python code, with dependencies expressed as DAGs. It's widely used for ETL, ML, and data integration tasks across many providers and backends. This is the mature, actively maintained Apache project behind countless production data pipelines.

PythonAttention 36
★47,564
qdrant/qdrant

Rust-built vector search engine for production AI workloads — Qdrant is a Rust-based vector database with extended filtering support, built for semantic search, recommendations, and neural-network matching at scale. It offers a production-ready API and a managed cloud option, aimed at developers building AI apps that need fast, reliable similarity search under heavy load.

Rust🇪🇺 EuropeAttention 39
★35,638
mlflow/mlflow

MLflow expands from ML tracking into a full agent/LLM engineering platform — MLflow now bills itself as an AI engineering platform covering agents, LLMs, and ML models, not just experiment tracking. It adds production-grade tracing, evaluation, prompt management, prompt optimization, and an AI Gateway for cost and access control, aimed at teams running LLM apps in production.

Python🇺🇸 North AmericaAttention 39
★28,902
microsoft/agent-lightning

A lightweight RL framework to train AI agents using real harnesses — Agent Lightning v1.0 is a ~3,500-line framework that trains agents via reinforcement learning without modifying existing agent harnesses, tools, or environments, using a proxy layer and native Kubernetes job support. It targets MLOps and RL researchers, with examples showing SWE-bench Verified gains from fine-tuning Qwen models on limited training samples.

Python🇺🇸 North AmericaAttention 31
★18,983
argoproj/argo-workflows

Container-native workflow engine, now with a Python-friendly path via Hera — Argo Workflows runs multi-step jobs as Kubernetes-native DAGs, with every step a container. It's a CNCF graduated project widely used for ML pipelines, data/batch processing, CI/CD, and infrastructure automation, and the Hera Python SDK makes it accessible to Python users too.

GoAttention 26
★17,107
lance-format/lance

A lakehouse file format built for multimodal AI workloads — Lance is an open lakehouse format combining vector search, BM25 full-text search, and SQL analytics on the same dataset, with fast random access and data versioning. It converts from Parquet in two lines of code and works with Pandas, DuckDB, Polars, PyArrow, Ray, and Spark, targeting ML training and multimodal data storage (images, video, audio, embeddings).

Rust🇺🇸 North AmericaAttention 54
★7,373
kserve/kserve

CNCF platform unifying generative and predictive AI serving on K8s — KServe is a CNCF incubating project that unifies LLM and traditional ML model serving on Kubernetes. It adds vLLM/llm-d backends, OpenAI-compatible APIs, KV cache offloading and model caching for generative workloads, alongside multi-framework support, canary rollouts and explainability for predictive AI. Useful for MLOps teams running both LLMs and classic models at scale.

GoAttention 37
★6,247
higgsfield-ai/higgsfield

Multi-node LLM training with a GitHub-Actions-style workflow — Higgsfield is an open-source GPU orchestration layer and ML framework for training billion- to trillion-parameter models like LLaMA. It handles node allocation, ZeRO-3/FSDP sharding, experiment queuing, and CI-style integration with GitHub Actions, letting developers launch distributed training with minimal boilerplate code.

Jupyter Notebook🇺🇸 North America
★6,228
vllm-project/vllm-ascend

vLLM Plugin Brings LLM Serving to Huawei Ascend NPUs — This community-maintained plugin lets vLLM run inference on Huawei Ascend hardware instead of GPUs, following vLLM's plugin architecture. It's for teams deploying LLMs on Ascend NPUs who want vLLM's serving features without vendor lock-in to NVIDIA. Active releases (v0.30.0 as of this writing) show ongoing support for new vLLM versions.

PythonAttention 51
★3,122
SemiAnalysisAI/InferenceX

A standardized benchmark suite for GPU inference stacks — InferenceX from SemiAnalysis bundles end-to-end serving, networking/collective, and kernel-level benchmarks across vLLM, SGLang, and TensorRT-LLM on NVIDIA and AMD GPUs. It targets researchers and MLOps teams comparing inference hardware and software, with results published on a public performance dashboard and new agentic long-context benchmarks like AgentX.

Python🇺🇸 North AmericaAttention 51
★2,069