benchmark
14 repositories
A statistics-driven CLI benchmarking tool for comparing commands — hyperfine runs shell commands repeatedly, auto-sizing the number of runs, with warmup passes, cache-clearing hooks, and outlier detection to flag interference. It compares multiple commands against a reference, supports parameterized runs, and exports to CSV, JSON, Markdown, or AsciiDoc. Useful for anyone needing reliable, reproducible command-line performance comparisons.
CI-Integrated Quality and Security Gates for AI Agent Skills — SkillEvaluator checks Agent Skills-spec folders with quality gates, SkillSpector security scanning, and live before/after comparisons to measure how a skill actually changes agent behavior, not just static review. It plugs into GitHub or GitLab CI via a CLI or ready templates, part of NVIDIA's Verified Skills pipeline, useful for teams shipping skills for Claude Code or Codex-style agents.
CPU-X is a Free software that gathers information on CPU, motherboard and more
Uber's open-sourced security stack for enterprise AI agents — ADR is Uber's production-deployed security system for agentic AI, now open-sourced with its MLSys 2026 paper. It covers discovery of AI tools, cross-platform observability for coding agents like Cursor and Claude Code, a 300+ task benchmark against 17 attack techniques, and a two-tier threat detector—useful for security teams managing enterprise AI agent risk.
A Reproducible, Multi-Engine Benchmark for Analytical Databases — ClickBench tests dozens of OLAP, OLTP, serverless, and specialized databases (including DuckDB, BigQuery, Snowflake, Doris, DataFusion, chDB) against a real anonymized web-analytics dataset. Results are reproducible in ~20 minutes via documented shell scripts, making it useful for anyone comparing analytical database performance before adopting one.
A modern iperf3 alternative with a live TUI, multi-client server, and QUIC support. Built in Rust.
Can AI agents rebuild websites through browser exploration?
:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.
Benchmarking LLM agents against a real Grafana observability stack — o11y-bench evaluates LLM agents on observability and SRE tasks, running them against live Prometheus, Loki, and Tempo via the Harbor framework. It grades results with deterministic checks plus rubric criteria, producing HTML reports and a leaderboard, useful for teams testing agentic AI on real-world monitoring workflows.
Benchmarking LLMs by making them load a dishwasher — DishwasherBench turns dishwasher loading into an LLM benchmark, using rigid-body physics and simulated wash chemistry on an exact model of a Bosch SMV4HAX40G. Humans play the same game, called Spotless, in a browser, while models act through MCP, with every episode deterministic and replayable for verification.
A leaderboard that measures code bloat in AI coding patches — Parsimony parses patches already published for benchmarks like SWE-bench and DeepSWE, counting syntax units added and removed without running any code. It ranks models by net code added per solved task relative to peers, useful for researchers comparing AI coding agents' verbosity, not just solve rates.
Pitting Mo, Codex, Claude Code and Devin against real app bugs — This benchmark runs four AI QA agents against 20 product areas of Gitea, Metabase, Twenty and Saleor, grading their bug reports against 1,396 human-reviewed labels. Results include recall, precision and cost-per-correct-bug, with fixtures and data published on Hugging Face for reproducibility. Useful for teams evaluating AI browser-testing agents against a shared, labeled baseline.
A Blog-Driven Playground Testing How Shopping Agents Get Manipulated — Gramli.Framework collects experiments tied to blog posts, including WebMCP Checkout Trap, an Angular electronics store built to test how tool descriptions can manipulate an external shopping agent. Six scenarios (budget override, authority spoofing, workflow hijacking, etc.) pair an Angular storefront with a Playwright agent using Gemini or NVIDIA models. Useful for anyone studying prompt injection and agent manipulation risks in commerce tools.
Independent benchmark tests TypeSafe's Jev against Sonnet 5 on AITA verdicts — An independent, self-funded test of TypeSafe's new 'System One' model Jev against Sonnet 5, GPT-5 nano and local LLMs, predicting Reddit AITA verdicts on 770 posts. Reports Brier scores, latency and cost, and checks TypeSafe's speed claims—useful for anyone evaluating fast probability-only models versus full LLMs.