EleutherAI's harness for few-shot benchmarking of language models across hundreds of academic tasks. It standardises prompt formatting so results are comparable between models and backends.
Preview of the business's website.
Source: Official GitHub repository. Last checked 2026-09-18. Spot an error? Tell us.
Nearby and similar
A library for building LLM applications that also auto-tunes them, optimizing prompt text and few-shot demonstrations against a training set. It aims to make prompt tuning as routine as model training.
An observability and evaluation platform that turns production traces into datasets and prompt improvements, marketed for teams running self-improving agents.
An open-source workspace where engineers and subject-matter experts iterate on prompts and agent configurations side by side, with versioning, evaluation runs and observability.
A companion LangChain package focused on scoring agent trajectories, comparing the sequence of tool calls an agent made against an expected path.
An open-source red-teaming kit that fuzzes LLM endpoints with adversarial prompt datasets and multimodal attacks, producing a report of which defences failed.
An automatic evaluator for instruction-following models that uses an LLM judge to compare outputs against reference responses, with published leaderboards and length-controlled scoring.