AI Evaluation CompaniesGet a quote
Menu

AI Evaluation Companies

Evaluation companies test models and AI products for you: some send trained people to score outputs and attack the system, others sell software your team runs. Below is who does which, what they publish about price, and what to demand before you sign.

What an evaluation vendor should give you

Tick what your shortlist offers

Get a quote

What kinds of AI evaluation can you buy?

Four kinds cover most evaluation work. They differ in who does the work and how often you need it.

The four common types. More detail, with examples, on the types of AI evaluation page.
TypeThe question it answersUsual methodWho usually runs itHow often
Capability and benchmarksCan the model do the task, and how well against a fixed bar?Held-out test sets with reference answers or verifiersYour team on a platform, or an expert-data vendor building private setsEach model or prompt change that matters
Safety and red-teamingWhat can someone make it do that it should not?Adversarial prompting by people, plus automated attack toolsSpecialist red-team firms and managed data providersBefore launch and after major changes
Human preferenceWhich answer do real people judge better, and why?Side-by-side ratings against a rubricManaged rating services or participant marketplacesPer release, or when automated graders disagree
RegressionDid this release break what worked last week?A fixed suite run in CI, often with model gradersYour engineers on eval toolingEvery build or deploy

Types of AI evaluation, in detail

Should you buy an evaluation service or eval tooling?

It depends on whether your bottleneck is people or plumbing. Most teams end up with tooling for the daily checks and a service for the work that needs judgment.

A managed service

You buy people: trained raters, domain experts or attackers, plus the project management around them. The vendor writes or adapts rubrics, runs the work and hands back scored data.

  • Right for human preference studies, red-teaming and expert-graded capability tests.
  • Priced per project; none of the managed firms we list publishes rates.
  • Ask who the raters are and how disagreements were settled.

Eval tooling

You buy software: a place to store test sets, run them on each change, score outputs with code or model graders and trace production calls. Your engineers do the work.

  • Right for regression suites, CI gates and monitoring after launch.
  • Published prices, often with a free tier to start.
  • Ask about data retention, seat limits and what happens when you pass the plan's trace or score cap.

Which AI evaluation companies publish their prices?

The software vendors mostly do; the people-heavy services mostly do not. Here are twelve of the 20 we track, with what each one's own page said.

Last checked against each company's own page on 7 November 2026.

Published pricing as read from each company's own page. Not a quote.
CompanyWhat it sells for evaluationDeliveryPublished pricing
BraintrustRuns evals, scores outputs and traces production LLM appsSoftware you runStarter $0 a month (1 GB processed data, 10k scores, 14-day retention); Pro $249 a month (5 GB, 50k scores, 30-day retention); Enterprise custom
LangSmith (LangChain)Tracing, testing and monitoring for LLM agentsSoftware you runDeveloper: free per seat, 5k base traces monthly. Plus: $39 per seat monthly, 10k base traces, then pay as you go. Enterprise: custom
Confident AI (DeepEval)Cloud platform built on DeepEval, its Apache-2.0 open-source eval framework for Python and TypeScriptSoftware you runFree tier; paid plan prices not shown on the page read
GiskardAgent testing and continuous red-teaming, with an open-source Python librarySoftware you runOpen-source edition free (runs locally); Professional plan price not shown
HumanSignal (Label Studio)Label Studio for human review and scoring of model outputs; open-source Community EditionSoftware you runStarter Cloud: $99 monthly covers 12 users, each extra user $49 monthly. Enterprise: custom
Kili TechnologyAnnotation, evaluation and post-training data platform, with optional specialist annotatorsPlatform and servicesFree trial $0 (2 to 5 seats, 100-asset limit); Grow and Enterprise custom
LabelboxLabeling platform billed in Labelbox Units (LBUs), with labeling services as an add-onPlatform and services20 LBU per data row for live multimodal and LLM work; 500 free LBU a month on free accounts; no dollar price per LBU published
ProlificRecruits paid participants for research and human data, including human ratings of AI outputParticipant marketplaceParticipant minimum $8 an hour, recommended $12 an hour; platform fee charged on top (rate not stated on the page read)
Scale AIData engine for training data, plus model evaluationPlatform and servicesNot published
Surge AIHuman evaluation, rubrics and verifiers, RLHF and SFT data; says it works in over 70 languagesManaged serviceNot published
TolokaAuto-verifiable evaluation tasks, preference data and red-teaming; based in AmsterdamManaged serviceNot published
Gray Swan AIShade for red-team testing, Cygnal for input and output filtering, and Arena, a red-teaming network it says has 23,000+ participantsPlatform and servicesNot published

Disclosure: vendors in this table can pay us a fee when an introduction we made ends in a signed deal. The listing, its order and its wording are never for sale.

All 20 vendors and tools, with security claims and the terms changelog

How is AI evaluation priced?

Five pricing models turn up again and again. Knowing which one a vendor uses tells you what will make the bill grow.

Plan tiers

A monthly fee with caps on data processed, scores and retention. Braintrust: $0 Starter, $249 a month Pro, custom Enterprise.

What grows the bill: passing a cap.

Per seat

A fee per user, sometimes with usage bundled. LangSmith's Plus tier: $39 per seat monthly, 10k base traces included. HumanSignal Starter Cloud: $99 monthly, covering 12 users.

What grows the bill: headcount.

Usage units

A count of units per data row or task. Labelbox bills in LBUs (20 per row for live multimodal and LLM work) but does not publish a dollar price per LBU.

What grows the bill: volume, at a rate you only learn in a quote.

Participant hours

You pay raters directly plus a platform fee. Prolific's participant minimum is $8 an hour, recommended $12.

What grows the bill: minutes per item times raters per item.

Project quote

Managed evaluation and red-teaming (Scale AI, Surge AI, Toloka, TELUS Digital and others). Nothing published.

What grows the bill: scope, expertise and how many reruns you need.

Work out a first-round eval budget

What should you check before signing an eval contract?

These are the clauses and habits that decide whether the results are worth the invoice.

  • Training on your outputs. Get a written yes or no on whether your prompts, outputs or scores train any model, the vendor's or another client's.
  • Contamination. A public benchmark may already sit in the model's training data. Ask how the vendor keeps private test items private, and whether items are reused across clients.
  • Grader conflicts. If the same firm supplied the training data, its raters may share the model's blind spots. Say who should not grade.
  • Rater quality. Ask for qualification criteria and an agreement figure. A preference score from raters who agree half the time is noise.
  • Ownership of the eval set. Prompts, rubrics and grader code you paid for should be yours to rerun elsewhere.
  • Security claims. A badge on a page is a claim. Ask for the actual report (SOC 2, ISO 27001) under NDA, and check its date and scope.

How does an evaluation project usually run?

Most first engagements follow the same five steps, whichever vendor you choose.

  1. Scope

    Agree the behaviors to test, the model version and the pass marks. This becomes the test plan.

  2. Pilot set

    A small set of items is written and scored to check the rubric reads the same to everyone.

  3. Calibrate

    Raters or model graders are tuned until agreement is acceptable; ambiguous items are fixed or dropped.

  4. Full run and readout

    The full set runs. You get item-level results and a written readout of failures.

  5. Rerun

    The same set runs on each release so you can see movement, not just a snapshot.

Questions teams ask

What does an AI evaluation company do?

It measures how a model or AI product behaves before and after release. Some send trained people to rate outputs and attack the system (a managed service); others sell software your engineers use to run tests, score outputs and trace production traffic. Many sell both.

How much does AI evaluation cost?

Software has public prices: Braintrust charges $249 monthly for Pro, and LangSmith's Plus tier costs $39 per seat monthly (both as seen on 7 November 2026). Human rating through Prolific starts at its $8 an hour participant minimum plus a platform fee. Managed evaluation and red-teaming firms do not publish prices; they quote per project. Our eval budget page shows the arithmetic.

Is red-teaming the same as evaluation?

Red-teaming is one kind of evaluation: people (or tools) try to make the system fail on purpose, then report what worked. Benchmarks and regression suites measure known behavior repeatedly. Most teams need both; red-teaming services covers who sells the adversarial side.

Can we start with open-source tools instead of a vendor?

Yes, for automated tests. DeepEval is Apache-2.0 open source, Giskard has a free open-source library that runs locally, and Label Studio has an open-source Community Edition for human review. The gap open source does not fill is trained human raters and attackers.

Will an evaluation vendor train on our prompts and outputs?

Ask, and get the answer in the contract rather than a sales email. Check where outputs are stored, how long they are kept, who can see them, and whether any model (theirs or a client's) is trained on them.

Do you test or rank these vendors?

No. Everything here comes from each company's own published pages, read on the date shown. We do not audit vendors or run trials, so treat security badges and plan limits as claims to verify.

Who pays for this site?

Vendors may pay us a fee when an introduction leads to work. Teams pay nothing. The details are on how this works.

Where these facts come from

Get a quote

Company ZIP first, then a contact, then the evaluation itself. A few lines tell us which vendors are worth naming.

We are an introduction service, not an evaluation vendor, and we may be paid by the company we introduce you to.

Please do not paste prompts, logs, model outputs or any other data here. Describe the project in words; data only ever moves under a contract you sign with the vendor.

Part one of three

Where is the company based?
Get a quote