Kolena AI Agent
LLM Output-Quality Evaluation
Independently score an AI/LLM output for accuracy, groundedness, completeness, compliance, and safety - with a rationale and cited evidence on every criterion and a pass/fail verdict.
Volume fit
Works best for
teams reviewing many AI/LLM outputs - a regression or acceptance set before a release, or ongoing sampling of a production AI system - where consistent, cited scoring at scale is the bottleneck
Too small for
a one-off eyeball of a single output where a person can simply read it
Typical inputs
Documents
- The AI/LLM output under review (the response, summary, draft, or extracted result)
- The task, prompt, or user question the output was meant to satisfy
- The source or grounding material the output should be faithful to (retrieved context, reference documents, system of record)
- The policy or rule set the output must comply with (content/safety policy, brand/style guide, disclosure or regulatory requirement, PII-handling rules)
- A reference or gold answer, when one exists
Systems
- AI / LLM application or pipeline that produced the output
- AI-governance / model-risk platform
- Evaluation / observability tooling
- Policy and standards repository
Output
An evaluation record - an evaluation summary, a per-criterion quality scorecard, a claim-level faithfulness/groundedness check, a policy-and-compliance adherence table, a safety/PII/fairness screen, an overall quality score, a pass/fail verdict, and a cited findings-and-remediation workpaper
Delivered to
- AI-governance / model-risk record and dashboards
- Release-gate / approval workflow
- Remediation / issue tracker
- Evaluation results store for trend monitoring
What it extracts
· 8 fields
- Form
Evaluation Summary
A one-look frame for the evaluation - what output is under review, the task it had to satisfy, the use case and risk tier, the grounding source and policy it is judged against, whether a reference answer exists, the evaluation mode (LLM-as-judge vs human-in-the-loop), and the model under review - so a reviewer knows what was tested and against what before reading the scores.
- Table
Quality Criteria Scorecard
The scoring rubric at the heart of the evaluation - one row per criterion (accuracy, faithfulness/groundedness, completeness, relevance/instruction-following, coherence/clarity, tone and style), each with a 1-5 score, a pass/partial/fail rating, a rationale tied to the output, and cited evidence.
- Table
Faithfulness and Groundedness Check
Claim-level hallucination detection - the output decomposed into atomic claims, each judged Grounded, Unsupported, Contradicted, or Not applicable against the provided source (a natural-language-inference check), with fabricated citations, numbers, and entities called out by name.
- Table
Policy and Compliance Adherence
The output tested against the supplied policy or rule set parsed into discrete rules, each marked Adheres, Violates, Partial, or Not applicable with a severity and cited evidence - including required disclosures that are present but incomplete, buried, or diluted.
- Table
Safety, PII and Fairness Screen
A screen for the trustworthy-AI harms beyond task quality - PII / sensitive-data leakage, unsafe or disallowed content, prompt-injection / instruction leakage, bias / fairness, and confident-but-wrong overreach - each recorded Clear, Flagged, or Needs human review with a severity, without reproducing sensitive values.
- Number
Overall Quality Score
A single 0-100 quality score summarizing the evaluation - the mean of the scorecard criteria with accuracy, faithfulness, and compliance weighted heaviest and hard penalties applied so a factually wrong, non-compliant, or unsafe output cannot score high no matter how fluent it is.
- Classification
Pass/Fail Determination
The overall verdict against the acceptance threshold - Pass, Pass with minor issues, Fail, or Needs human review - weighting the scorecard, compliance, safety, and score, holding higher-risk use cases to a stricter bar, and treating a single High-severity failure as disqualifying.
- Table
Evaluation Findings and Remediation
The findings workpaper - one severity-ordered row per defect surfaced above (failed/partial criterion, unsupported/contradicted claim, policy violation, or safety flag), each with cited evidence and a concrete remediation or an escalation to human review.
Prerequisites
- The output to be evaluated and the task it was meant to perform
- The grounding source (required to score faithfulness) and/or the policy set (required to score compliance) - the output can be scored on quality and internal consistency without them, but groundedness and compliance are then routed to human review
- The use case's risk tier, which sets how strict the pass threshold should be
Human review
The agent runs as an LLM-as-judge first pass that scores every criterion with cited evidence; a human reviewer confirms the verdict on high-risk outputs and on any item the agent routes to Needs human review before the output is trusted.
Where it fits
AI output-quality evaluation and AI-governance review (LLM-as-judge, human-confirmed)
An AI/LLM system produces an output (a customer reply, a summary, a drafted disclosure, a retrieval-augmented answer, an extracted field set) that must be reviewed before it is trusted, released, or used in a decision
This step
Output-quality evaluation - scoring the output against quality criteria, its grounding source, and the policy it must comply with, and returning a pass/fail verdict
After
Who uses it
Grounded in
- NIST AI Risk Management Framework (AI RMF 1.0), NIST AI 100-1verified as of 2026-07-22
- Federal Reserve SR 11-7 / OCC 2011-12 - Supervisory Guidance on Model Risk Managementverified as of 2026-07-22
- RAG evaluation triad - faithfulness/groundedness, answer relevance, context relevance (RAGAS-style claim-extraction + NLI method)verified as of 2026-07-22
See LLM Output-Quality Evaluation on your documents
We'll run it against a file of yours and walk through every cited field.