**Show HN: Jevals – replacing LLM judges with typed Jev decisions**

> Agent evals and guardrails in one request. Built on Jev, Kev and Laya. - openlayer-ai/jevals jevals Evals and guardrails for agents, using Jev-style decision models instead of an LLM judge. All the evals for a trace go out as one request that costs a few thousandths of a cent and comes back in a few hundred milliseconds, so you can run them on every trace and inside the agent loop. Works with Jev through the TypeSafe or Vercel APIs, with Kev or Laya running locally on a Mac, or with a regular chat LLM if that's all you have (slower, costs more). pip install jevals export AI\_GATEWAY\_API\_KEY=... \# Jev through Vercel AI Gateway. TYPESAFE\_API\_KEY and OPENROUTER\_API\_KEY also work. from jevals import evaluate from jevals . agent import ToolChoice , UsedToolResult , Grounded , StayedInScope from jevals . quality import AnswerRelevancy , Completeness from jevals . security import IndirectInjection , PHI r = evaluate ( { "messages" : messages , "tools" : tools }, \# the list you sent to the model and the tool schemas you gave it \[ ToolChoice (), UsedToolResult (), Grounded (), StayedInScope (), AnswerRelevancy (), Completeness (), IndirectInjection (), PHI ()\], ) r . tool\_choice . answer \# "correct" (p=0.99) r . grounded . score \# 0.5, 1 of 2 claims supported by tool results r . answer\_relevancy . score \# 0.84 r . indirect\_injection . passed \# True (p=0.03). False when a tool result tells the agent to do something. r . usage \# 1 request · 1,388 tokens · $0.00006 · 0.33s That's one HTTP request for all eight, and those are the numbers Jev returned for the trace in examples/quickstart.py . messages is the OpenAI chat format (user, assistant with tool\_calls , tool). If you have Anthropic content blocks or LangChain message objects you can pass those directly. The problem Most teams eval a small sample of their traffic, if they eval at all, and the reason is usually cost. The judge is a frontier LLM and it is the expensive part of the pipeline. Ragas is where most of us got our metric names, and its source shows where an LLM judge spends its money. Faithfulness is two LLM calls. Answer relevancy is three, plus embeddings. Context precision is one call per retrieved chunk. Every call carries few-shot examples, generates JSON token by token, and retries when the JSON doesn't parse. Running four metrics on one sample works out to six to eleven round trips depending on how many chunks you retrieved (measured below), and several seconds. At that price you sample 1%, run it nightly, and the results never get anywhere near the request path. Agents make this harder. The traces are long, there's more to check (did it pick the right tool, did it use the result, did it stay in scope, did a tool result contain instructions it shouldn't have followed), and the judge is non-deterministic on top of being slow. LangChain ran the comparison last week: on identical traces, GPT and Claude judges had 92x to 913x the score variance of Jev. A judge that changes its mind between runs makes a poor test suite. What changed Jev doesn't generate text. You send it some state and a set of typed questions (yes/no, pick one of these, score on this rubric) and it returns a calibrated probability for each one in a single forward pass. The questions are evaluated independently and in parallel, so asking 40 costs about the same latency as asking one. Pricing is $0.042 per million input tokens with no output tokens. Through Vercel's gateway we measured p50 244ms, p95 371ms per request. Within a week of the launch there were open-weight models speaking the same API: Kev (Qwen3, runs on a Mac) and Laya (ModernBERT, about 10ms on Apple Silicon). The request shape, state + {id: {type, instructions, criteria}} , looks like it's going to stick. M...

**来源信息**
- **来源**：Hacker News：AI 热帖
- **分类**：ai-models
- **发布时间**：2026-09-21 06:39（北京时间）
- **原文**：[打开原文](https://github.com/openlayer-ai/jevals)