verb · → decisions

evaluate

Ask small models typed questions about text, images or speech. Answers come back as probabilities in milliseconds, with no text generation. Use it for routing, moderation, policy checks, visual inspection, PII and entity extraction.

Ask questions#

import { boolean, choice, evaluate, score, spans } from 'edgewise';

const { answers, confidence } = await evaluate({
  model: 'judge:router',
  state: 'My package arrived broken and I want my money back.',
  questions: {
    topic: choice({ refund: 'a refund request', billing: 'a billing question', praise: 'praise' }),
    mood: score(['calm', 'annoyed', 'furious']),
    wantsRefund: boolean({ true: 'wants a refund' }),
  },
});

answers.topic.choice;          // 'refund'
answers.topic.probabilities;   // { refund: 0.93, billing: 0.05, praise: 0.02 }
answers.mood.level;            // 'annoyed'
answers.wantsRefund.probability; // 0.97
confidence.topic;              // 0..1

state is a string, or any JSON value. Objects are serialized with sorted keys so the same state always gives the same answer.

Images and speech#

Decision models such as d1-omni-600m (alias judge:omni) also judge images and audio. Pass them next to the state, or instead of it when the media are the whole state. Write each question's instructions: d1 reads them.

const photo = await evaluate({
  model: 'judge:omni',
  images: [frame],                       // any image Edgewise reads: canvas, Blob, URL, { data, width, height, channels }
  questions: {
    damaged: boolean({ instructions: 'Is the product damaged?' }),
    part: choice({ hinge: '', screen: '', keyboard: '' }, { instructions: 'Which part is shown?' }),
  },
});

const voice = await evaluate({
  model: 'judge:omni',
  state: { channel: 'support line' },
  audio: clip,                           // up to 30 s; Float32Array at 16 kHz, or pass sampleRate
  questions: { kind: choice({ request: 'a request', complaint: 'a complaint', chat: 'small talk' }, { instructions: 'What kind of utterance is this?' }) },
});

One call takes images or audio, not both. The media are encoded once and every question is answered in one batched pass. Models that do not accept images or audio throw UnsupportedInputError.

d1-omni's audio understanding is a research preview from LiquidAI, trained on English requests to an assistant. It reliably tells what kind of utterance a clip is, but can miss what the speaker asks for. Check it on your own clips before you rely on it.

The larger d1#

d1-3b (alias judge:large) is LiquidAI's 3-billion-parameter d1, built on LFM2.5-VL-3B. It leads LiquidAI's Decision Index among models under 10B, and is stronger than d1-omni-600m on knowledge, retrieval and tool questions. It takes text and images, not audio, and downloads about 4.6 GB. It runs on Bun and Node (on the CPU, or the GPU through vgpu); browsers cannot allocate its decoder, so in a browser use d1-omni-600m. Questions and answers work as with d1-omni-600m; a boolean's true and false descriptions are shown to it as "Yes:" and "No:" lines.

const { answers } = await evaluate({
  model: 'judge:large',
  state: { ticket: 'Charged twice for order 1182', customer: 'pro plan' },
  questions: { refund: boolean({ instructions: 'Is the customer asking for a refund?' }) },
  allowPreview: true,
});

Many items at once#

Pass items to ask the same questions about many states. Each item takes its own state, images or audio, and you get one result per item, in order. d1 models pack every question of every item into as few forward passes as fit, which is much faster than one call per item.

const { results } = await evaluate({
  model: 'judge:omni',
  items: tickets.map((t) => ({ state: t })),
  questions: { lane: choice({ billing: 'payments', tech: 'bugs', chat: 'small talk' }, { instructions: 'Which team should handle this?' }) },
});
results.map((r) => r.answers.lane.choice);

Truncated input#

Every model has a context limit. d1 answers carry truncated: true when the model saw only part of the input: a state, instructions or option cut to fit, or audio past 30 s. Its answer is still a decision about the part it saw. Other models leave truncated unset; NLI and classifier judges read the first 512 tokens.

Question types#

Answers are typed from the question: answers.topic.choice has type 'refund' | 'billing' | 'praise'.

Every question can set its own model, threshold and instructions. Answers can also carry truncated (see below). With a threshold, answers carry flagged: booleans, scores and spans are flagged when they reach it, and a choice falls back to otherwise (and is flagged) when its top probability is below it.

How it works#

Edgewise picks the method from the model:

• NLI models (nli-deberta-v3-xsmall) score each description as a hypothesis against the text: zero-shot, any labels.
• The LFM2.5 prompt router (lfm2.5-encoder-350m-router) embeds the text and each category description, and compares them. It is trained for routing and is multilingual.
• Classifiers (prompt-injection-deberta-v3) have fixed labels. boolean() maps to the positive label; label() returns all of them.
• Token classifiers (bert-base-ner, lfm2.5-encoder-350m-pii, piiranha-v1) answer spans(), with character offsets into your text.
• Decision models (d1-omni-600m, d1-3b) read the state, the question's instructions and its options, and score each option from one forward pass. Text answers use d1's learned calibration temperatures. JSON state is sent as JSON, the format d1 was trained on. Without instructions, a choice asks "Which option fits best?", a score asks "Which level fits best?", and a boolean asks its true description.

A question the model cannot answer throws UnsupportedInputError naming models that can.

Calibrate thresholds#

Zero-shot probabilities are relative, not absolute. Before you gate on a threshold, run 50 to 100 labelled examples from your own traffic and pick the value that gives the precision you need. confidence (1 minus normalized entropy) helps spot inputs where the model is unsure.

Models#

For the full, current list see Models.