Mid-run evaluation

Sanity-check the half-trained model against a fixed prompt at every checkpoint, before the run finishes.

onCheckpoint lets you call into the partially trained model from your own code while the run is still going: each time the backend uploads a checkpoint, the SDK calls back into your function and hands you an infer bound to that exact checkpoint adapter.

This recipe wires it up against a fixed prompt so you can spot regressions long before the loss curve says anything is wrong.

The pattern

// src/arkor/trainer.ts
import { createTrainer } from "arkor";

const GOLDEN_PROMPT = [
  { role: "user" as const, content: "I can't log in to my account." },
];

export const trainer = createTrainer({
  name: "support-bot-v1",
  model: "unsloth/gemma-4-E4B-it",
  dataset: { type: "huggingface", name: "arkorlab/triage-demo" },
  lora: { r: 16, alpha: 16 },
  maxSteps: 100,
  callbacks: {
    onCheckpoint: async ({ step, infer }) => {
      try {
        const res = await infer({
          messages: GOLDEN_PROMPT,
          stream: false,        // get a single JSON body so this snippet stays short
          maxTokens: 80,
        });
        const data = (await res.json()) as {
          choices: Array<{ message: { content?: string | null } }>;
        };
        const sample = data.choices[0]?.message.content ?? "";
        console.log(`step=${step} sample=${JSON.stringify(sample.slice(0, 80))}`);
      } catch (err) {
        console.error(`step=${step} infer failed:`, err);
      }
    },
  },
});

What this gives you, immediately:

• A short generated sample written to stdout for every checkpoint, side by side with the loss numbers.
• Confirmation that inference itself works against the new adapter (so a silent serving-side regression is caught at training time).
• A natural place to add comparisons or assertions later.

Why this is hard to do anywhere else

infer is bound to the just-saved checkpoint ({ kind: "checkpoint", jobId, step }). You cannot reach an intermediate checkpoint from Studio's Playground, and there is no separate CLI command for it; the only path today is from inside onCheckpoint. That is exactly why this recipe wants to run there, not after the fact.

The function returns the raw Response from the cloud API, so the streaming and decoding shape is up to you. The snippet above passes stream: false to keep the body a single JSON document; for true streaming, see SDK § infer.

Variations

Forward each checkpoint's sample to a review queue. The callback's infer is permanently bound to the just-saved checkpoint and InferArgs has no base-model override, so this callback alone cannot produce a base-vs-adapter comparison (issue a separate base-model request via CloudApiClient.chat({ baseModel }), or use Studio's Playground Base / Adapter toggle manually). What the callback can automate by itself is capturing one sample per checkpoint for later review:

async function generate(prompt: typeof GOLDEN_PROMPT, infer: (args: any) => Promise<Response>) {
  const res = await infer({ messages: prompt, stream: false, maxTokens: 80 });
  const data = (await res.json()) as {
    choices: Array<{ message: { content?: string | null } }>;
  };
  return data.choices[0]?.message.content ?? "";
}

onCheckpoint: async ({ step, infer }) => {
  const sample = await generate(GOLDEN_PROMPT, infer);
  await postSampleToReviewQueue({ step, sample });
},

Extract typed fields with a JSON schema. Replace the freeform parse with responseFormat: { type: "json_schema", json_schema: { name, schema, strict: true } }. data.choices[0].message.content becomes a JSON string that always satisfies your schema, so a single JSON.parse gives you a typed object you can branch on directly (parsed.urgency === "high", parsed.category in VALID_LABELS). Full pattern, including how to wire the typed fields into early stopping, in the Structured outputs and function calling recipe.

Trigger early stopping based on the sample. Pair this with the Early stopping recipe: if the checkpoint output drifts away from a reference text by more than your tolerance, abort the controller. The next checkpoint will not fire.

Send checkpoints to a Slack channel for review. Combine with the Notifications recipe. Post each step's sample as a Slack message; reviewers can vote with reactions while the run continues.

What to keep in mind

• Wrap in try / catch. A throw out of onCheckpoint rejects wait() immediately; it is not caught by the SSE reconnect loop or retried (that loop only handles transport failures) (see SDK § Lifecycle callbacks). For non-fatal handling, catch the error inside the callback and decide what to do.
• Inference costs a real call. The backend serves the request from the live training cluster. Keep maxTokens modest if you are hitting every checkpoint.
• infer is per-call, not memoized. Calling it twice in the same onCheckpoint makes two backend requests. Compose your prompts together in one call when you can.