Recommended actions
- [Eval problem | Medium] Preserve full publish diagnostics in scorer output — Change the scorer so that
publish_error results keep the complete compiler or bundler stderr, or at least its final lines, instead of stopping at the first Saving config to ... line. All 15 shown failures report only that line, so the actual TypeScript error is missing. Root causes below are therefore inferred from code diffs rather than confirmed by the compiler. Evidence: t_038, t_039, t_061, t_062, t_063, t_065, t_069, t_071, t_074, t_075, t_076, t_077, t_078, t_079, t_087.
Failure patterns
Truncated publish error output (15 tasks)
- Classification: Eval problem
- Confidence: Medium
- What the model wrote: Varies by task (see patterns below).
- What was expected: A diagnostic that names the failing symbol, type, or line.
- What the error says:
publish_error: spacetime publish (typescript) failed (exit=2) (exit=1 for t_061), followed only by --- stderr --- Saving config to /tmp/stdb-llm-cli-...
- Why this happened: The captured stderr ends after the config-save message, before any TypeScript or bundler error appears. The exit codes show the build failed, but the report does not say why. The other patterns therefore rely on comparing generated code against the golden answers.
- Suggested action: Capture and report the full stderr, or its tail, for publish failures.
- Suggested area: Benchmark scorer / publish step error reporting
- Affected tasks: t_038, t_039, t_061, t_062, t_063, t_065, t_069, t_071, t_074, t_075, t_076, t_077, t_078, t_079, t_087
Invented HTTP handler and router API (5 tasks)
- Classification: Model limitation
- Confidence: High
- What the model wrote:
import { schema, HttpRouter, HttpResponse } from "spacetimedb/server";
return new HttpResponse("list", { status: 200 });
export const router = new HttpRouter().get("/items", GET).post("/items", POST);
import { schema, table, t, httpRouter } from "spacetimedb/server";
new Response("https://files.local/object-1", { status: 201 })
export const router = httpRouter().post("/upload", upload);
In t_078 and t_087 the model returned new Response('invalid', { status: 400 }), and t_078 never registers a router. t_079 also uses t.bytes() and response.arrayBuffer().
- What was expected:
import { Router, SyncResponse, schema } from 'spacetimedb/server';
new SyncResponse(`echo:${request.text()}`, { status: 201, ... })
export const routes = spacetimedb.httpRouter(new Router().post('/echo', echo));
t_079 expects t.array(t.u8()), new Uint8Array(data), and response.bytes().
- What the error says: Publish failed with exit code 2. The specific diagnostic is not captured.
- Why this happened: With no documentation, the model guessed the API. It invented exports that do not exist (
HttpRouter, HttpResponse, httpRouter), used the web-standard Response instead of SyncResponse, and did not wrap the router in spacetimedb.httpRouter(...).
- Suggested action: None
- Suggested area: None
- Affected tasks: t_076, t_077, t_078, t_079, t_087
Scheduled table definitions without the (): any thunk annotation (5 tasks)
- Classification: Model limitation
- Confidence: Medium
- What the model wrote:
const reminder = table(
{ name: 'reminder', public: false, scheduled: () => send_reminder }, ...);
export const send_reminder = spacetimedb.reducer({ arg: reminder.rowType }, ...);
t_071 also passes a Timestamp to ScheduleAt.time:scheduledAt: ScheduleAt.time(new Timestamp(ctx.timestamp.microsSinceUnixEpoch + 1_000n)),
- What was expected:
const reminder = table({ name: 'reminder', scheduled: (): any => sendReminder }, ...);
scheduledAt: ScheduleAt.time(ctx.timestamp.microsSinceUnixEpoch + 1_000n),
- What the error says: Publish failed with exit code 2. The specific diagnostic is not captured.
- Why this happened: The table references the reducer, and the reducer's argument type references the table's
rowType. This circular inference likely fails TypeScript checking unless the thunk return is annotated (): any, which every golden answer does. All five generated modules omit the annotation, so this is the most likely common cause. t_071 additionally wraps the argument to ScheduleAt.time in a Timestamp, while the golden passes a raw bigint. Without the compiler output, the exact error cannot be confirmed.
- Suggested action: None
- Suggested area: None
- Affected tasks: t_038, t_039, t_069, t_071, t_075
Incorrect view and query-builder API usage (3 tasks)
Classification: Model limitation
Confidence: Medium
What the model wrote:
- t_063:
ctx.db.event.occurredAt.filter({ start: { tag: 'included', ... }, end: { ... } })
- t_062:
indexes: [{ accessor: 'memberId', algorithm: 'btree', columns: ['memberId'] }],
ctx => ctx.from.member.semijoin(ctx.from.eligibility, (member, eligibility) => ...)
- t_061:
t.array(t.object('OrderLineDetail', { lineId: t.u64(), ... }))
What was expected:
- t_063:
import { Range, ... } from 'spacetimedb/server';
ctx.db.event.occurredAt.filter(new Range({ tag: 'included', ... }, { tag: 'included', ... }))
- t_062:
memberId: t.u64().index('btree')
ctx => ctx.from.eligibility.rightSemijoin(ctx.from.member, (eligibility, member) => eligibility.memberId.eq(member.id))
- t_061:
const OrderLineDetail = t.row('ThreeTableJoinRow', { ... });
What the error says: Publish failed with exit code 2, or exit code 1 for t_061. The specific diagnostic is not captured.
Why this happened: The model guessed the API shapes:
- It passed a plain object where a
Range instance is expected.
- It called a
semijoin method instead of rightSemijoin.
- It used
t.object where the golden uses t.row for the view row type.
The generated code for t_061 also appears truncated in the report, so its exact failure point is uncertain.
Suggested action: None
Suggested area: None
Affected tasks: t_061, t_062, t_063
Invalid context typing and procedure return shape (2 tasks)
- Classification: Model limitation
- Confidence: Medium
- What the model wrote:
- t_065:
type Db = ReducerCtx<typeof spacetimedb.schemaType>.db;
- t_074:
fetched_record: table({ public: true }, { ... }),
// procedure declared t.unit() but has no `return {}`
- What was expected:
- t_065:
type Ctx = ReducerCtx<InferSchema<typeof spacetimedb>>;
- t_074:
const fetchedRecord = table({ name: 'fetched_record', public: true }, { ... });
...
return {};
- What the error says: Publish failed with exit code 2. The specific diagnostic is not captured.
- Why this happened:
- In t_065,
Type<...>.db is not valid TypeScript. Property access on a type needs indexed-access syntax, and the model also used a non-existent schemaType instead of InferSchema.
- In t_074, the
t.unit() procedure returns nothing, while the golden returns {}.
- Suggested action: None
- Suggested area: None
- Affected tasks: t_065, t_074
Note: 2 additional failures were not shown and are not analyzed.