Recommended actions
- [Eval problem | Medium] Include compiler diagnostics in publish errors — Every shown failure is reported only as
spacetime publish (typescript) failed (exit=2), followed by a single Saving config to ... line. The actual TypeScript or bundler error never appears, so the root causes below have to be inferred by comparing each answer against its golden answer. Capturing the full stderr/stdout of the publish or compile step in the scorer output would make these failures directly diagnosable. Evidence: t_014, t_015, t_017, t_018, t_019, t_028, t_029, t_047, t_049, t_050, t_051, t_052, t_066, t_069, t_071.
Failure patterns
The model received no documentation, so the API mistakes below reflect gaps in its base knowledge. Several tasks show more than one mistake, so the patterns overlap.
Publish diagnostics missing from scorer output (15 tasks)
- Classification: Eval problem
- Confidence: Medium
- What the model wrote: (varies; see the patterns below)
- What was expected: (varies)
- What the error says:
publish_error: spacetime publish (typescript) failed (exit=2) --- stderr --- Saving config to /tmp/stdb-llm-cli-..., with nothing after it.
- Why this happened: The recorded stderr stops after the config-saving line. No compiler message points to the offending line. The supplied output does not show whether the harness truncates the error or the CLI never emits it.
- Suggested action: Have the scorer capture and record the full publish/compile output, including the TypeScript diagnostics, in
publish_error results.
- Suggested area: Benchmark scorer's publish step
- Affected tasks: t_014, t_015, t_017, t_018, t_019, t_028, t_029, t_047, t_049, t_050, t_051, t_052, t_066, t_069, t_071
Index definitions use name instead of accessor (7 tasks)
- Classification: Model limitation
- Confidence: High
- What the model wrote:
indexes: [{ name: 'byOwner', algorithm: 'btree', columns: ['ownerId'] }],
- What was expected:
indexes: [{ accessor: 'byOwner', algorithm: 'btree', columns: ['ownerId'] }],
- What the error says: Publish failed with exit=2. No diagnostic was shown.
- Why this happened: The model used an older or guessed index option key. In t_018, t_047, t_050 and t_052 this is the only meaningful difference from the golden answer, which strongly suggests it alone causes the failure. t_049 also used
t.byteArray() where the golden answer uses t.array(t.u8()).
- Suggested action: None
- Suggested area: None
- Affected tasks: t_018, t_019, t_047, t_049, t_050, t_051, t_052
Reducer declared with a string name as the first argument (9 tasks)
- Classification: Model limitation
- Confidence: High
- What the model wrote:
export const seed = spacetimedb.reducer('seed', (ctx) => { ... });
spacetimedb.reducer('compute_stats', { category: t.string() }, (ctx, { category }) => { ... });
- What was expected:
export const seed = spacetimedb.reducer(ctx => { ... });
export const compute_stats = spacetimedb.reducer({ category: t.string() }, (ctx, { category }) => { ... });
- What the error says: Publish failed with exit=2. No diagnostic was shown.
- Why this happened: The model assumed an older-style signature that takes the reducer name as a string. In the current API, the name comes from the exported binding.
- In t_015, t_028 and t_066, this signature is the only meaningful difference from the golden answer.
- In t_029 and t_051, the reducers were also not exported.
- In t_014, the table options also omitted
name (table({ public: true }, ...)).
- Suggested action: None
- Suggested area: None
- Affected tasks: t_014, t_015, t_019, t_028, t_029, t_049, t_051, t_066, t_069
Scheduled-table imports and typing (3 tasks)
Classification: Model limitation
Confidence: Medium
What the model wrote:
import { schema, table, t, ScheduleAt, Timestamp, SenderError } from 'spacetimedb/server';
scheduled: () => runPrivateJob,
scheduledAt: ScheduleAt.time(new Timestamp(ctx.timestamp.microsSinceUnixEpoch + 1000n)),
What was expected:
import { ScheduleAt } from 'spacetimedb';
import { schema, table, t } from 'spacetimedb/server';
scheduled: (): any => runPrivateJob,
scheduledAt: ScheduleAt.time(ctx.timestamp.microsSinceUnixEpoch + 1_000n),
What the error says: Publish failed with exit=2. No diagnostic was shown.
Why this happened: The model made three related mistakes:
- It imported
ScheduleAt and Timestamp from spacetimedb/server instead of spacetimedb.
- In t_017 and t_071, it omitted the
(): any annotation on the circular scheduled thunk. That omission is a likely TypeScript inference error.
- It wrapped the schedule time in a
Timestamp instead of passing the bigint microseconds directly.
In t_069, the reducers also used string names.
Suggested action: None
Suggested area: None
Affected tasks: t_017, t_069, t_071
The remaining 35 failures were not shown and are not analyzed here.