Recommended actions
- [Eval problem | Medium] Capture the full publish/build diagnostics in scorer errors — Every shown failure reports only
Saving config to /tmp/... from stderr. The actual TypeScript or bundler error is never shown. The harness should keep the full compiler output (stdout and stderr, or at least the first error lines) in the recorded publish_error, so failures can be attributed to specific API misuse instead of inferred from code diffs. Evidence: t_030_two_table_join, t_071_scheduled_private, t_054_special_types, t_050_normalized_schema, t_048_heartbeat_isolation, t_070_connection_scoped_presence, t_069_scheduled_materialization, t_049_binary_storage, t_021_multi_column_index, t_047_normalized_collection, t_014_elementary_columns, t_052_autoinc_reference, t_015_product_type_columns, t_016_sum_type_columns, t_013_spacetime_sum_type.
Failure patterns
The model ran with no documentation, so it relied on base knowledge alone. The API misuse below therefore reflects the model's prior knowledge and is not evidence of a documentation or skill gap. Several tasks contain more than one mistake, so they appear in more than one group.
Scorer error omits the actual build diagnostic (15 tasks shown)
- Classification: Eval problem
- Confidence: Medium
- What the model wrote: Varies by task. See the groups below.
- What was expected: Varies by task.
- What the error says:
publish_error: spacetime publish (typescript) failed (exit=2) --- stderr --- Saving config to /tmp/stdb-llm-cli-.... No compiler message follows. t_016 exits with code 1 instead of 2, which points to a different failure stage that cannot be identified.
- Why this happened: The recorded error appears to keep only the first stderr line, or the tool sends diagnostics to stdout and the harness does not capture them. The root causes in the other groups are therefore inferred from diffs against the golden answers, not confirmed by the errors.
- Suggested action: Record the full publish/build output, or at least its error lines, in the failure record.
- Suggested area: Benchmark harness publish/scoring step
- Affected tasks: t_030, t_071, t_054, t_050, t_048, t_070, t_069, t_049, t_021, t_047, t_014, t_052, t_015, t_016, t_013
schema() called with positional tables instead of an object (7 tasks)
- Classification: Model limitation
- Confidence: High
- What the model wrote:
const spacetimedb = schema(
table({ name: 'primitive', public: true }, { ... })
);
export default schema(organization, department, employee);
- What was expected:
const spacetimedb = schema({ primitive });
export default schema({ organization, department, employee });
- What the error says: Publish failed with exit=2, which is consistent with a type or compile error. The diagnostic itself was not captured.
- Why this happened: The model does not know that
schema takes a single object whose keys become the ctx.db accessors.
- Suggested action: None
- Suggested area: None
- Affected tasks: t_054_special_types, t_014_elementary_columns, t_015_product_type_columns, t_013_spacetime_sum_type, t_070_connection_scoped_presence, t_050_normalized_schema, t_047_normalized_collection
Index definitions use name (or invented helpers) instead of accessor (6 tasks)
- Classification: Model limitation
- Confidence: High
- What the model wrote:
indexes: [{ name: 'byUserDay', algorithm: 'btree', columns: ['userId', 'day'] }]
owner: t.identity().indexed('btree', 'byOwner'),
- What was expected:
indexes: [{ accessor: 'byUserDay', algorithm: 'btree', columns: ['userId', 'day'] }]
owner: t.identity(), // with a table-level { accessor: 'byOwner', ... } index
- What the error says: Publish failed with exit=2. The diagnostic was not captured.
- Why this happened: The model guessed index option keys and invented a
.indexed() column method that does not exist.
- Suggested action: None
- Suggested area: None
- Affected tasks: t_030_two_table_join, t_050_normalized_schema, t_048_heartbeat_isolation, t_052_autoinc_reference, t_021_multi_column_index, t_049_binary_storage
Reducers registered by string name and not exported (3 tasks)
- Classification: Model limitation
- Confidence: High
- What the model wrote:
spacetimedb.reducer('build_order_details', (ctx) => { ... });
spacetimedb.reducer('seed', {}, ctx => { ... });
- What was expected:
export const build_order_details = spacetimedb.reducer((ctx) => { ... });
- What the error says: Publish failed with exit=2. The diagnostic was not captured.
- Why this happened: The model used an older or imagined API in which the reducer name is passed as a string. In the current API, the export binding name becomes the reducer name.
- Suggested action: None
- Suggested area: None
- Affected tasks: t_030_two_table_join, t_021_multi_column_index, t_052_autoinc_reference
Wrong import paths (6 tasks)
- Classification: Model limitation
- Confidence: Medium
- What the model wrote:
import { schema, table, t } from 'spacetimedb';
import { schema, table, t, ScheduleAt, ConnectionId } from 'spacetimedb/server';
- What was expected:
import { schema, table, t } from 'spacetimedb/server';
import { ScheduleAt, ConnectionId } from 'spacetimedb';
- What the error says: Publish failed with exit=2. The diagnostic was not captured, so it is not confirmed whether the
ScheduleAt/ConnectionId import location is fatal on its own.
- Why this happened: The model mixed up the server entry point (
spacetimedb/server) with the shared types entry point (spacetimedb).
- Suggested action: None
- Suggested area: None
- Affected tasks: t_050_normalized_schema, t_048_heartbeat_isolation, t_047_normalized_collection, t_071_scheduled_private, t_069_scheduled_materialization, t_070_connection_scoped_presence
Scheduled tables use a string reducer reference and an invented t.row() (2 tasks)
- Classification: Model limitation
- Confidence: High
- What the model wrote:
private_job: table({ scheduled: 'runPrivateJob' }, { ... })
{ job: t.row(spacetimedb.tables.private_job) }
{ job: t.row('refresh_job') }
- What was expected:
table({ name: 'private_job', scheduled: (): any => runPrivateJob }, { ... })
{ job: privateJob.rowType }
- What the error says: Publish failed with exit=2. The diagnostic was not captured.
- Why this happened: The model does not know that the scheduled reducer is referenced through a thunk and that the argument type comes from
table.rowType. It also omitted name on the job_result/private_job table options in t_071.
- Suggested action: None
- Suggested area: None
- Affected tasks: t_071_scheduled_private, t_069_scheduled_materialization
Sum-type values built with invented constructor methods (2 tasks)
- Classification: Model limitation
- Confidence: High
- What the model wrote:
a: Shape.circle(10),
b: Shape.rectangle({ width: 4, height: 6 }),
- What was expected:
a: { tag: 'circle', value: 10 },
b: { tag: 'rectangle', value: { width: 4, height: 6 } },
- What the error says: t_016 publish failed with exit=1. t_013 failed with exit=2, and it also has the positional
schema() mistake.
- Why this happened: The model assumed
t.enum produces variant factory functions, but values are tagged objects.
- Suggested action: None
- Suggested area: None
- Affected tasks: t_016_sum_type_columns, t_013_spacetime_sum_type
Truncated and speculative output (1 task)
- Classification: Model limitation
- Confidence: Medium
- What the model wrote: The generation ends mid-statement at
ctx.db.. It probes for nonexistent constructors (ConnectionId.fromU128, fromBigInt, and others) and uses (ctx as any).connectionId ?? (ctx as any).connection.
- What was expected:
new ConnectionId(1n), ctx.connectionId with a null check, and helper functions for adding and removing sessions.
- What the error says: Publish failed with exit=2. The incomplete source cannot compile.
- Why this happened: The model was unsure of the
ConnectionId API, generated defensive guesses, and did not finish the output. The supplied data does not show whether the cutoff came from an output length limit.
- Suggested action: None
- Suggested area: None
- Affected tasks: t_070_connection_scoped_presence
60 additional failures were not shown and are not analyzed here.