Community
LLM Benchmark
Detailed Eval Results
How well do leading LLMs write SpacetimeDB code? We prompt each model, run the generated code against live modules, and score with automated checks.
Claude Opus 4.8
Typescript · With Guidelines0.0%Task Pass Rate
0/89Checks Passed
47Total Evals
Pass Rate Over Time
Click a data point to view results from that date.
auth
0% · 8 evals| Status | Task ID | Checks | Run Time | |
|---|---|---|---|---|
| ✗ | t_026_auth_identity_check | 0/1 | 7.0s | ▼ |
| ✗ | t_027_private_vs_public_table | 0/1 | 6.0s | ▼ |
| ✗ | t_041_registered_user_gate | 0/1 | 8.3s | ▼ |
| ✗ | t_042_admin_bootstrap | 0/1 | 8.4s | ▼ |
| ✗ | t_043_role_based_access | 0/1 | 11.0s | ▼ |
| ✗ | t_044_ban_list | 0/1 | 8.7s | ▼ |
| ✗ | t_045_rate_limit | 0/1 | 8.7s | ▼ |
| ✗ | t_046_shared_document | 0/1 | 10.4s | ▼ |
basics
0% · 15 evals| Status | Task ID | Checks | Run Time | |
|---|---|---|---|---|
| ✗ | t_000_empty_reducers | 0/1 | 7.8s | ▼ |
| ✗ | t_001_basic_tables | 0/1 | 5.3s | ▼ |
| ✗ | t_002_scheduled_table | 0/1 | 7.6s | ▼ |
| ✗ | t_003_struct_in_table | 0/1 | 6.0s | ▼ |
| ✗ | t_004_insert | 0/2 | 5.7s | ▼ |
| ✗ | t_005_update | 0/3 | 7.2s | ▼ |
| ✗ | t_006_delete | 0/3 | 5.3s | ▼ |
| ✗ | t_007_crud | 0/4 | 5.6s | ▼ |
| ✗ | t_008_index_lookup | 0/3 | 8.1s | ▼ |
| ✗ | t_009_init | 0/4 | 7.4s | ▼ |
| ✗ | t_010_connect | 0/1 | 5.6s | ▼ |
| ✗ | t_011_helper_function | 0/3 | 6.0s | ▼ |
| ✗ | t_038_schedule_at_time | 0/1 | 6.4s | ▼ |
| ✗ | t_039_cancel_schedule | 0/1 | 7.9s | ▼ |
| ✗ | t_040_lifecycle_player | 0/1 | 5.8s | ▼ |
data modeling
0% · 6 evals| Status | Task ID | Checks | Run Time | |
|---|---|---|---|---|
| ✗ | t_024_event_table | 0/1 | 6.4s | ▼ |
| ✗ | t_025_optional_fields | 0/1 | 7.8s | ▼ |
| ✗ | t_028_cascade_delete | 0/1 | 11.8s | ▼ |
| ✗ | t_029_filter_and_aggregate | 0/1 | 9.6s | ▼ |
| ✗ | t_030_two_table_join | 0/1 | 8.4s | ▼ |
| ✗ | t_031_unique_constraint | 0/1 | 4.8s | ▼ |
queries
0% · 8 evals| Status | Task ID | Checks | Run Time | |
|---|---|---|---|---|
| ✗ | t_022_view_basic | 0/1 | 7.6s | ▼ |
| ✗ | t_023_view_per_user | 0/1 | 4.9s | ▼ |
| ✗ | t_032_range_query | 0/1 | 8.7s | ▼ |
| ✗ | t_033_sort_and_limit | 0/1 | 7.9s | ▼ |
| ✗ | t_034_find_first | 0/1 | 5.5s | ▼ |
| ✗ | t_035_select_distinct | 0/1 | 6.3s | ▼ |
| ✗ | t_036_count_without_collect | 0/1 | 7.1s | ▼ |
| ✗ | t_037_multi_column_filter | 0/1 | 8.3s | ▼ |
schema
0% · 10 evals| Status | Task ID | Checks | Run Time | |
|---|---|---|---|---|
| ✗ | t_012_spacetime_product_type | 0/3 | 6.3s | ▼ |
| ✗ | t_013_spacetime_sum_type | 0/3 | 7.3s | ▼ |
| ✗ | t_014_elementary_columns | 0/3 | 7.2s | ▼ |
| ✗ | t_015_product_type_columns | 0/3 | 7.3s | ▼ |
| ✗ | t_016_sum_type_columns | 0/3 | 7.6s | ▼ |
| ✗ | t_017_scheduled_columns | 0/2 | 7.9s | ▼ |
| ✗ | t_018_constraints | 0/3 | 7.4s | ▼ |
| ✗ | t_019_many_to_many | 0/5 | 10.9s | ▼ |
| ✗ | t_020_ecs | 0/7 | 11.5s | ▼ |
| ✗ | t_021_multi_column_index | 0/5 | 6.8s | ▼ |