Community
LLM Benchmark
Detailed Eval Results
How well do leading LLMs write SpacetimeDB code? We prompt each model, run the generated code against live modules, and score with automated checks.
Leaderboard
Models ranked by eval pass rate.2026-07-14| Model | Eval Pass% ▾ | Checks | Cost | Run Time | Auth | Basics | Data Modeling | Queries | Schema |
|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.8 | 0.0% | 0/89 | $1.65264.5K tokens | 3m 46s | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Claude Sonnet 4.6 | 0.0% | 0/89 | $0.74194.2K tokens | 3m 44s | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| DeepSeek V4 Flash | 0.0% | 0/89 | $0.02191.7K tokens | 7m 36s | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| DeepSeek V4 Pro | 0.0% | 0/89 | $0.09176.8K tokens | 6m 21s | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Gemini 3.1 Pro | 0.0% | 0/89 | $1.04227.7K tokens | 9m 21s | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Gemini 3.5 Flash | 0.0% | 0/89 | $0.38182.8K tokens | 1m 45s | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| GPT-5.4-mini | 0.0% | 0/89 | $0.16162.2K tokens | 1m 29s | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| GPT-5.5 | 0.0% | 0/89 | $1.58178.7K tokens | 7m 34s | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Grok 4.3 | 0.0% | 0/89 | $0.31200.4K tokens | 5m 54s | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Grok Build 0.1 | 0.0% | 0/89 | $0.42286.7K tokens | 19m 19s | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |