Skip to main content
Version: 2.0.0

Store files in S3

Prerequisites

You need a module and a client connected to it, and an Amazon S3 bucket, or a bucket on an S3-compatible service such as Cloudflare R2. If you don't have a module yet, follow the quickstart for your language first.

In this guide, you'll let players upload files, list them, download them and delete them. The files go straight from the client to your bucket, so large files never pass through SpacetimeDB, and your database only keeps a record of each file. Your module signs short-lived URLs that each allow one kind of request on one file, so your S3 secret key never reaches the client. You'll sign them in procedures, with the keys stored in environment variables.

How it works​

  • S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY environment variables store your S3 keys on the database.
  • A presign function signs a URL for one kind of request on one file, such as an upload, valid for 5 minutes.
  • A request_upload procedure checks the file's size, chooses where the file goes in the bucket, records it in a private pending_upload table, and returns a signed upload URL for a file of that size.
  • The client uploads the file to that URL with an HTTP PUT.
  • A confirm_upload procedure checks that the file is in the bucket and isn't too big, then moves it to the public uploaded_file table.
  • A request_download procedure returns a signed download URL, and a delete_file procedure deletes a file from the bucket and from uploaded_file.
  • Once an hour, a clean_up_uploads procedure deletes the uploads that players requested but never confirmed.

The bucket stays private: anyone who wants a file needs a URL that your module signed. The upload URL also fixes the file's size and content type, so the bucket rejects a file bigger than the one the player asked to upload.

Set up the bucket​

In your storage service:

  • Create a bucket, and keep it private.
  • Create an access key that can only list, read, write and delete files in that bucket. On AWS, that's an IAM user with the s3:GetObject, s3:PutObject and s3:DeleteObject permissions on arn:aws:s3:::your-bucket/*, and the s3:ListBucket permission on arn:aws:s3:::your-bucket. Without s3:ListBucket, S3 answers a request for a file that doesn't exist with an access error, so confirm_upload can't tell that the file was never uploaded.
  • If your client runs in a web browser, add a CORS rule to the bucket that allows PUT and GET requests, and the Content-Type header, from your site's origin. Browsers block the upload otherwise.

Define the tables and the keys​

Declare the S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY environment variables. They're required, so SpacetimeDB refuses to publish the module without them.

Then add three tables:

  • pending_upload is private. It has one row per upload that a player requested but hasn't confirmed yet, with the file's name and content type. Its primary key is the file's key: its path in the bucket.
  • uploaded_file is public, so clients can subscribe to it and list the files. It has one row per uploaded file, with its owner, key, name, content type, size and upload time.
  • upload_cleanup is the schedule table that runs clean_up_uploads, as described in Clean up abandoned uploads.
import { ScheduleAt, TimeDuration } from 'spacetimedb';
import { schema, table, t, type InferSchema, type ProcedureCtx } from 'spacetimedb/server';
import { hmac } from '@noble/hashes/hmac.js';
import { sha256 } from '@noble/hashes/sha2.js';
import { bytesToHex, utf8ToBytes } from '@noble/hashes/utils.js';

const pendingUpload = table(
  { name: 'pending_upload' },
  {
    key: t.string().primaryKey(),
    owner: t.identity(),
    name: t.string(),
    contentType: t.string(),
    requestedAt: t.timestamp(),
  }
);

const uploadedFile = table(
  { name: 'uploaded_file', public: true },
  {
    id: t.u64().primaryKey().autoInc(),
    owner: t.identity(),
    key: t.string(),
    name: t.string(),
    contentType: t.string(),
    size: t.u64(),
    uploadedAt: t.timestamp(),
  }
);

const uploadCleanup = table(
  { name: 'upload_cleanup' },
  {
    scheduledId: t.u64().primaryKey().autoInc(),
    scheduledAt: t.scheduleAt(),
  }
);

const spacetimedb = schema(
  { pendingUpload, uploadedFile, uploadCleanup },
  { env: { S3_ACCESS_KEY_ID: t.string(), S3_SECRET_ACCESS_KEY: t.string() } }
);
export default spacetimedb;

type Ctx = ProcedureCtx<InferSchema<typeof spacetimedb>>;

Add @noble/hashes to your module's dependencies, to compute the signatures:

npm install @noble/hashes

Set the keys when you publish, by putting them in the publishing command's environment:

S3_ACCESS_KEY_ID='your-access-key-id' S3_SECRET_ACCESS_KEY='your-secret-access-key' spacetime publish my-game

The keys stay in the database across later publishes, so you only pass them once. See Environment Variables for other ways to supply them, and to replace them.

Sign requests​

Add a presign function that signs a URL for one kind of request, such as a PUT or a GET, on one file. Anyone who has the URL can make that request until it expires, 5 minutes later, without your keys. It uses AWS Signature Version 4, which S3-compatible services also accept:

  1. It builds the request's canonical form: the method, the file's path, the signature's parameters and the signed headers, one per line.
  2. It derives a signing key from your secret key, the date, the region and the service, with HMAC-SHA256.
  3. It signs the canonical request with that key, and adds the signature to the URL.

For an upload, the URL also requires the request to send that exact Content-Type header, and a body of exactly the size that the module signed. The bucket rejects any other upload.

Set the constants at the top to your bucket's. With AWS, the host includes the bucket's name, and the bucket path is empty. Services that put the bucket in the path instead, such as Cloudflare R2, take their own host, and the bucket path /your-bucket.

// Your bucket. With AWS: '<bucket>.s3.<region>.amazonaws.com', and an empty path.
const S3_HOST = 'my-bucket.s3.us-east-1.amazonaws.com';
const S3_BUCKET_PATH = '';
const S3_REGION = 'us-east-1';
const URL_EXPIRY_SECONDS = 300; // 5 minutes

// Returns a URL that allows `method` requests on `key`, until it expires.
// For an upload, pass the content type and the size: the request must send that
// `Content-Type` header, and a body of exactly `contentLength` bytes.
function presign(ctx: Ctx, method: string, key: string, contentType = '', contentLength = 0n): string {
  // For example, 20260115T093000Z.
  const amzDate = ctx.timestamp.toDate().toISOString().replace(/[-:]/g, '').replace(/\.\d+/, '');
  const date = amzDate.slice(0, 8);
  const scope = `${date}/${S3_REGION}/s3/aws4_request`;
  const path = `${S3_BUCKET_PATH}/${key}`;
  const headers = contentType
    ? `content-length:${contentLength}\ncontent-type:${contentType}\nhost:${S3_HOST}\n`
    : `host:${S3_HOST}\n`;
  const signedHeaders = contentType ? 'content-length;content-type;host' : 'host';
  const query = [
    'X-Amz-Algorithm=AWS4-HMAC-SHA256',
    `X-Amz-Credential=${encodeURIComponent(`${ctx.env.S3_ACCESS_KEY_ID}/${scope}`)}`,
    `X-Amz-Date=${amzDate}`,
    `X-Amz-Expires=${URL_EXPIRY_SECONDS}`,
    `X-Amz-SignedHeaders=${encodeURIComponent(signedHeaders)}`,
  ].join('&');

  const canonicalRequest = [method, path, query, headers, signedHeaders, 'UNSIGNED-PAYLOAD'].join('\n');
  const stringToSign = [
    'AWS4-HMAC-SHA256',
    amzDate,
    scope,
    bytesToHex(sha256(utf8ToBytes(canonicalRequest))),
  ].join('\n');
  let signingKey = utf8ToBytes(`AWS4${ctx.env.S3_SECRET_ACCESS_KEY}`);
  for (const part of [date, S3_REGION, 's3', 'aws4_request']) {
    signingKey = hmac(sha256, signingKey, utf8ToBytes(part));
  }
  const signature = bytesToHex(hmac(sha256, signingKey, utf8ToBytes(stringToSign)));
  return `https://${S3_HOST}${path}?${query}&X-Amz-Signature=${signature}`;
}

The file keys that your module chooses only contain letters, digits and slashes, so presign doesn't need to URL-encode them.

Request an upload​

Add a request_upload procedure that clients call with the file's name, its content type, such as image/png, and its size in bytes. It:

  • Rejects names that are empty or longer than 200 characters, content types that aren't a type and a subtype, such as image/png, and files bigger than 50 MB.
  • Chooses the file's key from the caller's identity and the current time, such as uploads/c200…/1767864600000000. Players can't choose where their file goes, so they can't overwrite anyone else's.
  • Records the upload in pending_upload, and returns the key and a signed PUT URL. The URL only accepts a file of that size and content type, so a player can't upload a bigger file than the one they asked for.

The procedure returns an UploadTicket value with three fields: key, url, and error, which explains what went wrong, or is empty on success. Returning errors this way, instead of throwing, means clients get the same kind of result whichever language your module is written in.

const MAX_NAME_LENGTH = 200;
const MAX_FILE_SIZE = 50 * 1024 * 1024; // 50 MB

const uploadTicket = t.object('UploadTicket', {
  key: t.string(),
  url: t.string(),
  error: t.string(),
});

export const requestUpload = spacetimedb.procedure(
  { name: t.string(), contentType: t.string(), size: t.u64() },
  uploadTicket,
  (ctx, { name, contentType, size }) => {
    if (name.length === 0 || name.length > MAX_NAME_LENGTH) {
      return { key: '', url: '', error: `File names must be 1 to ${MAX_NAME_LENGTH} characters long` };
    }
    if (!/^[\w.+-]+\/[\w.+-]+$/.test(contentType)) {
      return { key: '', url: '', error: "That doesn't look like a content type" };
    }
    if (size > BigInt(MAX_FILE_SIZE)) {
      return { key: '', url: '', error: `Files must be at most ${MAX_FILE_SIZE / (1024 * 1024)} MB` };
    }
    const key = `uploads/${ctx.sender.toHexString()}/${ctx.timestamp.microsSinceUnixEpoch}`;
    ctx.withTx(tx => {
      tx.db.pendingUpload.insert({ key, owner: tx.sender, name, contentType, requestedAt: tx.timestamp });
    });
    return { key, url: presign(ctx, 'PUT', key, contentType, size), error: '' };
  }
);
Accepting only some types

To accept only some kinds of files, such as images, check the content type against a list in request_upload. The content type is signed into the upload URL, so the bucket rejects an upload with a different Content-Type header. It's still the type that the client says the file has: if you need to know what the file really contains, check its content after the upload.

Confirm the upload​

Add an s3_request function (s3Request in TypeScript, S3Request in C#) that sends a signed request for a file to the bucket, and returns the status of the reply, or 0 if there was no reply. For a HEAD request, it also returns the file's size.

Then add a confirm_upload procedure that clients call once their upload is done. It:

  • Rejects keys that aren't one of the caller's pending uploads.
  • Sends a HEAD request for the file, which returns its size without downloading it. It rejects the upload if the file isn't in the bucket.
  • As a second check, if the file is bigger than 50 MB, deletes it from the bucket and from pending_upload, and rejects the upload.
  • Otherwise, moves the upload from pending_upload to uploaded_file, in one transaction.

It returns an error message, or an empty string on success.

// Sends a signed request for `key` to the bucket.
// Returns the status of the reply, or 0 if there was no reply, and the size of the file.
function s3Request(ctx: Ctx, method: string, key: string) {
  try {
    const response = ctx.http.fetch(presign(ctx, method, key), {
      method,
      timeout: TimeDuration.fromMillis(10_000),
    });
    return { status: response.status, size: Number(response.headers.get('content-length') ?? 0) };
  } catch (error) {
    console.error(`Could not reach the bucket: ${error}`);
    return { status: 0, size: 0 };
  }
}

export const confirmUpload = spacetimedb.procedure({ key: t.string() }, t.string(), (ctx, { key }) => {
  const pending = ctx.withTx(tx => tx.db.pendingUpload.key.find(key));
  if (!pending || !pending.owner.isEqual(ctx.sender)) {
    return 'Upload not found';
  }
  const head = s3Request(ctx, 'HEAD', key);
  if (head.status !== 200) {
    return head.status === 404 ? 'The file was not uploaded' : 'Could not check the file';
  }
  if (head.size > MAX_FILE_SIZE) {
    s3Request(ctx, 'DELETE', key);
    ctx.withTx(tx => tx.db.pendingUpload.key.delete(key));
    return `Files must be at most ${MAX_FILE_SIZE / (1024 * 1024)} MB`;
  }
  ctx.withTx(tx => {
    // Do nothing if another call already confirmed this upload.
    if (tx.db.pendingUpload.key.delete(key)) {
      tx.db.uploadedFile.insert({
        id: 0n,
        owner: pending.owner,
        key,
        name: pending.name,
        contentType: pending.contentType,
        size: BigInt(head.size),
        uploadedAt: tx.timestamp,
      });
    }
  });
  return '';
});

Clients can only confirm their own uploads, and only with the size that the bucket reports, so the uploaded_file table only lists files that are really in the bucket.

Until the upload URL expires, 5 minutes after request_upload, the player can use it again. They can only send a file of the same size and type to the same key, so at most they replace their own file. If they delete the file in that time, they can also put it back in the bucket, without a row in uploaded_file.

Download and delete files​

Add two more procedures:

  • request_download returns a signed GET URL for a file in uploaded_file. Any player can download any file, as with a public file list. To limit who can, check the caller first, as delete_file does.
  • delete_file deletes a file from the bucket, then from uploaded_file. Only the file's owner can delete it. For other players, it answers as if the file didn't exist.

request_download returns a DownloadLink value, with a url and an error. delete_file returns an error message, or an empty string on success.

const downloadLink = t.object('DownloadLink', {
  url: t.string(),
  error: t.string(),
});

export const requestDownload = spacetimedb.procedure({ fileId: t.u64() }, downloadLink, (ctx, { fileId }) => {
  const row = ctx.withTx(tx => tx.db.uploadedFile.id.find(fileId));
  if (!row) {
    return { url: '', error: 'File not found' };
  }
  return { url: presign(ctx, 'GET', row.key), error: '' };
});

export const deleteFile = spacetimedb.procedure({ fileId: t.u64() }, t.string(), (ctx, { fileId }) => {
  const row = ctx.withTx(tx => tx.db.uploadedFile.id.find(fileId));
  if (!row || !row.owner.isEqual(ctx.sender)) {
    return 'File not found';
  }
  const { status } = s3Request(ctx, 'DELETE', row.key);
  if (status < 200 || status >= 300) {
    return 'Could not delete the file';
  }
  ctx.withTx(tx => tx.db.uploadedFile.id.delete(fileId));
  return '';
});

Clean up abandoned uploads​

A player can request an upload and never confirm it, for example because they closed the game halfway through. Its row stays in pending_upload, and its file, if they sent one, stays in the bucket.

Insert a row into upload_cleanup from the init reducer, so that SpacetimeDB calls clean_up_uploads every hour. Then add the clean_up_uploads procedure. It finds the pending uploads that were requested more than an hour ago, and for each of them:

  1. Deletes its row from pending_upload. From then on, confirm_upload rejects it, so it can't list a file that's about to be deleted.
  2. Deletes its file from the bucket. Deleting a file that was never uploaded succeeds too, so there's no need to check first.

An upload URL expires after 5 minutes, so an hour leaves plenty of time for a slow upload to finish and be confirmed.

const CLEANUP_INTERVAL_MICROS = 60n * 60n * 1_000_000n; // 1 hour
const ABANDONED_AFTER_MICROS = 60n * 60n * 1_000_000n; // 1 hour

export const init = spacetimedb.init(ctx => {
  ctx.db.uploadCleanup.insert({
    scheduledId: 0n,
    scheduledAt: ScheduleAt.interval(CLEANUP_INTERVAL_MICROS),
  });
});

export const cleanUpUploads = spacetimedb.procedure(
  { onSchedule: uploadCleanup },
  { cleanup: uploadCleanup.rowType },
  t.unit(),
  ctx => {
    const cutoff = ctx.timestamp.microsSinceUnixEpoch - ABANDONED_AFTER_MICROS;
    const abandoned = ctx.withTx(tx =>
      Array.from(tx.db.pendingUpload.iter())
        .filter(upload => upload.requestedAt.microsSinceUnixEpoch < cutoff)
        .map(upload => upload.key)
    );
    for (const key of abandoned) {
      if (ctx.withTx(tx => tx.db.pendingUpload.key.delete(key))) {
        s3Request(ctx, 'DELETE', key);
      }
    }
    return {};
  }
);

If the bucket can't be reached, s3_request (s3Request in TypeScript, S3Request in C#) logs the error, and the file stays in the bucket without a row. It no longer appears anywhere in your database, so it's harmless, but it still uses storage.

note

init only runs when the database is created. Publishing new code to an existing database doesn't run it again, so the cleanup won't start there. To start it, either republish with --delete-data, which erases all data, or insert the row from another reducer. Before inserting, check that upload_cleanup is empty: with two rows, clean_up_uploads would run twice as often.

Upload from the client​

On the client, subscribe to uploaded_file to list the files. To upload a file:

  1. Call request_upload with the file's name, content type and size.
  2. Send the file to the returned URL with an HTTP PUT, and a Content-Type header with the same content type. HTTP libraries send the file's size in a Content-Length header themselves, and it must match the size you passed.
  3. Call confirm_upload with the returned key.

To download a file, call request_download, and open or fetch the returned URL within 5 minutes.

import { DbConnection, tables } from './module_bindings';

const conn = DbConnection.builder()
  .withUri('ws://localhost:3000')
  .withDatabaseName('my-game')
  .onConnect(ctx => {
    ctx.subscriptionBuilder().subscribe([tables.uploadedFile]);
  })
  .build();

function listFiles() {
  for (const file of conn.db.uploadedFile.iter()) {
    console.log(`${file.id}: ${file.name} (${file.size} bytes)`);
  }
}

// Uploads a file, such as one the player picked in a file input.
async function uploadFile(file: File) {
  const contentType = file.type || 'application/octet-stream';
  try {
    const ticket = await conn.procedures.requestUpload({ name: file.name, contentType, size: BigInt(file.size) });
    if (ticket.error) {
      console.error(`Could not upload the file: ${ticket.error}`);
      return;
    }
    const response = await fetch(ticket.url, {
      method: 'PUT',
      headers: { 'content-type': contentType },
      body: file,
    });
    if (!response.ok) {
      console.error(`The upload failed with status ${response.status}`);
      return;
    }
    const error = await conn.procedures.confirmUpload({ key: ticket.key });
    if (error) {
      console.error(`Could not upload the file: ${error}`);
    }
  } catch (error) {
    // The procedure call or the upload itself failed, for example because a browser blocked it.
    console.error(`Could not upload the file: ${error}`);
  }
}

// Returns a URL to download the file from, valid for 5 minutes.
async function downloadUrl(fileId: bigint) {
  try {
    const link = await conn.procedures.requestDownload({ fileId });
    if (link.error) {
      console.error(`Could not download the file: ${link.error}`);
      return undefined;
    }
    return link.url;
  } catch (error) {
    console.error(`Could not call the procedure: ${error}`);
    return undefined;
  }
}

async function deleteFile(fileId: bigint) {
  try {
    const error = await conn.procedures.deleteFile({ fileId });
    if (error) {
      console.error(`Could not delete the file: ${error}`);
    }
  } catch (error) {
    console.error(`Could not call the procedure: ${error}`);
  }
}

What's next?​

You now store players' files in a bucket, and keep track of them in your database:

  • Clients upload and download files directly from the bucket, with URLs that your module signs and that expire after 5 minutes.
  • Your module chooses where each file goes and how big it can be, and checks that it's in the bucket before it lists it.
  • A scheduled procedure deletes the uploads that players never confirmed.
  • The bucket stays private, and your S3 keys never leave the database.

To adapt it to your app:

  • Limit who can download. Check the caller in request_download, for example to let only the owner download a file, or the owner's friends, as in Build a friends or follow list. To also hide the file list, make uploaded_file private and give each player their own files through a view.
  • Other S3-compatible services. With Cloudflare R2, set the host to <account-id>.r2.cloudflarestorage.com, the bucket path to /your-bucket, and the region to auto.
  • Public files. To serve files that anyone can see, such as avatars, make their part of the bucket public, or put a CDN in front of it, and store the public URL in uploaded_file instead of signing download URLs.

To learn more about the features used here, see Procedures, Environment Variables and File Storage.

Next, you can keep usernames unique and block offensive ones.