Storage: S3, GCS, Azure and disk

The key

On Cloudflare R2: R2 → Create bucket, then Manage API tokens → Create API token with “Object Read & Write” for that bucket only.

# The S3 API of the account, not the public address of the bucket (r2.dev)
S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
S3_BUCKET=acme-files
S3_REGION=auto
S3_ACCESS_KEY_ID=…
S3_SECRET_ACCESS_KEY=…
# Optional: before every key, to share the bucket with something else
S3_PREFIX=files/
# What tokens are signed with: long, random, the same in every process
REBOFILES_SECRET=…

s3Storage(s3OptionsFrom(process.env)) reads them. On AWS, leave S3_ENDPOINT out and give the region; MinIO and Backblaze take the same variables.

The rule of origins

In Settings → CORS policy, with the addresses your application is served from. ETag has to be exposed.

[
  {
    "AllowedOrigins": ["https://your-app.com", "http://localhost:3000"],
    "AllowedMethods": ["GET", "PUT"],
    "AllowedHeaders": ["content-type"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3600
  }
]

A lifecycle rule that aborts multipart uploads left unfinished after a day keeps the parts of an upload never finished from staying in the bucket.

Google Cloud Storage

Through its XML API, which speaks S3: the same addresses, signed with an HMAC key of the bucket.

import { gcsOptionsFrom, gcsStorage } from '@rebofiles/server';

// An HMAC key of the bucket: Cloud Storage → Settings → Interoperability.
// GCS_BUCKET, GCS_ACCESS_KEY_ID (GOOG1…), GCS_SECRET_ACCESS_KEY, GCS_PREFIX
createDriveHandler({ ...options, storage: gcsStorage(gcsOptionsFrom(process.env)) });

What that API does not do is done otherwise: each key deleted by itself, and every part of a new version sent, as it copies none. The bucket’s CORS, set with gcloud storage buckets update --cors-file, has to expose ETag.

Azure Blob Storage

A container of a storage account: addresses signed with the account’s key (a SAS of each blob, for minutes), each part a block, the file made of its list of blocks.

import { azureOptionsFrom, azureStorage } from '@rebofiles/server';

// AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_KEY, AZURE_STORAGE_CONTAINER, AZURE_STORAGE_PREFIX
createDriveHandler({ ...options, storage: azureStorage(azureOptionsFrom(process.env)) });

• Uploads resumed from the blocks Azure lists, and a new version sent with only its parts that changed (Put Block From URL).
• Blocks of an upload never finished go by themselves after a week: there is nothing to sweep.
• The account’s CORS (Resource sharing) lets in the origins of your pages, with PUT, GET and HEAD; no header has to be exposed, as the blocks are put together by their numbers.
• Tried against Azurite, Microsoft’s emulator, in the tests of the server.

On the disk

diskStorage({ folder, secret }) keeps the bytes in a folder, and the server carries them itself, at addresses of its own: for one machine, a demo, or a test.

When something fails

In the repository, npm run storage:check goes the whole way a file goes against the bucket of apps/storage/.env and says what the bucket answered at the step that failed.