Query Data on S3

A Beacon server reads its datasets from one store: a local directory, or one S3-compatible bucket. The bucket is chosen at startup, not per query.

Your SQL does not change either way. Paths are relative to the datasets root:

SELECT * FROM read_parquet('obs/*.parquet') LIMIT 10;

On a local server obs/ is a directory. On a bucket-backed server it is a key prefix. You write the same thing.

A scheme in the path does not work

read_parquet('s3://my-bucket/obs/*.parquet') does not reach my-bucket. The scheme is ignored and the string is joined onto the datasets root. Write relative paths.

Point the server at your bucket

Credentials come from the standard AWS_* environment chain:

docker run -d --name beacon -p 5001:5001 \
  -e BEACON_S3_DATASETS=true \
  -e BEACON_S3_BUCKET=my-bucket \
  -e AWS_ACCESS_KEY_ID=… \
  -e AWS_SECRET_ACCESS_KEY=… \
  -e AWS_REGION=eu-west-1 \
  ghcr.io/maris-development/beacon:v2.0.0

For a public bucket, drop the keys and set AWS_SKIP_SIGNATURE=true.

See Object Storage for every setting, and Configuration for the full list.

There is no SQL statement for storage credentials

CREATE SECRET covers one case. It holds the credentials for another Beacon server, which you reach with ATTACH. Storage credentials come from the configuration. A server has one store. It selects that store at startup.

Make it fast

Object storage has a high latency. Fetch as few bytes as possible:

• Use cloud-optimized formats. Parquet, Zarr, Atlas and Cloud-Optimized GeoTIFF support range requests. Beacon then fetches only the chunks that it needs.
• Select only the columns that you need. Projection pushdown turns a narrow SELECT into fewer bytes.
• Filter early. A predicate prunes row groups and chunks before any transfer.
• Keep the pure-Rust reader for NetCDF and HDF5. It is the default, and it reads byte ranges from a private bucket. The netCDF-C library reads an anonymous bucket only. See NetCDF.

Register a prefix as a table

A stable name is better than a repeated glob. This is the same as with local files, because the path is relative either way:

CREATE EXTERNAL TABLE remote_obs
STORED AS PARQUET
LOCATION 'obs/';

See External Tables.