How It Works

Beacon uses Rust and two Apache projects. Arrow holds columnar data in memory. DataFusion plans and runs the SQL. All of this happens inside your process.

The query pipeline

A query moves through four stages:

1. Parse and plan. Beacon parses the SQL and builds a logical plan. It resolves the plan against the catalog: tables, views, attached catalogs and reader functions.
2. Optimize. The planner rewrites the plan. It also decides how much work it can push into the data sources.
3. Scan. The readers open only the files, chunks and columns that the plan still needs. They decode them into Arrow record batches.
4. Execute and stream. Operators run over Arrow batches. The results stream out. A query over terabytes of files therefore never holds everything in memory.

Read in place

There is no load step. A reader table function such as read_netcdf() or read_parquet() opens your files at query time. It shows them as a table:

SELECT time, latitude, temperature
FROM read_netcdf('argo/**/*.nc')
WHERE temperature > 20;

A glob expands across directories. One statement can therefore cover thousands of files. Beacon merges their schemas and reads them in parallel. You can also register the same files once as an external table and query them by name.

Pushdown

Pushdown gives most of the performance. Beacon does not read everything and filter afterwards. It moves filters and column selections as close to the data as possible:

• Projection pushdown: Beacon decodes only the columns that a query names. A query over 3 of 200 variables reads about 3 columns of bytes.
• Predicate pushdown: Beacon turns filters into file, row group and chunk pruning. A time range filter on a Zarr store fetches only the chunks in that range. On Atlas, Beacon uses the stored statistics. It can drop whole datasets before it reads any array data.
• Federated pushdown: Beacon sends filters, projections, limits and whole aggregates to SQL databases and to remote Beacons. Only the reduced result travels back.

Use EXPLAIN to see what the planner pushes down:

EXPLAIN SELECT count(*) FROM read_parquet('obs/*.parquet') WHERE depth < 50;

Arrays become tables

Scientific formats often hold multi-dimensional data. Beacon maps an array variable onto columns. Ordinary SQL then works on it. Beacon also exposes the metadata as columns. The attributes of a variable appear as variable.attribute, for example temperature.units. File attributes appear as .attribute.

Arrays to tables specifies the mapping: the row count, the broadcast rule and the dimensions argument. File Formats documents the behaviour of each format.

Storage: what Beacon owns

Most data stays where it is. Beacon owns the content of one beacon.db file. That file holds the catalog with the table, view and secret definitions. It also holds the managed table data. Copy that one file and the database goes with it. The copy still references the external files and the remote systems.

your query
    │
    ▼
Beacon engine    ──►  beacon.db     (catalog + managed tables, owned)
    │
    ├──────────────►  files          (NetCDF, Zarr, Parquet, ... read in place)
    ├──────────────►  object storage (S3, GCS, Azure)
    └──────────────►  other systems  (Postgres, MySQL, other Beacon servers)

The service layer

The engine above sits inside the server. Around it the server adds an HTTP API, an Arrow Flight SQL API, a managed dataset store, crawlers, role-based access control and a web admin UI. Those surfaces change who reaches the engine, not what it does. The planner, the readers, the formats and the SQL dialect are the same on every transport.

See Access Control for the grant model, and Getting Started to run a server.