BBF

Read the files

read_bbf(glob_paths)

Beacon reads Beacon Binary Format files.

SELECT time, depth, temperature FROM read_bbf('bbf/**/*.bbf')

A BBF query must name its columns. SELECT * and SELECT count(*) fail at plan time. The reader flattens each n-dimensional column on the dimensions of the selected columns. A scan of every column flattens on every dimension, and a scan of no column has no dimensions, so Beacon refuses both. Count over a named column: SELECT count(time) FROM read_bbf('bbf/**/*.bbf'). SELECT * works on an external table that declares its columns, because you chose that subset.

Inspect the schema

Check the columns and the types before you write a query:

SELECT * FROM read_bbf_schema('data/*.bbf');

Inspect a schema compares the _schema functions, SUMMARIZE, DESCRIBE and LIMIT 0, and says what each one costs.

Format details

The Beacon Binary Format (BBF) is the columnar format of Beacon. It suits the queries of earth science and oceanography.

• Full object storage support, with authenticated access.
• Chunk-level predicate pruning, like the row group filter of Parquet.
• Fast for repeated range queries over coordinate columns such as time, depth, latitude and longitude.

Convert your NetCDF files to BBF with the beacon-binary-format-toolbox. This makes queries over a large collection much faster.

As an external table

CREATE EXTERNAL TABLE my_table
STORED AS BBF
LOCATION 'path/to/files';

Declare a column list to read a subset of the files. Give each column the type that the files store. SELECT * then reads only those columns. SELECT count(*) still fails, because it selects no column:

CREATE EXTERNAL TABLE my_subset (depth FLOAT, temperature FLOAT)
STORED AS BBF
LOCATION 'path/to/files';

SELECT * FROM my_subset;

See Create External Tables for the full DDL. See Data Sources for the full read model.

BBF reads through the nd pipeline, like NetCDF, Zarr and Atlas. Each entry of a file is one dataset. A WHERE on a coordinate column runs before Beacon broadcasts the entry, so the rows it drops never exist in memory. A BBF table does not support PARTITIONED BY.

OPTIONS

STORED AS BBF reads no keys. A file carries its own schema, and the nd pipeline decides the shape of a batch. Beacon ignores a key you write, so a table created with the old split_streams_slice key keeps working.

See OPTIONS for the rules that hold for every key.