HDF5

SELECT * FROM read_hdf5('experiments/**/*.h5') LIMIT 10;

Beacon recognizes .h5 and .hdf5. It finds them in the dataset store automatically.

Two readers

A netCDF-4 file is an HDF5 file, and the netCDF-c library also opens plain HDF5. Beacon reads HDF5 through the pure-Rust reader by default. Beacon reads it through that library when you set the Rust reader to false. HDF5 behaves exactly like NetCDF: the same data model, the same array to table mapping, the same CF decoding and the same attribute columns.

The pure-Rust HDF5 reader is the default. It reads the same files and gives the same answer for a netCDF-4 file, and it adds five things over netCDF-c:

	netCDF-c	Pure-Rust reader (default)
Nested groups	Root group only	Every group
Compound datasets	Not read	One column for each member
Object storage	Anonymous access only	Full credential chain, no local copy
File statistics	None	Per-file column ranges
Concurrent scans	One file at a time	In parallel
Writes	netCDF-c	netCDF-c

The flag is on by default, and it is separate from BEACON_NETCDF_USE_RUST_READER, so you move one format at a time. Set it for one table instead of the whole server:

CREATE EXTERNAL TABLE experiments
STORED AS HDF5
LOCATION 'experiments/'
OPTIONS ('use_rust_reader' 'false');

How a dataset becomes a column

An HDF5 dataset reads as a variable. An HDF5 attribute reads as a <dataset>.<attribute> column, and a file attribute as .<attribute>.

The pure-Rust reader adds two more shapes. Both use /, so quote the column name in SQL.

A dataset inside a group

A dataset outside the root group keeps its path as its column name. Two groups can then hold the same name without a collision.

/                       .title
  station_id            station_id
  observations/         observations/.units
    temperature         observations/temperature
    qc/
      flag              observations/qc/flag

SELECT station_id, "observations/temperature", "observations/qc/flag"
FROM read_hdf5('nested.h5');

A group attribute takes the group's path and the leading dot: observations/.units.

A compound dataset

An HDF5 compound dataset holds a record in each element, the way a table row holds columns. Each member becomes its own column, named <dataset>/<member>. The columns share the dataset's shape, so a query joins them by row the way it joins two ordinary datasets.

SELECT "measurements/station", "measurements/depth", "measurements/label"
FROM read_hdf5('compound.h5');

Beacon reads a member of a fixed-width numeric or string type. It skips a member that holds a pointer into a heap — a variable-length string, a nested compound, an array — and logs the dataset and every member type. A compound whose members are all of that kind gives no columns, and the log names it.

Read one file, a glob or a list

-- one file
SELECT * FROM read_hdf5('experiments/run-01.h5');

-- every level below a directory
SELECT * FROM read_hdf5('experiments/**/*.h5');

-- an explicit list
SELECT * FROM read_hdf5(['a.h5', 'b.hdf5']);

Select the dimensions

Like NetCDF, read_hdf5 takes an optional second argument: the dimensions to read. It sets the grid, and Beacon returns a dataset only if the list holds all of that dataset's dimensions.

SELECT * FROM read_hdf5(['experiments/**/*.h5'], ['sample', 'channel']);

A netCDF-4 file names its axes with HDF5 dimension scales, and Beacon uses those names. A plain HDF5 file attaches no scales, so Beacon names the axes phony_dim_0, phony_dim_1 and so on, one for each axis. Two datasets of the same rank then share those names and broadcast together.

See Arrays to tables.

Inspect the schema

Check the columns and the types before you write a query:

SELECT * FROM read_hdf5_schema('experiments/**/*.h5');

Inspect a schema compares the _schema functions, SUMMARIZE, DESCRIBE and LIMIT 0, and says what each one costs.

As an external table

CREATE EXTERNAL TABLE experiments
STORED AS HDF5
LOCATION 'experiments/';

STORED AS H5 is accepted as a synonym.

On object storage

On the netCDF-c reader, HDF5 supports anonymous access only, for the same reason as NetCDF: that library opens a file by URL and does not go through the credential chain. For a public bucket, set AWS_SKIP_SIGNATURE=true.

The default pure-Rust reader has no such limit. It reads byte ranges through the object store, so a private S3, GCS or Azure bucket works and no local copy is made. A bucket needs no option at all:

CREATE EXTERNAL TABLE experiments
STORED AS HDF5
LOCATION 's3://bucket/experiments/';

See Object Storage.

As a query output

COPY TO ... STORED AS HDF5 uses the netCDF-4 writer. The result is a netCDF-4 file with an HDF5 extension. Every HDF5 reader opens it. A write always uses netCDF-c, whatever the read flag says.

What does not map

The pure-Rust reader covers nested groups and compound datasets. It does not read a variable-length or opaque type, a reference, or a region reference.

On the netCDF-c reader the netCDF data model bounds what Beacon sees. A compound datatype, a variable-length or opaque type, a reference, and a group outside the root are not read.

One more limit is worth knowing. Beacon drops an attribute narrower than 8 bytes on a file written with the earliest HDF5 library version — h5py's default — because the reader takes the object header padding as part of the value. The datasets are unaffected. This is an upstream defect; a netCDF-4 file never meets it.

See also

• NetCDF: the same readers, with the full detail
• Arrays to tables: the row count and the grid rule
• CF decoding: units, packing and fill values
• Performance tuning: when to change the reader