NetCDF

Read the files

read_netcdf(glob_paths)
read_netcdf(glob_paths, dimensions)

Beacon reads the NetCDF files that match one or more glob patterns.

The optional dimensions argument selects the variables. Beacon returns a variable only if the list holds all of its dimensions. Use the argument to drop variables with many dimensions. Also use it when the files hold variables with different dimensions.

SELECT time, latitude, longitude, temperature
FROM read_netcdf('argo/**/*.nc')

-- With explicit dimension columns
SELECT *
FROM read_netcdf(['argo/**/*.nc'], ['time', 'pressure'])

Inspect the schema

Check the columns and the types before you write a query:

SELECT * FROM read_netcdf('argo/**/*.nc') LIMIT 0;

Inspect a schema compares the _schema functions, SUMMARIZE, DESCRIBE and LIMIT 0, and says what each one costs.

Format details

Beacon streams the data and reads one chunk at a time. It reads a large file step by step. It does not load the whole file into memory.

Supported dialects:

• NetCDF4 (recommended)
• NetCDF3: a char* array with a string dimension such as STRLEN becomes a fixed-length string

Variable attributes

A NetCDF variable carries metadata attributes such as units, long_name and valid_min. Beacon shows these as extra columns. It uses dot notation: <variable>.<attribute>. The units attribute of the temperature variable becomes the column temperature.units.

An attribute column keeps the type from the file: string, integer, float and so on. Every query returns these columns next to the variable columns.

A file attribute has no variable prefix. Beacon shows it with a leading dot: .<attribute>. The global attribute source becomes the column .source.

SELECT temperature, "temperature.units", "temperature.long_name", ".source"
FROM read_netcdf(['argo/**/*.nc'])
LIMIT 1

Limitations:

• Beacon does not support user-defined types.
• Object storage authenticates on the default BEACON_NETCDF_USE_RUST_READER=true. On the netCDF-C library a bucket supports anonymous access only, because that library opens a file by URL and never sees the credential chain.

TIP

For a large NetCDF collection, convert the files into one Atlas collection. Atlas merges many NetCDF files into one array store with statistics. Beacon can then drop whole datasets and read only the arrays that you select. This is much faster than a scan of the original NetCDF files.

As an external table

CREATE EXTERNAL TABLE argo
STORED AS NC
LOCATION 'argo/**/*.nc'

See Create External Tables for the full DDL. See Data Sources for the full read model.

OPTIONS

STORED AS NC reads three keys:

Option	Type	Default	Description
read_dimensions	List of dimension names	The default grid of each file	The dimensions the table reads. Beacon returns a variable only if the list holds every dimension of that variable.
use_rust_reader	Boolean	true (BEACON_NETCDF_USE_RUST_READER)	Read with the pure-Rust reader. Set it to false to read with the netCDF-C library. That library needs anonymous access to a bucket.
enable_statistics	Boolean	true (BEACON_NETCDF_ENABLE_STATISTICS)	Accepted, and without effect today. Beacon rejects a value that is not a boolean, and then reads the server setting alone: ANALYZE FILES resolves a format per file, not per table. Set BEACON_NETCDF_ENABLE_STATISTICS to turn the column ranges off.

-- One grid, and the netCDF-C reader for this table alone
CREATE EXTERNAL TABLE argo
STORED AS NC
LOCATION 'argo/**/*.nc'
OPTIONS ('read_dimensions' 'time,latitude,longitude', 'use_rust_reader' 'false')

See OPTIONS for the rules that hold for every key. See Arrays to tables for the grid rule.