Parquet

Read the files

read_parquet(glob_paths)

SELECT * FROM read_parquet('obs/**/*.parquet') LIMIT 100

Inspect the schema

Check the columns and the types before you write a query:

SELECT * FROM read_parquet('obs/*.parquet') LIMIT 0;

Inspect a schema compares the _schema functions, SUMMARIZE, DESCRIBE and LIMIT 0, and says what each one costs.

Format details

DataFusion reads Parquet directly. Parquet suits analytical work, because it stores data by column and supports predicate pushdown.

• Beacon supports column pruning and predicate pushdown in full.
• Beacon supports Hive-style directory partitions. Use PARTITIONED BY on an external table.
• Beacon reads files from DuckDB, Spark, pandas and similar tools.

As an external table

CREATE EXTERNAL TABLE ocean_profiles
STORED AS PARQUET
LOCATION 'profiles/'

Point at a folder. Beacon then finds every .parquet file under it. You can also give the glob:

CREATE EXTERNAL TABLE ocean_profiles
STORED AS PARQUET
LOCATION 'profiles/**/*.parquet'

See Create External Tables for the full DDL. See Data Sources for the full read model.