Exploring the Data Lake (REST API)

Use these endpoints to discover what data is available on a running Beacon instance — without running a full query.

Concepts:

• Datasets — individual files (a single .nc file, a .parquet file, a Zarr group, etc.)
• Tables — named logical tables registered in Beacon, often spanning many datasets
• Schemas — Arrow field lists (name + type) describing the columns available for select and filter

System info

GET /api/info

Returns Beacon version, configuration summary, and registered table count.

Datasets

List datasets

GET /api/list-datasets

Deprecated alias

GET /api/datasets is a deprecated alias of /api/list-datasets. Use /api/list-datasets in new code.

Optional query parameters:

Parameter	Description
pattern	Glob to filter paths (e.g. *.nc, **/*.parquet)
offset	Pagination offset
limit	Pagination limit

GET /api/list-datasets?pattern=argo/**/*.nc&limit=50&offset=0

Dataset count

GET /api/total-datasets

Dataset schema

Returns the Arrow schema (fields + types) for a single path:

GET /api/dataset-schema?file=argo/profile_001.nc

To infer a merged schema across multiple files using a glob:

GET /api/dataset-schema?file=argo/**/*.nc

The response contains an Arrow schema JSON. Column names are under .fields[].name.

Tables

List tables

GET /api/tables

Default table

Beacon uses this table when a query omits from:

GET /api/default-table

Table schema

GET /api/table-schema?table_name=default

Default table schema

The Arrow schema of the default table (the one queried when a request omits from):

GET /api/default-table-schema

Deprecated alias

GET /api/query/available-columns is a deprecated endpoint that returns only the column names of the default table schema. Use /api/default-table-schema in new code.

All tables with schemas

Convenient for UI discovery, but can be slow on large installations:

GET /api/tables-with-schema

Table configuration

Shows how a table was constructed — paths, file format, statistics settings, etc. This endpoint is admin-only (see Admin) and requires HTTP Basic auth; unauthenticated requests get 401. Sensitive options such as SQL-database passwords are redacted (the secret field is returned as ***).

GET /api/admin/table-config?table_name=default
Authorization: Basic <base64(username:password)>

Functions

List all registered DataFusion scalar functions:

GET /api/functions

List all registered Beacon table functions (e.g. read_netcdf, read_zarr):

GET /api/table-functions

See the Function Reference for descriptions and signatures.

Table lifecycle

Table lifecycle is SQL-only. Create, replace, or remove tables by sending SQL DDL to the query endpoint:

POST /api/query
Content-Type: application/json

{ "sql": "CREATE EXTERNAL TABLE argo STORED AS PARQUET LOCATION 'argo/'" }

POST /api/query
Content-Type: application/json

{ "sql": "DROP TABLE argo" }

Write operations require the admin credentials (BEACON_ADMIN_USERNAME / BEACON_ADMIN_PASSWORD) via HTTP basic auth; anonymous requests are read-only.

Admin

All /api/admin/* endpoints require the admin credentials (BEACON_ADMIN_USERNAME / BEACON_ADMIN_PASSWORD) via HTTP basic auth; unauthenticated requests get 401.

Creating, replacing, and removing tables can still be done through authenticated SQL DDL on /api/query (see Table lifecycle above). In addition, these dedicated, JSON-typed admin endpoints are available:

Method	Path	Purpose
GET	/api/admin/check	Connectivity check; returns { "is_admin": true }
GET	/api/admin/table-config	Inspect a table's storage format and configuration
POST	/api/admin/crawlers	Define (or replace) a crawler
GET	/api/admin/crawlers	List defined crawlers
GET	/api/admin/crawlers/{name}	Get one crawler (or 404)
POST	/api/admin/crawlers/{name}/run	Run a crawler once; returns its crawl report
DELETE	/api/admin/crawlers/{name}	Drop a crawler (crawled tables are left in place)
POST	/api/admin/external-tables	Create an external table from structured fields

Every example below sends the credentials via HTTP Basic auth (Authorization: Basic <base64(username:password)>); the header is omitted from the snippets after the first for brevity.

Check that your credentials are accepted:

GET /api/admin/check
Authorization: Basic <base64(username:password)>

Inspect a table's storage format and configuration (sensitive options such as SQL-database passwords are returned as ***):

GET /api/admin/table-config?table_name=default

Create a crawler (the structured equivalent of CREATE CRAWLER):

POST /api/admin/crawlers
Authorization: Basic <base64(username:password)>
Content-Type: application/json

{ "name": "argo", "target_prefix": "argo/", "format_filter": ["parquet"],
  "schedule_secs": 900, "table_naming": "crawler_prefixed" }

Create an external table (the structured equivalent of CREATE EXTERNAL TABLE):

POST /api/admin/external-tables
Authorization: Basic <base64(username:password)>
Content-Type: application/json

{ "name": "observations", "file_type": "PARQUET", "location": "obs/",
  "partition_cols": ["year", "month"] }

List the defined crawlers, or fetch a single one by name:

GET /api/admin/crawlers

GET /api/admin/crawlers/argo

Run a crawler once on demand (returns its crawl report):

POST /api/admin/crawlers/argo/run

Drop a crawler (its already-crawled tables are left in place):

DELETE /api/admin/crawlers/argo

OpenAPI

This page is a curated subset. The complete, always-current request and response shapes are generated from the server itself:

• Swagger UI: /swagger
• Scalar UI: /scalar/
• OpenAPI document: /openapi.json