Builds the pyramid of a5_pyramid() and writes it, leaves included, to
one Parquet file laid out for a browser to read with HTTP range requests,
as with hyparquet. The build and the write run in Rust, so the levels
never pass through R.
Usage
a5_write_pyramid(
x,
dest,
...,
lods = NULL,
aggregate = c("mean", "rep_child"),
reducers = NULL,
weights = NULL,
types = NULL,
rep_sd = NULL,
codecs = NULL,
tile_size = 4096L,
row_group_size = NULL,
storage = NULL,
weights_type = NULL,
compression = c("snappy", "zstd", "none"),
metadata = NULL,
threads = NULL
)Arguments
- x
A data frame with one a5R::a5_cell column, an a5R::a5_cell vector, or Arrow data with one
uint64column of cells: an ArrowTable,RecordBatch,RecordBatchReader,Datasetor query from the arrow package. The cells must share one resolution and appear once each; rows withNAcells are dropped. An a5R::a5_covering holds cells at mixed resolutions: convert it witha5R::a5_uncompact()first. Arrow data is read into the engine batch by batch without passing through R, so aDatasetover Parquet files (fromarrow::open_dataset()) builds without R holding the leaves.- dest
Path of the Parquet file to write. An existing file is replaced. The file is written under a temporary name and renamed when complete, so a failed or interrupted write leaves no partial file.
- ...
<
tidy-select> Value columns to carry into the pyramid. Default: every numeric column other than the cells. For an ArrowDatasetthat includes its partitioning columns, such asyearfromyear=2025/directories, so name the value columns there.- lods
Integer vector of the coarser LODs to build, each below the leaves' resolution; the leaves are always kept. Default: every resolution from 2 up to the leaves'. Resolutions 0 and 1 hold the 12 and 60 largest cells, which no map view draws.
- aggregate
The reducer for every value column:
"mean"(default) or"rep_child". See "Reducers".- reducers
Named character vector of reducers for single columns, overriding
aggregate, such asc(count = "sum", class = "mode").- weights
Name of a numeric column of non-negative leaf weights for
"mean","mode"and"rep_child", such as thenpixcolumn froma5px::a5_read_raster().NAweights count as 0, and leaves with weight 0 do not count. DefaultNULL: every leaf counts once.- types
Named character vector of storage types for single columns, such as
c(red = "uint8", ndvi = "float32"). See "Column types".- rep_sd
Named numeric vector of standard deviations for
"rep_child"columns, such asc(red = 60, green = 60). DefaultNULL: each column's standard deviation over the leaves, or 1 where that is 0 or undefined (a constant column). That depends on every leaf, so pyramids built from different parts of a region choose representatives differently even where their cells are the same; fix the standard deviations (from a whole-region build'sa5_pyramid_info()$columns$sd, say) for builds whose cells must agree.- codecs
Named character vector of codecs for
"mean"columns whose values are codes, such asc(embedding = "aef"). See "Codecs". DefaultNULL: every column stores its values.- tile_size
The most rows in a tile. Larger tiles mean fewer requests and a smaller footer; smaller tiles mean less data fetched outside the view. Default: 4096.
- row_group_size
NULL(default) for one tile a row group, or the most rows in a row group of several tiles, at leasttile_size(see "File layout"). The embeddings store uses 131072 with tiles of 16384.- storage
Named character vector of how to store vector columns:
"fixed_binary"(default), one fixed-length binary value a row, the smallest footer and the fastest decode; or"list", a Parquet list of the elements, dictionary encoded, which engines such as DuckDB read as an array column (TINYINT[]), as the a5embeddings store keeps its embeddings (format version 4). Such asc(embedding = "list").- weights_type
NULL(default), or a storage type (see "Column types") in which to keep each row's summedweightsat every level, as a column named as the weights, beside the value columns and not one of them: a reader that merges rows of one cell weights them by it (format version 4 in a file). Needsweights.- compression
"snappy"(default),"zstd"or"none". hyparquet decodes snappy itself; zstd needs thehyparquet-compressorspackage in the browser.- metadata
Named character vector or list of strings to add to the file's key-value metadata, such as a renderer's settings as JSON. The keys
"a5pyramid","a5_index"and"ARROW:schema"are reserved.- threads
Number of threads, or
NULL(default) fora5pyramid_threads().
File layout
Each level is split into tiles of at most tile_size rows, and each tile
is one Parquet row group. Levels follow each other in ascending LOD
order, coarsest first. A tile is a run of consecutive cell ids: A5 ids
follow a space-filling curve, so such a run is a compact patch of the
globe. A level is split down the A5 index hierarchy only where a run
holds more than tile_size cells, which makes tiles coarse where data
is sparse and fine where it is dense, and keeps every level sorted by
cell id.
With row_group_size, each level is split down the hierarchy into row
groups of at most row_group_size rows instead, and each row group into
tiles of tile_size rows (the last shorter), each one data page of
every column. Engines that read whole row groups, such as DuckDB over
S3, then make fewer requests, while a browser still reads one tile at a
time: the file carries the Parquet page index, which locates each tile's
pages. Such a file is in format version 4.
The file's key-value metadata holds a JSON index under "a5pyramid":
the leaves' resolution, the value columns with their types and reducers,
each level's LOD, row count and tiles, and the bounding box of each tile,
computed from its cells' boundaries. A reader fetches the footer, picks
the level for the zoom and the tiles whose boxes meet the view, and
decodes only those row groups. "a5_index" records the A5 index
version of the ids, "1.0", as across the a5R stack.
The columns are cell (uint64), lod (uint8), ncell (uint32)
and the value columns with their types, NA stored as null. cell and
ncell are delta encoded and lod dictionary encoded; Parquet
statistics are kept for cell and lod only, so engines such as DuckDB
can skip row groups by LOD or cell while the footer stays small. The
format is specified in FORMAT.md in the package source
(src/rust/a5pyramid/FORMAT.md), with a reading recipe for hyparquet.
See also
a5_pyramid_info() to inspect the file, a5_read_pyramid() to
read levels back, a5_pyramid_arrow() for the same tiles in memory.
Examples
library(a5R)
centre <- a5_lonlat_to_cell(-3.19, 55.95, resolution = 10)
cells <- a5_uncompact(a5_spherical_cap(centre, 3000), resolution = 15)
x <- data.frame(cell = cells, value = runif(length(cells)))
path <- tempfile(fileext = ".parquet")
a5_write_pyramid(x, path, types = c(value = "float32"), tile_size = 1000)
a5_pyramid_info(path)
#> ── a5pyramid file file29677cba0aed.parquet ─────────────────────────────────────
#> Size: 13.2 KB (footer 4.8 KB)
#> A5 index: 1.0
#> Leaves: resolution 15, 1,024 cells
#> LODs: 2-15 (14 levels)
#> Rows: 1,426
#> Tiles: 15 row groups (tile size 1000)
#> Columns: value <float32, mean>
#> Bbox: -3.2194, 55.9355, -3.127, 56.0338
#> Written by: a5pyramid 0.0.0.9000, a5 crate 1.0.0
