Builds the pyramid of a5_pyramid() and writes it as a pyramid set: a
directory of small pyramid files, shards, and an index, set.json. Use
it where one file would be too large to open in a browser. A browser
reads a file's whole footer before it draws anything, and the footer
grows by a few hundred bytes a tile, so a single file of 150 million
leaves has a footer of about 20 MB. In a set, a view opens only the few
shards around it.
Usage
a5_write_pyramid_set(
x,
dest,
...,
lods = NULL,
tiers = NULL,
aggregate = c("mean", "rep_child"),
reducers = NULL,
weights = NULL,
types = NULL,
rep_sd = NULL,
codecs = NULL,
storage = NULL,
weights_type = NULL,
index = c("json", "parquet"),
compression = c("snappy", "zstd", "none"),
metadata = NULL,
threads = NULL
)Arguments
- x
A data frame with one a5R::a5_cell column, an a5R::a5_cell vector, or Arrow data with one
uint64column of cells: an ArrowTable,RecordBatch,RecordBatchReader,Datasetor query from the arrow package. The cells must share one resolution and appear once each; rows withNAcells are dropped. An a5R::a5_covering holds cells at mixed resolutions: convert it witha5R::a5_uncompact()first. Arrow data is read into the engine batch by batch without passing through R, so aDatasetover Parquet files (fromarrow::open_dataset()) builds without R holding the leaves.- dest
Directory to write the set to. It must not exist or be empty, so a set never mixes with another's files.
- ...
<
tidy-select> Value columns to carry into the pyramid. Default: every numeric column other than the cells. For an ArrowDatasetthat includes its partitioning columns, such asyearfromyear=2025/directories, so name the value columns there.- lods
Integer vector of the coarser LODs to build. Default: every LOD the
tiershold below the leaves, or, withouttiers, every resolution from 2 up to the leaves'.- tiers
A data frame with one row per tier and columns
lod_minandlod_max(the tier's LODs, inclusive),shard_resolution(from 0 up tolod_min), and optionallytile_size(default 16384) androw_group_size(defaultNA, one tile a row group; seea5_write_pyramid()). Between them the tiers must hold every level, the leaves included, and each must hold at least one. Default:a5_pyramid_set_tiers().- aggregate
The reducer for every value column:
"mean"(default) or"rep_child". See "Reducers".- reducers
Named character vector of reducers for single columns, overriding
aggregate, such asc(count = "sum", class = "mode").- weights
Name of a numeric column of non-negative leaf weights for
"mean","mode"and"rep_child", such as thenpixcolumn froma5px::a5_read_raster().NAweights count as 0, and leaves with weight 0 do not count. DefaultNULL: every leaf counts once.- types
Named character vector of storage types for single columns, such as
c(red = "uint8", ndvi = "float32"). See "Column types".- rep_sd
Named numeric vector of standard deviations for
"rep_child"columns, such asc(red = 60, green = 60). DefaultNULL: each column's standard deviation over the leaves, or 1 where that is 0 or undefined (a constant column). That depends on every leaf, so pyramids built from different parts of a region choose representatives differently even where their cells are the same; fix the standard deviations (from a whole-region build'sa5_pyramid_info()$columns$sd, say) for builds whose cells must agree.- codecs
Named character vector of codecs for
"mean"columns whose values are codes, such asc(embedding = "aef"). See "Codecs". DefaultNULL: every column stores its values.- storage
Named character vector of how to store vector columns:
"fixed_binary"(default), one fixed-length binary value a row, the smallest footer and the fastest decode; or"list", a Parquet list of the elements, dictionary encoded, which engines such as DuckDB read as an array column (TINYINT[]), as the a5embeddings store keeps its embeddings (format version 4). Such asc(embedding = "list").- weights_type
NULL(default), or a storage type (see "Column types") in which to keep each row's summedweightsat every level, as a column named as the weights, beside the value columns and not one of them: a reader that merges rows of one cell weights them by it (format version 4 in a file). Needsweights.- index
"json"(default) forset.json, or"parquet"forset.parquet, a shard table (format version 4; seea5_write_pyramid_set_table()).- compression
"snappy"(default),"zstd"or"none". hyparquet decodes snappy itself; zstd needs thehyparquet-compressorspackage in the browser.- metadata
Named character vector or list of strings to add to the file's key-value metadata, such as a renderer's settings as JSON. The keys
"a5pyramid","a5_index"and"ARROW:schema"are reserved.- threads
Number of threads, or
NULL(default) fora5pyramid_threads().
Tiers and shards
The LODs are split into tiers, each a range of LODs with a shard
resolution. A cell of a tier belongs to the shard keyed by the cell at
the shard resolution containing its centre
(a5R::a5_cell_to_supercell()), so every cell is in exactly one shard.
Each shard is one file holding its tier's LODs for the cells it owns, at
tier=<t>/shard=<cell>.parquet. The levels are built from all the
leaves before the pyramid is split, so a coarse cell near a shard's edge
summarises every leaf under it, whichever shard holds them.
a5_pyramid_set_tiers() gives the default tiers: three LODs a tier from
the leaves down, each keyed 12 resolutions coarser than its finest LOD,
and the remaining LODs in one tier keyed at resolution 0. A shard full of
data then holds about 22 million rows, and a map view meets at most a
few shards of the tier it draws at any zoom.
Files
A shard is an a5pyramid file in format version 2: as a file from
a5_write_pyramid(), with the index also recording the shard, and with
data_resolution the leaves' resolution even in shards of coarser
tiers. a5_pyramid_info() and a5_read_pyramid() read shards. A shard
can hold no cells at some LOD of its tier, never at all of them; shards
without cells are not written. set.json lists the tiers and every
shard with its path, rows, tiles, size, footer size and bounding box,
and is written last, so a set with an index is complete. A write that
fails or is interrupted removes the shards it wrote, and dest if it
created it, so it can be run again. The format is specified in
FORMAT.md in the package source.
See also
a5_pyramid_set_info() to read the index, a5_write_pyramid()
for a single file.
Examples
library(a5R)
centre <- a5_lonlat_to_cell(-3.19, 55.95, resolution = 10)
cells <- a5_uncompact(a5_spherical_cap(centre, 3000), resolution = 15)
x <- data.frame(cell = cells, value = runif(length(cells)))
dest <- tempfile()
tiers <- data.frame(
lod_min = c(2, 13), lod_max = c(12, 15),
shard_resolution = c(0, 10), tile_size = c(4096, 4096)
)
a5_write_pyramid_set(x, dest, tiers = tiers)
info <- a5_pyramid_set_info(dest)
info
#> ── a5pyramid set file29673b6b5f3a ──────────────────────────────────────────────
#> Size: 24.7 KB in 6 shards
#> A5 index: 1.0
#> Leaves: resolution 15
#> Tier 0: LODs 2-12, shards at resolution 0: 2 shards, 46 rows, 6.6 KB (footers
#> up to 3.5 KB)
#> Tier 1: LODs 13-15, shards at resolution 10: 4 shards, 1,380 rows, 18.1 KB
#> (footers up to 1.4 KB)
#> Columns: value <float64, mean>
#> Bbox: -21, 49.9853, 15, 71.6383
#> Written by: a5pyramid 0.0.0.9000, a5 crate 1.0.0
info$shards
#> # A tibble: 6 × 11
#> tier cell path rows tiles bytes footer_bytes xmin ymin xmax
#> <int> <a5_cell> <chr> <dbl> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 0 0200000000000000 tier… 1 1 1228 1093 -21 52.2 15
#> 2 0 1200000000000000 tier… 45 10 5361 3460 -13.2 50.0 -1.52
#> 3 1 6316008000000000 tier… 162 3 3202 1388 -3.22 56.0 -3.16
#> 4 1 63bb948000000000 tier… 977 3 9212 1398 -3.22 55.9 -3.14
#> 5 1 63bb958000000000 tier… 32 3 2153 1375 -3.15 56.0 -3.13
#> 6 1 63bb968000000000 tier… 209 3 3571 1386 -3.18 55.9 -3.12
#> # ℹ 1 more variable: ymax <dbl>
