a5pyramid (development version)
-
Partial rows for pyramids of a region built in parts (a store’s partitions, each built alone), whose coarse levels span the parts:
-
a5_pyramid_partials()reduces leaves to exact sums at coarse LODs: for each cell, its leaves, their summed weights, and each mean vector’s weighted sums of lifted values (the codec’s lift,code * |code|for"aef"), keyed by the cell’s supercell at a key resolution. An Arrow table whose schema metadata ("a5pyramid_partials") records how to encode it; Parquet keeps it. -
a5_pyramid_merge_partials()adds the rows of every part, cell by cell, optionally for some keys only (one shard), reading Arrow tables, readers or datasets batch by batch, and encodes each mean asa5_pyramid()does. The result is the rowsa5_pyramid()gives for all the leaves at once, ready fora5_write_pyramid_level(). - With integer weights the sums are exact, so the merged codes do not depend on how the leaves were split. Only mean vectors merge; every weight must be above zero.
-
a5R comes from CRAN:
Remotesno longer namesbelian-earth/a5R, now that a5R 1.0.0 is released. An unpinned remote is a request for a5R’s default branch, and pak refused to install a5pyramid beside any other pinned a5R ref (belian-production’s base image pins a5R by commit).-
The rest of format version 4, for one store that engines and browsers read alike (belian-production
docs/store-v4.md):-
List vectors:
storage = c(embedding = "list")stores a vector as a Parquet list of its elements, dictionary encoded, which DuckDB reads asTINYINT[]; Arrow list columns (DuckDB’s output) are accepted as vectors, each row checked fordimelements.columns$storagein the info functions. -
A weights column:
weights_typekeeps each row’s summedweightsat every level, named as the weights and not a value column, so a reader can merge rows of one cell (weightsin the info functions). -
a5_write_pyramid_level()writes one level of precomputed rows as a file, optionally a shard with anownercolumn (each row cell’s supercell at the shard resolution) for a partition’s partial rows, withoutlod(one level) and withoutncell(leaves). Rows are sorted and checked, not built. In the crate:Pyramid::from_rows(),TileOptions::owner,lod_column,ncell_column. -
Shard tables:
a5_write_pyramid_set(index = "parquet")writesset.parquet, one row per shard with the set’s fields in the footer, boxes unwrapped;a5_write_pyramid_set_table()writes one for shards written apart, listing empty shards and carrying the writer’s own columns.a5_pyramid_set_info()reads both forms. In the crate:write_set_table(),read_set_table(),SetMeta::assemble(),SetIndexFormat;ShardEntry::pathandbboxare optional. - FORMAT.md: “Partial rows” (draw every row; merge by weights) and “Shard tables”.
-
List vectors:
Row groups of several tiles (format version 4):
row_group_sizeina5_write_pyramid()and in a set’stierssplits each level down the hierarchy into row groups of at most that many rows, and each row group into tiles oftile_sizerows, each one data page of every column chunk. The file carries the Parquet page index (an offset index for every column, a column index forcell), and the writer checks every page against the tiles before the file is renamed into place. Engines that read whole row groups make fewer requests: the embeddings store’s prototype replay over S3 ran 27 to 41% slower with 16,384-row groups than with 122,880, while a browser reading 16,384-row pages of 131,072-row groups fetched the same bytes as with 16,384-row groups. Withoutrow_group_sizefiles are unchanged.a5_pyramid_info()gainsrow_group_size,row_groupsandtiles$row_group, andtiles$bytescounts a tile’s pages; data pages are pinned to version 1. In the crate:TileOptions::row_group_size,Tier::row_group_size,PyramidMeta::row_group_size,PyramidMeta::locate_tiles(),TiledPyramid::row_groups(),PyramidFile::tiles(),tile_bytes()andparquet_metadata();PyramidFile::read()takes tiles, andWriteSummarygainstiles.Codecs:
codecs = c(embedding = "aef")gives ameancolumn of codes the mean of the values they stand for. Each leaf’s code is decoded, the (weighted) mean taken and the result encoded, so a coarse cell holds the code of the mean rather than the mean of the codes, which differ for a nonlinear code."aef"is the AlphaEarth Foundations int8 code, encoded as the a5embeddings store encodes its cell means (rounded half away from zero, clamped to -128 to 127), taking the mean onx * |x|so that whole codes and weights sum exactly and a mean half way between two codes rounds away from zero in any order. The index records each column’scodec(columns$codecin the info functions), an optional field added without a format version change. In the crate:Codec,ColumnSpec::codec,VectorSpec::codec.Vector columns can be reduced by
mean, element by element, as well asrep_child:reducers = c(embedding = "mean"). Vectors stayrep_childby default, whateveraggregatesays, and ameanvector takes no part in choosing representatives. In the crate:VectorSpec::reducer,VectorColumn::with_reducer().Vector columns (format version 3): a fixed number of
int8oruint8values per cell, such as a 64-element embedding, from an integer matrix column of a data frame or a fixed-size list (DuckDB’sTINYINT[64]) in Arrow data. The engine holds one byte an element; files store the vector as one fixed-length binary column, so its footer and requests grow like one column’s, not 64. Vectors are reduced byrep_childwith one standard deviation over all their elements (fixed byrep_sd), together with any scalarrep_childcolumns.a5_pyramid()anda5_read_pyramid()return them as integer matrix columns;columns$dimin the info functions. Files and sets without vector columns keep versions 1 and 2. In the crate:VectorColumn,VectorSpec,Leaves::vectors,Level::vectors.Pyramid sets:
a5_write_pyramid_set()writes one pyramid as a directory of small files, shards, with an index,set.json, for data too large for one file to open in a browser, whose footer grows by about 365 bytes a tile (about 20 MB for 150 million leaves). The LODs are split into tiers; a cell belongs to the shard keyed by its supercell at the tier’s shard resolution, so every cell is in exactly one shard, and levels are built from all the leaves before the split.a5_pyramid_set_tiers()gives the default tiers (three LODs a tier, shards 12 resolutions coarser than the tier’s finest LOD, so a view meets a few shards at any zoom), anda5_pyramid_set_info()reads the index. A write that fails or is interrupted removes what it wrote, so it can be run again into the same directory. In the crate:write_set(),SetMeta,Tieranddefault_tiers().Format version 2 (
FORMAT.md, “Pyramid sets”): shards are a5pyramid files whose index records theshard(tier, shard cell, resolution), whose finest level need not be the leaves, and whosebboxcovers every tile rather than only the leaves. A writer records the oldest version a file follows, so single files are still version 1 and a5view reads them as before; readers of this version read both.a5_pyramid_info()returns theshardand prints a shard as one.Arrow data as input:
xmay be an ArrowTable,RecordBatch,RecordBatchReader,Datasetor query with oneuint64cell column. It is read into the engine batch by batch through the Arrow C stream interface, without passing through R, so a build fromarrow::open_dataset()over Parquet files holds no copy of the leaves in R; value columns default to their own Arrow types.rep_sdfixes the standard deviations that scale"rep_child"columns, so pyramids of separate parts of a region choose a cell’s representative from its own leaves alone. The standard deviations used, given or computed, are recorded in the index (sdincolumns) and returned bya5_pyramid_info()anda5_pyramid_set_info().Builds need less memory: sorting the leaves rewrites the cells in place and gathers one column at a time, levels and shards are grouped without holding their row pairs twice, and shard rows are grouped by a counting sort (4 bytes a row instead of 16). The Sabah AEF set (149.6 million res-18 leaves, three bands) peaks at 13.3 GB from a data frame, down from 21.2 GB, and at 9.3 GB from an Arrow
Dataset, with identical shards.First version: a Rust engine, with R bindings, that builds level-of-detail pyramids of A5 cells and writes them to one Parquet file for browser rendering. It replaces the R implementation in a5view 0.3.
a5_pyramid()builds the levels and returns them as a tibble;a5_write_pyramid()builds and writes them to Parquet in Rust. Each coarser level groups the leaves by supercell (a5R::a5_cell_to_supercell()), from the leaves directly, and reduces each column by"mean","sum","min","max","mode"or"rep_child", optionally weighted."rep_child"scales each column by its standard deviation, so units do not decide the representative, and breaks ties by lowest cell id; a5view used unscaled distances and let rounding break ties. Columns take a storage type ("uint8"to"float64") shared by every level.a5_pyramid_arrow()returns the tiles of a pyramid in memory as an Arrow table, one record batch per tile, with the index in the schema metadata: the file’s layout without the file. The tiles cross from Rust through the Arrow C stream interface, so no value passes through R. Record batchiholds the rows of row groupiof the same pyramid written witha5_write_pyramid().a5_pyramid_info()reads a file’s index from its footer, with atilestable of every tile’s level, rows, bytes and box, anda5_read_pyramid()reads levels within a bounding box, as a renderer does.a5pyramid_threads()sets the session’s thread count, by default every available CPU. Results do not depend on it, and builds can be interrupted.-
The file format (
src/rust/a5pyramid/FORMAT.md) differs from a5view’s:- tiles are runs of consecutive cell ids split along the A5 hierarchy to at most
tile_sizerows, so sparse data no longer makes many tiny row groups, and each tile is one row group; - row group boxes come from cell boundaries rather than padded centroids, and cross the antimeridian (
west > east) instead of widening to the whole globe; - the index is one JSON object under the key
"a5pyramid", with the levels, the value columns, and each tile’s box as flat arrays. It names tilestilesandtile_bbox, the same in the Parquet file and the Arrow form (row_groupsandrow_group_bboxbefore 2026-11-09, which are still read); - columns are
cell(uint64),lod,ncelland the value columns under their own names;cellis delta encoded, which takes it from 8 bytes per row to about 3.
- tiles are runs of consecutive cell ids split along the A5 hierarchy to at most
On 439,276 cells, writing a pyramid takes 0.48 s on 20 threads against 4.06 s with a5view 0.3, and the file is 28% smaller.
The engine is the
a5pyramidRust crate insrc/rust/a5pyramid, with no R dependency, to be published once the format settles. The R bindings are a separate crate in the same workspace. ItsTiledPyramidholds the tiles as Arrow record batches and streams them as an arrow-rsRecordBatchReader;write_parquet()writes them as row groups.
