Reduces leaves to exact partial sums at coarse levels, so that pyramids
of separate parts of a region, each built alone (a store's partitions),
can have coarse levels that span the parts. a5_pyramid_merge_partials()
adds the partial rows of every part and encodes the means: the rows that
a5_pyramid() gives for all the leaves at once.
Usage
a5_pyramid_partials(
x,
...,
lods,
key_resolution,
weights = NULL,
weights_type = "uint32",
types = NULL,
codecs = NULL,
threads = NULL
)Arguments
- x
A data frame with one a5R::a5_cell column, an a5R::a5_cell vector, or Arrow data with one
uint64column of cells: an ArrowTable,RecordBatch,RecordBatchReader,Datasetor query from the arrow package. The cells must share one resolution and appear once each; rows withNAcells are dropped. An a5R::a5_covering holds cells at mixed resolutions: convert it witha5R::a5_uncompact()first. Arrow data is read into the engine batch by batch without passing through R, so aDatasetover Parquet files (fromarrow::open_dataset()) builds without R holding the leaves.- ...
The vector columns to sum (tidyselect, as in
a5_pyramid()). Each is reduced by its mean; other columns do not merge and are refused.- lods
The coarse LODs, A5 resolutions below the leaves'.
- key_resolution
The resolution of each row's key, the supercell of its cell there: the shard its merged row belongs to. One for every LOD, or one for all, each no finer than its LOD.
- weights
The leaf weights column, every weight above zero (a mean over leaves that all weigh nothing has no partial form), or
NULLfor a weight of 1 each.- weights_type
The type the merged rows store their summed weights in, as
weights_typeina5_pyramid(); ignored withoutweights.- types
Named character vector of storage types for single columns, such as
c(red = "uint8", ndvi = "float32"). See "Column types".- codecs
Named character vector of codecs for
"mean"columns whose values are codes, such asc(embedding = "aef"). See "Codecs". DefaultNULL: every column stores its values.- threads
Number of threads, or
NULL(default) fora5pyramid_threads().
Value
An Arrow Table in ascending LOD and cell order, with the
columns lod (uint8), key and cell (uint64), ncell
(uint64), the summed weights named as weights (float64) with
weights, and each vector's sums named as the vector (a fixed-size list
of float64). Its schema metadata ("a5pyramid_partials") records the
vectors' types, sizes and codecs and the weights' name and type, which
a5_pyramid_merge_partials() needs: arrow::write_parquet() keeps it.
Details
A row holds a cell of a LOD that has a leaf (the cell containing the
leaf's centre, a5R::a5_cell_to_supercell()) and the sums over its
leaves: their number, their weights, and for each vector the weighted sum
of every element's value. With a codec the values are lifted as the
builder lifts them for a mean (for "aef", code * |code|), so with
integer weights the sums are integers, exact in doubles below 2^53, and
the merged means do not depend on how the leaves were split.
See also
a5_pyramid_merge_partials() to merge them,
a5_write_pyramid_level() to write the merged rows.
Examples
library(a5R)
centre <- a5_lonlat_to_cell(-3.19, 55.95, resolution = 9)
cells <- a5_uncompact(a5_spherical_cap(centre, 4000), resolution = 14)
emb <- matrix(sample(-128:127, length(cells) * 4, replace = TRUE), ncol = 4)
x <- tibble::tibble(cell = cells, embedding = emb, n = 1 + seq_along(cells) %% 5)
# two parts, built one at a time
west <- a5_cell_to_supercell(cells, 11) < a5_cell_to_supercell(cells[1], 11)
parts <- lapply(split(seq_along(cells), west), function(i) {
a5_pyramid_partials(x[i, ], embedding, lods = 10:12, key_resolution = 8,
weights = "n", codecs = c(embedding = "aef"))
})
merged <- a5_pyramid_merge_partials(do.call(arrow::concat_tables, unname(parts)))
merged
#> Table
#> 105 rows x 6 columns
#> $lod <uint8 not null>
#> $key <uint64 not null>
#> $cell <uint64 not null>
#> $ncell <uint32 not null>
#> $n <uint32 not null>
#> $embedding: fixed_size_list<element <int8 not null>[4] not null>
#>
#> See $metadata for additional Schema metadata
