Aggregate Predicted Fishing Tracks into an H3 Effort Grid

Downloads all per-trip predicted fishing track files produced by predict_pds_tracks() and aggregates them into an H3 hexagonal grid of cumulative fishing effort. The result is uploaded as a versioned parquet file to the country-level cloud storage bucket.

Usage

aggregate_pds_effort(
  log_threshold = logger::DEBUG,
  h3_res = 9L,
  max_gap_hours = 0.25,
  gap_policy = c("cap", "drop"),
  batch_size = 8000L,
  package = "coasts"
)

Arguments

log_threshold: The logging threshold to use. Default is logger::DEBUG.
h3_res: Integer (0–15). H3 resolution level for the output grid. Default is 9 (~174 m edge length). Different resolutions write to separate cloud prefixes.
max_gap_hours, gap_policy: How silent stretches of a track are counted, passed to prepare_tracks_for_effort(). Both are recorded with the aggregation state: changing either makes every stored figure incomparable, so the grid is rebuilt from the predicted tracks.
batch_size: Integer. Most predicted-track objects to hold in memory at once. The function reads the whole backlog either way; this only sets how much of it is in flight at a time. The default of 8000 is far above a normal delta, so ordinary runs are a single batch and are unaffected. Lower it on a smaller machine. See the batching section below.
package: Name of the package whose inst/conf.yml to read. Defaults to "coasts". Pass your own package name when calling from a downstream package with a compatible configuration.

Value

Invisibly returns the merged H3 grid data frame (columns: h3_index, year, gear, country, fishing_hours, unique_trips, n_active_days, first_active_date, last_active_date, avg_fidelity_sum, n_trips_for_fidelity, fishing_pings), or NULL if there was nothing to process.

Details

Predicted track files contain fishing-only GPS points (columns: trip, timestamp, latitude, longitude). This function:

Lists all files under conf$pds$pds_tracks_predicted$file_prefix in the PDS bucket.
Downloads only new files (incremental via manifest) in parallel using furrr.
Prepares each track with prepare_tracks_for_effort(), which computes per-ping time intervals (dt_hours), assigns H3 cell indices, and records per-trip total hours for fidelity computation.
Removes tracks PDS has already delivered under another trip identifier, whole trips first (index_trip_duplicates()) and then the pings two partially overlapping trips share (drop_overlapping_pings()).
Adds what survives to the per-trip effort store (build_trip_effort()).
Derives the whole grid from that store (derive_effort_grid()) and uploads it.

Incremental state

Four side files sit next to the grid. aggregated_manifest.rds lists the objects already aggregated with the modification time they were aggregated at; aggregated_trips.rds lists every trip seen, with its device, its window, the fingerprint of its track and the trip it duplicates if it was dropped; aggregated_trip_effort.parquet holds fishing hours and pings per trip, cell, year, gear, country and day; aggregated_settings.rds records the gap handling those hours were measured under.

The store — not the published grid — is what carries between runs, and the grid is rebuilt from it in full every time. That is what makes a trip removable: PDS revises trips continuously (identifiers are retired, and the same identifier can come to describe a different window), so predict_pds_tracks() deletes and replaces files, and effort that was already counted has to be taken back out. A grid of running totals cannot do that; a per-trip store can, by dropping the affected trips' rows — together with any trip that was dropped as a duplicate of them, and any trip that might share pings with the incoming batch (recall_overlapping_trips()) — and re-reading only the handful of files involved. Deriving from the whole store each run also makes the result independent of how trips were batched, so unique_trips is a distinct count rather than a sum of per-batch counts.

A trip's rows are replaced rather than added to, so reading a file twice changes nothing. Everything is re-read from the predicted tracks only when the state is missing or written by an older schema, or when max_gap_hours / gap_policy differ from the values the stored hours were measured under — settings that leave no trace in the files themselves. A new model version needs no trigger of its own: predict_pds_tracks() deletes every old-version file and writes new ones, which the per-file comparison already sees, and it sees precisely which files moved rather than assuming all of them did.

Batching large backlogs

Peak memory is set by how many points are read at once, not by how many the store holds, and the two are normally unrelated: a daily run reads the delta since the last one — of the order of 300 objects and 100,000 points. A region registered for the first time breaks that assumption, because its whole history is new at once. Read in one go, 25 million points need roughly 25 GB and the run is killed by the operating system rather than failing in R.

So the backlog is read in batches of batch_size objects, and the loop is inside this function: there is one way to call it, and it does the whole backlog whatever size that is. A normal delta is smaller than a single batch and behaves exactly as it did before batching existed.

Each batch checkpoints the aggregation state before the next begins, so an interrupted backlog resumes from the last completed batch instead of starting over — which matters when the reading alone can run for hours. The grid is derived once at the end rather than per batch, since it is rebuilt from the whole store every time regardless.

Batching does not change the result. Trips that might share pings are recalled whatever batch they fall in (recall_overlapping_trips()), and the grid is derived from the entire store, so how the trips were divided leaves no trace. A store built over eleven batches and one built in a single pass agree on cell count, unique_trips and per-cell effort to floating-point summation order.

As a guide to sizing, predicted-track objects hold a few hundred points each, so the default 8,000 is roughly two million points, or about 2.7 GB. Lower it if the machine is smaller than that; there is no reason to raise it.

Grid schema

The grid is partitioned by year, gear, and country, so there may be several rows per (h3_index, year) — one per (gear, country) combination. gear and country come from the per-trip device metadata (trip → IMEI → Airtable pds_devices); trips with no device metadata are kept under gear = "unknown" / country = "unknown" so totals are preserved. Because each trip maps to exactly one gear and one country, summing over gear and country (and year) recovers the all-time, all-fleet totals — see plot_effort_map() for temporal maps. Primary effort columns:

fishing_hours: accumulated fishing time (sum of capped inter-ping intervals). This is the primary effort metric.
unique_trips: count of distinct trips contributing to the cell.
n_active_days: count of distinct calendar days with fishing activity.
first_active_date / last_active_date: date range for inferring the study period length (n_total_days) downstream.
avg_fidelity_sum: sum of per-trip fidelity values (fraction of each trip's total fishing hours spent in this cell). Divide by n_trips_for_fidelity to get avg_fidelity, between 0 and 1.
n_trips_for_fidelity: number of trips contributing to avg_fidelity_sum.
fishing_pings: raw GPS point count (retained for QA; not used as a primary metric because ping frequency is irregular).

Multi-resolution support: passing different h3_res values writes to separate cloud prefixes (e.g. predicted-pds-h3_grid_r9, predicted-pds-h3_grid_r7), so grids at multiple resolutions can coexist. Use rollup_h3_resolution() to derive coarser views from a stored fine grid, or pass a coarser h3_res directly to recompute from raw tracks. derive_fishing_grounds() can further roll up to any resolution before extracting contiguous fishing ground polygons.