Skip to contents

The Automated Pipeline

Functions that drive the bi-daily GitHub Actions workflow.

ingest_assets()
Ingest Fisheries Asset Metadata
enrich_taxa()
Enrich Taxa with FishBase and SeaLifeBase Biological Data
ingest_pds_trips()
Ingest Pelagic Data Systems (PDS) Trip Data
merge_survey_trips()
Merge GPS Trips with Survey Data
ingest_pds_tracks()
Ingest Pelagic Data Systems (PDS) Track Data
preprocess_pds_tracks()
Preprocess Pelagic Data Systems (PDS) Track Data
predict_pds_tracks()
Predict Fishing Activity from PDS Tracks
aggregate_pds_effort()
Aggregate Predicted Fishing Tracks into an H3 Effort Grid
model_cpue()
Run the Full Spatial CPUE Pipeline

Data Export & Web Readiness

Preparing the analyzed data for the Peskas portal and web maps.

export_geos()
Export Geospatial Data and Regional Metrics to MongoDB
export_fishers_stats()
Export Fisher Performance Statistics to MongoDB
export_portal()
Export Summary Data to MongoDB
export_pds_spatial()
Export Spatial Pipeline Outputs as Web-Ready JSON / GeoJSON

Spatial Utilities

Under-the-hood tools for gridding, mapping, and metric conversion.

assign_h3_indices()
Assign H3 Hexagon Indices to GPS Points
aggregate_h3_effort()
Aggregate Fishing Effort by H3 Hexagon
rollup_h3_resolution()
Roll Up H3 Fishing Effort to a Coarser Resolution
create_spatial_grid()
Convert an H3 Effort Summary to a Spatial Grid
derive_fishing_grounds()
Derive Fishing Ground Polygons from an H3 Effort Grid

Survey & Fleet Analysis

Functions for survey summarization and fleet activity estimation.

calculate_catch_record_usage()
Calculate catch record usage by week
calculate_district_totals()
Calculate District-Level Total Catch and Revenue
calculate_fishery_metrics()
Calculate Fishery Metrics
calculate_monthly_trip_stats()
Calculate Monthly Trip Statistics by District
estimate_fleet_activity()
Estimate Fleet-Wide Activity from Sample Data
summarize_data()
Summarize WorldFish Survey Data
generate_fleet_analysis()
Generate Complete Fleet Activity Analysis Pipeline
prepare_boat_registry()
Prepare Boat Registry Data from Metadata
process_trip_data()
Process Trip Data with District Information
create_usage_heatmap()
Create usage heatmap plot
create_usage_table()
Create interactive reactable

API & Data Ingestion (PDS & Airtable)

Functions interacting with Pelagic Data Systems, Airtable, and KoboToolbox.

airtable_to_df()
Get All Records from Airtable with Pagination
bulk_update_airtable()
Bulk Update Multiple Airtable Records
device_sync()
Sync Data with Airtable (Update + Create)
df_to_airtable()
Create New Airtable Records
update_airtable_record()
Update Single Airtable Record
get_writable_fields()
Get Writable Fields from Airtable Table
pelagic_auth()
Authenticate with Pelagic Analytics API
pelagic_refresh_token()
Refresh Authentication Token
get_pelagic_boats()
Get Boats from Pelagic Analytics API (with server-side filtering)
get_pelagic_devices()
Get Devices from Pelagic Analytics API (with server-side filtering)
ingest_pelagic_boats()
Ingest Pelagic Boats Data and Sync to Airtable
sync_device_users()
Sync Device Users to MongoDB and Update Airtable
get_trips()
Retrieve Trip Details from Pelagic Data API
get_trip_points()
Get Trip Points from Pelagic Data Systems API

Other Utilities and Internals

Additional exported helper functions.

add_version()
Add timestamp and sha string to a file name
aggregate_daily_effort()
Aggregate GPS Points to a Reference Grid
aggregate_h3_effort()
Aggregate Fishing Effort by H3 Hexagon
aggregate_pds_effort()
Aggregate Predicted Fishing Tracks into an H3 Effort Grid
airtable_to_df()
Get All Records from Airtable with Pagination
assign_h3_indices()
Assign H3 Hexagon Indices to GPS Points
backup_tracks()
Backup Pelagic Tracks (Fallback)
bulk_update_airtable()
Bulk Update Multiple Airtable Records
calculate_catch_record_usage()
Calculate catch record usage by week
calculate_district_totals()
Calculate District-Level Total Catch and Revenue
calculate_fishery_metrics()
Calculate Fishery Metrics
calculate_monthly_trip_stats()
Calculate Monthly Trip Statistics by District
cloud_storage_authenticate()
Authenticate to a Cloud Storage Provider
create_reference_grid()
Create a Deterministic Reference Grid
create_spatial_grid()
Convert an H3 Effort Summary to a Spatial Grid
create_usage_heatmap()
Create usage heatmap plot
create_usage_table()
Create interactive reactable
derive_fishing_grounds()
Derive Fishing Ground Polygons from an H3 Effort Grid
device_sync()
Sync Data with Airtable (Update + Create)
df_to_airtable()
Create New Airtable Records
download_cloud_file()
Download File from Cloud Storage
download_parquet_from_cloud()
Download Parquet File from Cloud Storage
enrich_taxa()
Enrich Taxa with FishBase and SeaLifeBase Biological Data
estimate_fleet_activity()
Estimate Fleet-Wide Activity from Sample Data
expand_taxonomic_info()
Expand Taxa to FishBase / SeaLifeBase Species Matches
export_fishers_stats()
Export Fisher Performance Statistics to MongoDB
export_frame_data()
Export Fishing Frame Gear Composition as Web-Ready JSON
export_geos()
Export Geospatial Data and Regional Metrics to MongoDB
export_pds_spatial()
Export Spatial Pipeline Outputs as Web-Ready JSON / GeoJSON
export_portal()
Export Summary Data to MongoDB
fetch_asset()
Fetch and Filter Asset Data from Airtable
fetch_assets()
Fetch Multiple Asset Tables from Airtable
generate_app_usage_report()
Generate complete app usage report
generate_fleet_analysis()
Generate Complete Fleet Activity Analysis Pipeline
generate_track_summaries()
Generate Grid Summaries for Track Data
get_fisher_summaries()
Aggregate Fisher Catch Events by Trip
get_kobo_data()
Retrieve Data from Kobotoolbox API
get_pelagic_boats()
Get Boats from Pelagic Analytics API (with server-side filtering)
get_pelagic_devices()
Get Devices from Pelagic Analytics API (with server-side filtering)
get_trip_points()
Get Trip Points from Pelagic Data Systems API
get_trips()
Retrieve Trip Details from Pelagic Data API
get_writable_fields()
Get Writable Fields from Airtable Table
ingest_assets()
Ingest Fisheries Asset Metadata
ingest_pds_tracks()
Ingest Pelagic Data Systems (PDS) Track Data
ingest_pds_trips()
Ingest Pelagic Data Systems (PDS) Trip Data
ingest_pelagic_boats()
Ingest Pelagic Boats Data and Sync to Airtable
mdb_collection_pull()
Retrieve Data from MongoDB
mdb_collection_push()
Push Data to MongoDB Collection
merge_survey_trips()
Merge GPS Trips with Survey Data
model_cpue()
Run the Full Spatial CPUE Pipeline
pelagic_auth()
Authenticate with Pelagic Analytics API
pelagic_refresh_token()
Refresh Authentication Token
plot_cpue_map()
Create an Interactive Leaflet Map of Spatial CPUE by Species
plot_effort_map()
Create an Interactive Leaflet Map of Fishing Effort by Year
predict_pds_tracks()
Predict Fishing Activity from PDS Tracks
prep_fishing_points()
Project Fishing GPS Points to a Metric CRS
prepare_boat_registry()
Prepare Boat Registry Data from Metadata
prepare_usage_summary()
Prepare usage summary table
preprocess_pds_tracks()
Preprocess Pelagic Data Systems (PDS) Track Data
preprocess_track_data()
Preprocess Track Data into Spatial Grid Summary
process_trip_data()
Process Trip Data with District Information
pull_catch_events()
Pull catch events data from MongoDB
pull_users_data()
Pull users data from MongoDB
read_config()
Read configuration file
rollup_h3_resolution()
Roll Up H3 Fishing Effort to a Coarser Resolution
summarize_data()
Summarize WorldFish Survey Data
sync_device_users()
Sync Device Users to MongoDB and Update Airtable
update_airtable_record()
Update Single Airtable Record
upload_cloud_file()
Upload File to Cloud Storage
upload_parquet_to_cloud()
Upload Data as Parquet File to Cloud Storage