Package index • peskas.mozambique.data.pipeline

Workflow

These are arguably the most important functions in the package. Each of these functions executes a step in the data pipeline.

export_landings(): Export Processed Landings Data
export_validation_flags(): Export Validation Flags to MongoDB
get_validation_status(): Get Validation Status from KoboToolbox
ingest_landings_adnap(): Download and Process ADNAP Surveys from Kobotoolbox
ingest_landings_lurio(): Download and Process Lurio Surveys from Kobotoolbox
ingest_pds_tracks(): Ingest Pelagic Data Systems (PDS) Track Data
ingest_pds_trips(): Ingest Pelagic Data Systems (PDS) Trip Data
preprocess_landings_adnap(): Preprocess ADNAP Landings Data
preprocess_landings_lurio(): Preprocess Lurio Landings Data
preprocess_pds_tracks(): Preprocess Pelagic Data Systems (PDS) Track Data
summarize_data(): Summarize WorldFish Survey Data
sync_validation_submissions(): Synchronize Validation Statuses with KoboToolbox
update_validation_status(): Update Validation Status in KoboToolbox
validate_surveys_adnap(): Validate ADNAP Survey Data
validate_surveys_lurio(): Validate Lurio Survey Data

Cloud Storage

Functions that interact with cloud storage providers.

cloud_object_name(): Generate Cloud Object Name
cloud_storage_authenticate(): Authenticate to a Cloud Storage Provider
download_cloud_file(): Download File from Cloud Storage
download_parquet_from_cloud(): Download Parquet File from Cloud Storage
get_metadata(): Get metadata tables
mdb_collection_pull(): Retrieve Data from MongoDB
mdb_collection_push(): Upload Data to MongoDB and Overwrite Existing Content
upload_cloud_file(): Upload File to Cloud Storage
upload_parquet_to_cloud(): Upload Data as Parquet File to Cloud Storage

Ingestion

Functions dedicated to the ingestion module

get_kobo_data(): Retrieve Data from Kobotoolbox API
get_trip_points(): Get Trip Points from Pelagic Data Systems API
get_trips(): Retrieve Trip Details from Pelagic Data API
ingest_landings_adnap(): Download and Process ADNAP Surveys from Kobotoolbox
ingest_landings_lurio(): Download and Process Lurio Surveys from Kobotoolbox
ingest_pds_tracks(): Ingest Pelagic Data Systems (PDS) Track Data
ingest_pds_trips(): Ingest Pelagic Data Systems (PDS) Trip Data

Preprocessing

Functions dedicated to the preprocessing module

calculate_catch_adnap(): Calculate Catch Weight from Length-Weight Relationships or Bucket Measurements
calculate_catch_lurio(): Calculate Catch Weight from Length-Weight Relationships or Bucket Measurements
calculate_fishery_metrics(): Calculate Fishery Metrics
fetch_asset(): Fetch and Filter Asset Data from Airtable
fetch_assets(): Fetch Multiple Asset Tables from Airtable
generate_track_summaries(): Generate Grid Summaries for Track Data
getLWCoeffs(): Get Length-Weight Coefficients and Morphological Data for Species
get_airtable_form_id(): Get Airtable Form ID from KoBoToolbox Asset ID
get_fao_groups(): Extract and Format FAO Taxonomic Groups
get_length_weight_batch(): Get Length-Weight and Morphological Parameters for Species (Batch Version)
get_species_areas_batch(): Get FAO Areas for Species (Batch Version)
load_taxa_databases(): Load Taxa Data from FishBase and SeaLifeBase
map_surveys(): Map Survey Labels to Standardized Taxa, Gear, and Vessel Names
match_species_from_taxa(): Match Species from Taxa Databases
preprocess_general_adnap(): Preprocess General Survey Information for ADNAP
preprocess_landings_adnap(): Preprocess ADNAP Landings Data
preprocess_landings_lurio(): Preprocess Lurio Landings Data
preprocess_pds_tracks(): Preprocess Pelagic Data Systems (PDS) Track Data
preprocess_track_data(): Preprocess Track Data into Spatial Grid Summary
process_species_group(): Process Species Length and Catch Data
process_species_list(): Process Species List with Taxonomic Information
reshape_catch_data(): Reshape Catch Data with Length Groupings
reshape_species_groups(): Reshape Species Groups from Wide to Long Format
standardize_enumerator_names(): Standardize Enumerator Names

Validation

Functions dedicated to the validation module

alert_outlier(): Outlier Alert for Numeric Vectors
export_validation_flags(): Export Validation Flags to MongoDB
get_catch_bounds_taxon(): Get Catch Bounds by Gear + Taxon
get_price_bounds(): Get Price Bounds by Gear + Taxon
get_total_catch_bounds(): Get Total Catch Bounds by Landing Site and Gear
get_validation_status(): Get Validation Status from KoboToolbox
process_submissions_parallel(): Process Submissions in Parallel with Rate Limiting
sync_validation_submissions(): Synchronize Validation Statuses with KoboToolbox
update_validation_status(): Update Validation Status in KoboToolbox
validate_catch_taxa(): Validate Catch at Taxon Level
validate_price(): Validate Catch Price
validate_surveys_adnap(): Validate ADNAP Survey Data
validate_surveys_lurio(): Validate Lurio Survey Data
validate_total_catch(): Validate Total Catch

Export

Functions dedicated dissemination of processed and analysed fisheries data

export_landings(): Export Processed Landings Data

Helper functions

Functions dedicated to data processing.

add_version(): Add timestamp and sha string to a file name
fetch_asset(): Fetch and Filter Asset Data from Airtable
fetch_assets(): Fetch Multiple Asset Tables from Airtable
get_airtable_form_id(): Get Airtable Form ID from KoBoToolbox Asset ID
load_dotenv(): Load environment variables from .env file
map_surveys(): Map Survey Labels to Standardized Taxa, Gear, and Vessel Names
read_config(): Read configuration file
standardize_enumerator_names(): Standardize Enumerator Names

Airtable Integration

Functions for interacting with Airtable API.

airtable_to_df(): Get All Records from Airtable with Pagination
get_writable_fields(): Get Writable Fields from Airtable Table
update_airtable_record(): Update Single Airtable Record
bulk_update_airtable(): Bulk Update Multiple Airtable Records
df_to_airtable(): Create New Airtable Records
device_sync(): Sync Data with Airtable (Update + Create)