Package index • peskas.mozambique.data.pipeline

Workflow

These are arguably the most important functions in the package. Each of these functions executes a step in the data pipeline.

export_landings(): Export Processed Landings Data

ingest_landings(): Download and Process Surveys from Kobotoolbox

ingest_pds_tracks(): Ingest Pelagic Data Systems (PDS) Track Data

ingest_pds_trips(): Ingest Pelagic Data Systems (PDS) Trip Data

preprocess_landings(): Preprocess Landings Data

preprocess_pds_tracks(): Preprocess Pelagic Data Systems (PDS) Track Data

validate_landings(): Validate Wild Fisheries Survey Data

Cloud Storage

Functions that interact with cloud storage providers.

cloud_object_name(): Generate Cloud Object Name

cloud_storage_authenticate(): Authenticate to a Cloud Storage Provider

download_cloud_file(): Download File from Cloud Storage

download_parquet_from_cloud(): Download Parquet File from Cloud Storage

get_metadata(): Get metadata tables

mdb_collection_pull(): Retrieve Data from MongoDB

mdb_collection_push(): Upload Data to MongoDB and Overwrite Existing Content

upload_cloud_file(): Upload File to Cloud Storage

upload_parquet_to_cloud(): Upload Data as Parquet File to Cloud Storage

Ingestion

Functions dedicated to the ingestion module

get_kobo_data(): Retrieve Data from Kobotoolbox API

get_trip_points(): Get Trip Points from Pelagic Data Systems API

get_trips(): Retrieve Trip Details from Pelagic Data API

ingest_landings(): Download and Process Surveys from Kobotoolbox

ingest_pds_tracks(): Ingest Pelagic Data Systems (PDS) Track Data

ingest_pds_trips(): Ingest Pelagic Data Systems (PDS) Trip Data

Preprocessing

Functions dedicated to the preprocessing module

calculate_catch(): Calculate Catch Weight from Length-Weight Relationships or Bucket Measurements

calculate_fishery_metrics(): Calculate Fishery Metrics

generate_track_summaries(): Generate Grid Summaries for Track Data

getLWCoeffs(): Get Length-Weight Coefficients and Morphological Data for Species

get_fao_groups(): Extract and Format FAO Taxonomic Groups

get_length_weight_batch(): Get Length-Weight and Morphological Parameters for Species (Batch Version)

get_species_areas_batch(): Get FAO Areas for Species (Batch Version)

load_taxa_databases(): Load Taxa Data from FishBase and SeaLifeBase

match_species_from_taxa(): Match Species from Taxa Databases

preprocess_landings(): Preprocess Landings Data

preprocess_pds_tracks(): Preprocess Pelagic Data Systems (PDS) Track Data

preprocess_track_data(): Preprocess Track Data into Spatial Grid Summary

process_species_group(): Process Species Length and Catch Data

process_species_list(): Process Species List with Taxonomic Information

Validation

Functions dedicated to the validation module

alert_outlier(): Outlier Alert for Numeric Vectors

get_catch_bounds_taxon(): Get Catch Bounds by Gear + Taxon

get_price_bounds(): Get Price Bounds by Gear + Taxon

get_total_catch_bounds(): Get Total Catch Bounds by Landing Site and Gear

validate_catch_taxa(): Validate Catch at Taxon Level

validate_landings(): Validate Wild Fisheries Survey Data

validate_price(): Validate Catch Price

validate_total_catch(): Validate Total Catch

Export

Functions dedicated dissemination of processed and analysed fisheries data

export_landings(): Export Processed Landings Data

Helper functions

Functions dedicated to data processing.

add_version(): Add timestamp and sha string to a file name

load_dotenv(): Load environment variables from .env file

read_config(): Read configuration file

Airtable Integration

Functions for interacting with Airtable API.

airtable_to_df(): Get All Records from Airtable with Pagination

get_writable_fields(): Get Writable Fields from Airtable Table

update_airtable_record(): Update Single Airtable Record

bulk_update_airtable(): Bulk Update Multiple Airtable Records

df_to_airtable(): Create New Airtable Records

device_sync(): Sync Data with Airtable (Update + Create)

sync_device_users(): Sync Device Users to MongoDB and Update Airtable