Package index • peskas.zanzibar.data.pipeline

Cloud Storage

Functions that interact with cloud storage providers.

cloud_object_name(): Retrieve Full Name of Versioned Cloud Object

cloud_storage_authenticate(): Authenticate to a Cloud Storage Provider

download_cloud_file(): Download Object from Cloud Storage

get_metadata(): Get metadata tables

get_preprocessed_surveys(): Download Preprocessed Surveys

get_validated_surveys(): Download WCS Validated Surveys

mdb_collection_pull(): Retrieve Data from MongoDB

mdb_collection_push(): Upload Data to MongoDB and Overwrite Existing Content

upload_cloud_file(): Upload File to Cloud Storage

Workflow

These are arguably the most important functions in the package. Each of these functions executes a step in the data pipeline.

export_data(): Export Processed Fisheries Data to MongoDB

ingest_pds_tracks(): Ingest Pelagic Data Systems (PDS) Track Data

ingest_pds_trips(): Ingest Pelagic Data Systems (PDS) Trip Data

ingest_surveys(): Ingest WCS and WF Catch Survey Data

preprocess_ba_surveys(): Pre-process Blue Alliance Surveys

preprocess_pds_tracks(): Preprocess Pelagic Data Systems (PDS) Track Data

preprocess_wcs_surveys(): Pre-process Zanzibar WCS Surveys

preprocess_wf_surveys(): Pre-process WorldFish Surveys

validate_ba_surveys(): Validate Blue Alliance (BA) Surveys Data

validate_wcs_surveys(): Validate WCS Surveys Data

Ingestion

Functions dedicated to data pulling and ingestion from servers

get_trip_points(): Get Trip Points from Pelagic Data Systems API

get_trips(): Retrieve Trip Details from Pelagic Data API

ingest_pds_tracks(): Ingest Pelagic Data Systems (PDS) Track Data

ingest_pds_trips(): Ingest Pelagic Data Systems (PDS) Trip Data

ingest_surveys(): Ingest WCS and WF Catch Survey Data

retrieve_surveys(): Retrieve Surveys from Kobotoolbox

Preprocessing

Functions dedicated to the preprocessing module and general data processing

preprocess_ba_surveys(): Pre-process Blue Alliance Surveys

preprocess_pds_tracks(): Preprocess Pelagic Data Systems (PDS) Track Data

preprocess_track_data(): Preprocess Track Data into Spatial Grid Summary

preprocess_wcs_surveys(): Pre-process Zanzibar WCS Surveys

preprocess_wf_surveys(): Pre-process WorldFish Surveys

expand_taxa(): Expand Taxonomic Vectors into a Data Frame

Validation

Functions dedicated to the validation module

add_validation_flags(): Add validation flags to catch data

aggregate_survey_data(): Aggregate survey data and calculate metrics

calculate_catch_revenue(): Calculate catch revenue from validated data

extract_trips_info(): Extract trip information from preprocessed surveys

get_catch_bounds(): Get catch bounds for survey data

get_length_bounds(): Get length bounds for survey data

process_catch_data(): Process catch data from surveys

validate_ba_surveys(): Validate Blue Alliance (BA) Surveys Data

validate_catches(): Validate catches using quality flags

validate_prices(): Validate market prices

validate_wcs_surveys(): Validate WCS Surveys Data

Helper functions

Functions dedicated to data processing.

add_version(): Add timestamp and sha string to a file name

read_config(): Read configuration file

Export

Functions dedicated dissemination of processed and validated data and visualizations

export_data(): Export Processed Fisheries Data to MongoDB

kepler_mapper(): Generate a Kepler.gl map