Package index • peskas.kenya.data.pipeline

Workflow

These are arguably the most important functions in the package. Each of these functions executes a step in the data pipeline.

export_summaries(): Export Summarized Fishery Data for Dashboard Integration

ingest_kefs_surveys(): Download and Process KEFS Catch Surveys from Kobotoolbox

ingest_landings(): Download and Process All Catch Surveys (WCS + KEFS)

ingest_landings_price(): Download and Process WCS Price Surveys from Kobotoolbox

ingest_pds_tracks(): Ingest Pelagic Data Systems (PDS) Track Data

ingest_pds_trips(): Ingest Pelagic Data Systems (PDS) Trip Data

ingest_wcs_surveys(): Download and Process WCS Catch Surveys from Kobotoolbox

merge_landings(): Merge Legacy and Ongoing Landings Data

merge_prices(): Merge Price Data

preprocess_landings_v1(): Preprocess Landings Data (Version 1)

preprocess_landings_v2(): Preprocess Landings Data (Version 2)

preprocess_legacy_landings(): Preprocess Legacy Landings Data

preprocess_pds_tracks(): Preprocess Pelagic Data Systems (PDS) Track Data

preprocess_price_landings(): Preprocess Price Data

validate_landings(): Validate Fisheries Data

Cloud Storage

Functions that interact with cloud storage providers.

cloud_object_name(): Retrieve Full Name of Versioned Cloud Object

cloud_storage_authenticate(): Authenticate to a Cloud Storage Provider

download_cloud_file(): Download Object from Cloud Storage

download_parquet_from_cloud(): Download Parquet File from Cloud Storage

get_metadata(): Get metadata tables

mdb_collection_pull(): Retrieve Data from MongoDB

mdb_collection_push(): Upload Data to MongoDB and Overwrite Existing Content

upload_cloud_file(): Upload File to Cloud Storage

upload_parquet_to_cloud(): Upload Processed Data to Cloud Storage

Ingestion

Functions dedicated to the ingestion module

airtable_to_df(): Get All Records from Airtable with Pagination

get_kobo_data(): Retrieve Data from Kobotoolbox API

get_trip_points(): Get Trip Points from Pelagic Data Systems API

get_trips(): Retrieve Trip Details from Pelagic Data API

ingest_kefs_surveys(): Download and Process KEFS Catch Surveys from Kobotoolbox

ingest_landings(): Download and Process All Catch Surveys (WCS + KEFS)

ingest_landings_price(): Download and Process WCS Price Surveys from Kobotoolbox

ingest_pds_tracks(): Ingest Pelagic Data Systems (PDS) Track Data

ingest_pds_trips(): Ingest Pelagic Data Systems (PDS) Trip Data

ingest_wcs_surveys(): Download and Process WCS Catch Surveys from Kobotoolbox

Preprocessing

Functions dedicated to the preprocessing module

clean_catch_names(): Clean Catch Names

generate_track_summaries(): Generate Grid Summaries for Track Data

preprocess_landings_v1(): Preprocess Landings Data (Version 1)

preprocess_landings_v2(): Preprocess Landings Data (Version 2)

preprocess_legacy_landings(): Preprocess Legacy Landings Data

preprocess_pds_tracks(): Preprocess Pelagic Data Systems (PDS) Track Data

preprocess_price_landings(): Preprocess Price Data

preprocess_track_data(): Preprocess Track Data into Spatial Grid Summary

Validation

Functions dedicated to the validation module

alert_outlier(): Generate an alert vector based on the univOutl::LocScaleB() function

alert_outlier_iqr(): Generate an alert vector based on IQR method

check_outliers_iqr(): Check for outliers using IQR method

get_catch_bounds(): Get fish groups Catch Bounds

get_catch_bounds_iqr(): Get fish groups Catch Bounds using IQR method

get_total_catch_bounds(): Get Total Catch Bounds

get_total_catch_bounds_iqr(): Get Total Catch Bounds using IQR method

validate_catch(): Validate Individual Catch Data

validate_catch_iqr(): Validate Individual Catch Data using IQR method

validate_dates(): Validate Landing Dates

validate_fishers_catch(): Validate Catch per Fisher

validate_landings(): Validate Fisheries Data

validate_nboats(): Validate Number of Boats

validate_nboats_iqr(): Validate Number of Boats using IQR method

validate_nfishers(): Validate Number of Fishers

validate_nfishers_iqr(): Validate Number of Fishers using IQR method

validate_total_catch(): Validate Total Catch Data

validate_total_catch_iqr(): Validate Total Catch Data using IQR method

Export

Functions dedicated dissemination of processed and analysed fisheries data

bulk_update_airtable(): Bulk Update Multiple Airtable Records

create_geos(): Generate Geographic Regional Summaries of Fishery Data

df_to_airtable(): Create New Airtable Records

export_summaries(): Export Summarized Fishery Data for Dashboard Integration

update_airtable_record(): Update Single Airtable Record

Helper functions

Functions dedicated to data analytics and general statistics

add_version(): Add timestamp and sha string to a file name

get_fishery_metrics(): Calculate Fishery Performance Metrics

get_fishery_metrics_long(): Calculate key fishery metrics by landing site and month in normalized long format

get_individual_gear_metrics(): Calculate Individual Fisher Performance Metrics by Gear Type

get_individual_metrics(): Calculate Individual Fisher Performance Metrics

get_writable_fields(): Get Writable Fields from Airtable Table

impute_price(): Impute Missing Fish Prices Using Median Values

read_config(): Read configuration file

summarise_catch_price(): Summarize Catch Price Data