Package index • peskas.kenya.data.pipeline

Workflow

These are arguably the most important functions in the package. Each of these functions executes a step in the data pipeline.

export_summaries(): Export Summarized Fishery Data for Dashboard Integration
get_validation_status(): Get Validation Status from KoboToolbox
ingest_kefs_surveys_v1(): Download and Process KEFS (BMU DAILY ARTISANAL 2025) Catch Surveys from Kobotoolbox
ingest_kefs_surveys_v2(): Download and Process KEFS (CATCH ASSESSMENT QUESTIONNAIRE) Catch Surveys from Kobotoolbox
ingest_landings_price(): Download and Process WCS Price Surveys from Kobotoolbox
ingest_pds_tracks(): Ingest Pelagic Data Systems (PDS) Track Data
ingest_pds_trips(): Ingest Pelagic Data Systems (PDS) Trip Data
ingest_wcs_surveys(): Download and Process WCS Catch Surveys from Kobotoolbox
merge_landings(): Merge Legacy and Ongoing Landings Data
merge_prices(): Merge Price Data
preprocess_kefs_surveys_v1(): Preprocess KEFS Survey Data
preprocess_kefs_surveys_v2(): Preprocess KEFS (CATCH ASSESSMENT QUESTIONNAIRE) Survey Data
preprocess_landings_v1(): Preprocess Landings Data (Version 1)
preprocess_landings_v2(): Preprocess Landings Data (Version 2)
preprocess_legacy_landings(): Preprocess Legacy Landings Data
preprocess_pds_tracks(): Preprocess Pelagic Data Systems (PDS) Track Data
preprocess_price_landings(): Preprocess Price Data
sync_validation_submissions(): Synchronize Validation Statuses with KoboToolbox
update_validation_status(): Update Validation Status in KoboToolbox
validate_kefs_surveys_v2(): Validate KEFS Surveys Data (Version 2)
validate_landings(): Validate Fisheries Data

Cloud Storage

Functions that interact with cloud storage providers.

cloud_object_name(): Retrieve Full Name of Versioned Cloud Object
cloud_storage_authenticate(): Authenticate to a Cloud Storage Provider
download_cloud_file(): Download Object from Cloud Storage
download_parquet_from_cloud(): Download Parquet File from Cloud Storage
get_metadata(): Get metadata tables
mdb_collection_pull(): Retrieve Data from MongoDB
mdb_collection_push(): Upload Data to MongoDB and Overwrite Existing Content
upload_cloud_file(): Upload File to Cloud Storage
upload_parquet_to_cloud(): Upload Processed Data to Cloud Storage

Ingestion

Functions dedicated to the ingestion module

airtable_to_df(): Get All Records from Airtable with Pagination
get_kobo_data(): Retrieve Data from Kobotoolbox API
get_trip_points(): Get Trip Points from Pelagic Data Systems API
get_trips(): Retrieve Trip Details from Pelagic Data API
ingest_kefs_surveys_v1(): Download and Process KEFS (BMU DAILY ARTISANAL 2025) Catch Surveys from Kobotoolbox
ingest_kefs_surveys_v2(): Download and Process KEFS (CATCH ASSESSMENT QUESTIONNAIRE) Catch Surveys from Kobotoolbox
ingest_landings_price(): Download and Process WCS Price Surveys from Kobotoolbox
ingest_pds_tracks(): Ingest Pelagic Data Systems (PDS) Track Data
ingest_pds_trips(): Ingest Pelagic Data Systems (PDS) Trip Data
ingest_wcs_surveys(): Download and Process WCS Catch Surveys from Kobotoolbox

Preprocessing

Functions dedicated to the preprocessing module

clean_catch_names(): Clean Catch Names
fetch_asset(): Fetch and Filter Asset Data from Airtable
fetch_assets(): Fetch Multiple Asset Tables from Airtable
generate_track_summaries(): Generate Grid Summaries for Track Data
get_airtable_form_id(): Get Airtable Form ID from KoBoToolbox Asset ID
map_surveys(): Map Survey Labels to Standardized Taxa, Gear, and Vessel Names
preprocess_kefs_surveys_v1(): Preprocess KEFS Survey Data
preprocess_kefs_surveys_v2(): Preprocess KEFS (CATCH ASSESSMENT QUESTIONNAIRE) Survey Data
preprocess_landings_v1(): Preprocess Landings Data (Version 1)
preprocess_landings_v2(): Preprocess Landings Data (Version 2)
preprocess_legacy_landings(): Preprocess Legacy Landings Data
preprocess_pds_tracks(): Preprocess Pelagic Data Systems (PDS) Track Data
preprocess_price_landings(): Preprocess Price Data
preprocess_track_data(): Preprocess Track Data into Spatial Grid Summary
reshape_catch_data_v1(): Reshape catch details from wide to long format
reshape_overall_sample(): Reshape Overall Sample Weight Data from Wide to Long Format
reshape_priority_species(): Reshape Priority Species Catch Data from Wide to Long Format
standardize_enumerator_names(): Standardize Enumerator Names

Validation

Functions dedicated to the validation module

alert_outlier(): Generate an alert vector based on the univOutl::LocScaleB() function
alert_outlier_iqr(): Generate an alert vector based on IQR method
check_outliers_iqr(): Check for outliers using IQR method
get_catch_bounds(): Get fish groups Catch Bounds
get_catch_bounds_iqr(): Get fish groups Catch Bounds using IQR method
get_catch_flags(): Generate Catch-Level Validation Flags
get_indicators_flags(): Generate Composite Indicator Validation Flags
get_total_catch_bounds(): Get Total Catch Bounds
get_total_catch_bounds_iqr(): Get Total Catch Bounds using IQR method
get_trips_flags(): Generate Trip-Level Validation Flags
get_validation_status(): Get Validation Status from KoboToolbox
sync_validation_submissions(): Synchronize Validation Statuses with KoboToolbox
update_validation_status(): Update Validation Status in KoboToolbox
validate_catch(): Validate Individual Catch Data
validate_catch_iqr(): Validate Individual Catch Data using IQR method
validate_dates(): Validate Landing Dates
validate_fishers_catch(): Validate Catch per Fisher
validate_kefs_surveys_v2(): Validate KEFS Surveys Data (Version 2)
validate_landings(): Validate Fisheries Data
validate_nboats(): Validate Number of Boats
validate_nboats_iqr(): Validate Number of Boats using IQR method
validate_nfishers(): Validate Number of Fishers
validate_nfishers_iqr(): Validate Number of Fishers using IQR method
validate_total_catch(): Validate Total Catch Data
validate_total_catch_iqr(): Validate Total Catch Data using IQR method

Export

Functions dedicated dissemination of processed and analysed fisheries data

bulk_update_airtable(): Bulk Update Multiple Airtable Records
create_geos(): Generate Geographic Regional Summaries of Fishery Data
df_to_airtable(): Create New Airtable Records
export_summaries(): Export Summarized Fishery Data for Dashboard Integration
update_airtable_record(): Update Single Airtable Record

Helper functions

Functions dedicated to data analytics and general statistics

add_version(): Add timestamp and sha string to a file name
fetch_asset(): Fetch and Filter Asset Data from Airtable
fetch_assets(): Fetch Multiple Asset Tables from Airtable
get_airtable_form_id(): Get Airtable Form ID from KoBoToolbox Asset ID
get_fishery_metrics(): Calculate Fishery Performance Metrics
get_fishery_metrics_long(): Calculate key fishery metrics by landing site and month in normalized long format
get_individual_gear_metrics(): Calculate Individual Fisher Performance Metrics by Gear Type
get_individual_metrics(): Calculate Individual Fisher Performance Metrics
get_writable_fields(): Get Writable Fields from Airtable Table
impute_price(): Impute Missing Fish Prices Using Median Values
map_surveys(): Map Survey Labels to Standardized Taxa, Gear, and Vessel Names
read_config(): Read configuration file
reshape_overall_sample(): Reshape Overall Sample Weight Data from Wide to Long Format
reshape_priority_species(): Reshape Priority Species Catch Data from Wide to Long Format
standardize_enumerator_names(): Standardize Enumerator Names
summarise_catch_price(): Summarize Catch Price Data