Package index • peskas.kenya.data.pipeline

Workflow

These are arguably the most important functions in the package. Each of these functions executes a step in the data pipeline.

export_summaries(): Export Summarized Fishery Data for Dashboard Integration

ingest_landings(): Download and Process WCS Catch Surveys from Kobotoolbox

ingest_landings_price(): Download and Process WCS Price Surveys from Kobotoolbox

merge_landings(): Merge Legacy and Ongoing Landings Data

merge_prices(): Merge Price Data

preprocess_landings_v1(): Preprocess Landings Data (Version 1)

preprocess_landings_v2(): Preprocess Landings Data (Version 2)

preprocess_legacy_landings(): Preprocess Legacy Landings Data

preprocess_price_landings(): Preprocess Price Data

validate_landings(): Validate Fisheries Data

Cloud Storage

Functions that interact with cloud storage providers.

cloud_object_name(): Retrieve Full Name of Versioned Cloud Object

cloud_storage_authenticate(): Authenticate to a Cloud Storage Provider

download_cloud_file(): Download Object from Cloud Storage

download_parquet_from_cloud(): Download Parquet File from Cloud Storage

get_metadata(): Get metadata tables

mdb_collection_pull(): Retrieve Data from MongoDB

mdb_collection_push(): Upload Data to MongoDB and Overwrite Existing Content

upload_cloud_file(): Upload File to Cloud Storage

upload_parquet_to_cloud(): Upload Processed Data to Cloud Storage

Ingestion

Functions dedicated to the ingestion module

get_kobo_data(): Retrieve Data from Kobotoolbox API

ingest_landings(): Download and Process WCS Catch Surveys from Kobotoolbox

ingest_landings_price(): Download and Process WCS Price Surveys from Kobotoolbox

Preprocessing

Functions dedicated to the preprocessing module

clean_catch_names(): Clean Catch Names

preprocess_landings_v1(): Preprocess Landings Data (Version 1)

preprocess_landings_v2(): Preprocess Landings Data (Version 2)

preprocess_legacy_landings(): Preprocess Legacy Landings Data

preprocess_price_landings(): Preprocess Price Data

Validation

Functions dedicated to the validation module

alert_outlier(): Generate an alert vector based on the univOutl::LocScaleB() function

alert_outlier_iqr(): Generate an alert vector based on IQR method

check_outliers_iqr(): Check for outliers using IQR method

get_catch_bounds(): Get fish groups Catch Bounds

get_catch_bounds_iqr(): Get fish groups Catch Bounds using IQR method

get_total_catch_bounds(): Get Total Catch Bounds

get_total_catch_bounds_iqr(): Get Total Catch Bounds using IQR method

validate_catch(): Validate Individual Catch Data

validate_catch_iqr(): Validate Individual Catch Data using IQR method

validate_dates(): Validate Landing Dates

validate_fishers_catch(): Validate Catch per Fisher

validate_landings(): Validate Fisheries Data

validate_nboats(): Validate Number of Boats

validate_nboats_iqr(): Validate Number of Boats using IQR method

validate_nfishers(): Validate Number of Fishers

validate_nfishers_iqr(): Validate Number of Fishers using IQR method

validate_total_catch(): Validate Total Catch Data

validate_total_catch_iqr(): Validate Total Catch Data using IQR method

Export

Functions dedicated dissemination of processed and analysed fisheries data

create_geos(): Generate Geographic Regional Summaries of Fishery Data

export_summaries(): Export Summarized Fishery Data for Dashboard Integration

Helper functions

Functions dedicated to data analytics and general statistics

add_version(): Add timestamp and sha string to a file name

get_fishery_metrics(): Calculate Fishery Performance Metrics

impute_price(): Impute Missing Fish Prices Using Median Values

read_config(): Read configuration file

summarise_catch_price(): Summarize Catch Price Data