Package index • peskas.timor.data.pipeline

Workflow

These are arguably the most important functions in the package. Each of these functions executes a step in the data pipeline, and each is called from .github/workflows/data-pipeline.yaml by name.

calculate_weights(): Add weight of species to merged landings
describe_pds_tracks(): Extract per-trip descriptors from PDS tracks
estimate_fishery_indicators(): Estimate fisheries indicators
export_api_raw(): Export the raw trips table to the cross-country API bucket
export_api_validated(): Export the validated trips table to the cross-country API bucket
export_files(): Export portal datasets as versioned JSON files to public cloud storage
format_public_data(): Format public data
ingest_landings(): Ingest landings surveys from KoboToolbox
merge_landings(): Merge the pre-processed landings of every form version
merge_trips(): Merge trips
preprocess_landings(): Pre-process the landing surveys into a flat long catch table
sync_validation_status(): Write validation statuses back to KoBoToolbox
upload_dataverse(): Upload and publish a dataset on Dataverse
validate_landings(): Validate landings
validate_pds_trips(): Validate Pelagic Data System trips

Cloud storage

Accessors over the versioned artefacts in Google Cloud Storage. The storage layer itself is coasts::*; these resolve the latest version of each artefact and give it back in the shape its readers expect.

get_assets(): Download the Airtable frame mapping tables for Timor
get_merged_landings(): Download the merged landings
get_merged_trips(): Download Peskas surveys and PDS data
get_models(): Download models estimates
get_tracks_descriptors(): Extract summaries from one pds track
get_validated_landings(): Download Peskas validated landings, in the nested portal shape
get_validated_landings_long(): Download the validated landings in the flat long shape
get_weighted_landings(): Download the merged landings with catch weights

Ingestion

Functions dedicated to the ingestion module.

ingest_landings(): Ingest landings surveys from KoboToolbox

Preprocessing

Functions dedicated to the preprocessing module.

expand_length_frequency(): Expand each catch into one row per length bin
join_weights(): Join length-weights and nutritional parameters info to preprocessed landings
merge_landings(): Merge the pre-processed landings of every form version
preprocess_landings(): Pre-process the landing surveys into a flat long catch table
reshape_landings(): Reshape one raw landings table into the flat long catch table
reshape_species_groups(): Reshape KoBo species groups into one row per catch
survey_labels(): Label lookup tables for preprocessing

Validation

Functions dedicated to the validation module. Deeper than any other country pipeline — the alert codes are a contract, documented in the header of R/validation-functions.R and in the validation.alerts block of inst/config.yml.

push_validation_flags(): Push the validation flags to the shared validation database
sync_validation_status(): Write validation statuses back to KoBoToolbox
validate_catch_params(): Validate surveys' catch parameters
validate_catch_price(): Validate surveys' total catch values
validate_conservation(): Resolve the recorded catch-preservation method
validate_fuel(): Validate the recorded fuel use
validate_gear_type(): Validate the recorded gear type
validate_gleaners(): Validate the recorded number of gleaners
validate_habitat(): Validate the recorded habitat
validate_happiness(): Carry through the recorded happiness rating
validate_imeis(): Validate the tracker IMEIs recorded on a survey
validate_landing_regularity(): Validate the internal consistency of a landing
validate_landings(): Validate landings
validate_mesh(): Validate the recorded mesh size
validate_n_fishers(): Validate the recorded crew size
validate_pds_data(): Validate pds trips duration and distance
validate_pds_trips(): Validate Pelagic Data System trips
validate_price_weight(): Outlier identification of the price-weight relation
validate_sites(): Validate the recorded landing site
validate_surveys_time(): Validate surveys' temporal parameters
validate_vessel_type(): Validate the recorded vessel type
validation_submissions(): One row per submission

Export

Publication of processed and analysed fisheries data — the portal JSON contract, the cross-country API parquet, Dataverse and the emailed reports.

check_portal_contract(): Check the portal JSON contract
export_api_raw(): Export the raw trips table to the cross-country API bucket
export_api_validated(): Export the validated trips table to the cross-country API bucket
generate_description(): Generate data description
publish_last_dataset(): Publish latest dataset created
send_validation_mail(): Send validation summary email
upload_dataverse(): Upload and publish a dataset on Dataverse
upload_files(): Upload files to Dataverse

Helper functions

Reshaping, taxonomy, geometry and configuration helpers.

add_version(): Add timestamp and sha string to a file name
convert_taxa_names(): Assign each taxon code to a broad species group
fill_missing_regions(): Fill missing regions
get_airtable_form_id(): Resolve a KoBo asset id to its Airtable form record id
get_distance(): Estimate distance between consecutive trips
get_fao_composition(): Get FAO Food Composition Data
get_morphometric_tables(): Build per-taxon length-weight coefficients
get_nutrients_table(): Get taxa nutritional values
get_timor_boundaries(): Extract Timor Geographical Boundaries
load_dotenv(): Load environment variables from a .env file
merge_consecutive_trips(): Merge short intervalled trips
read_config(): Read configuration file