Skip to contents

Changelog

peskas.kenya.data.pipeline 5.8.2

KEFS surveys that weighed the whole catch by species

  • FIXED A KEFS survey whose species weights add up to the total catch (within 5%) takes the total catch as its sample weight. Some enumerators weigh every species in the whole catch but still type the small weighed sample in the sample weight field, so each species looked heavier than the sample and alert 5.2 rejected the trip. Since July 2026 this removed almost every survey from three Lunga Lunga enumerators (and from one in Kilifi North between November 2025 and June 2026), which emptied those districts from the dashboard’s estimates. Alert 5.2 itself is unchanged.
  • On the prod data of 9 November 2026: 1,606 surveys get the total catch as sample weight, alert 5.2 goes from 2,118 surveys to 703, and no survey gains an alert. The sample weight is used only by alerts 5.1 and 5.2.

peskas.kenya.data.pipeline 5.8.1

Trip duration on the new KEFS form

  • FIXED KEFS trips have a trip duration again on the form versions deployed since 22 April 2026. Those versions record when the vessel left and landed in two new fields (time_out, time_in), and only the old ones were read, so the trips had no duration, no CPUE or RPUE, and alerts 3, 6.1 and 6.2 could not fire on them.

peskas.kenya.data.pipeline 5.8.0

One name per KEFS enumerator

  • CHANGED KEFS enumerator names typed in the form are matched to the enumerators listed in the Airtable enumerators table for the KEFS form (the form’s single-select list since 22 April 2026). Words can come in any order and with small typos (“mahmoud madi yusuf” is Mahmud Madi), a name written without spaces counts (“abdulhakim mohamed” is Abdul Hakim), and the table’s new Aliases field holds the spellings the data cannot settle alone, a different surname or two people with one name (“athman lali (kiunga)” limits a spelling to one BMU). Names not on the list are grouped among themselves.
  • CHANGED A first name alone (“naomi”) goes to the one person with that name at the same BMU, or stays a name of its own; it used to become “undefined”. Two names that recorded landings on the same days at different BMUs are never merged.
  • On the prod data of 30 October 2026 the v2 form goes from 144 enumerators and 4,305 “undefined” submissions to 96 and 84. The other names are real: about 90 people recorded at least 10 submissions since 2024, many before the list existed. The new names reach the Management Platform at the next validation run; reviewers’ decisions are kept.

peskas.kenya.data.pipeline 5.7.1

The low-price alert matches the KoBo form

  • FIXED Alert 6.4 flags a trip sold for less than 50 KSH per kg (about 0.40 USD) instead of 20, the floor the KEFS form has had since 22 April 2026.

peskas.kenya.data.pipeline 5.7.0

A price too low to be real is flagged

  • NEW Alert 6.4 flags a trip sold for less than 20 KSH per kg (about 0.15 USD), the low end of alert 6.3. Only a price too high was checked, so a mistyped trip price per kg went out as validated, often the weight typed again (11 kg sold at 1 KSH per kg) while the species prices in the same form were right. On the data of 27 October 2026 it flags 36 KEFS trips for review.

peskas.kenya.data.pipeline 5.6.2

Catch outcome goes out as a number

  • FIXED Whether a trip caught anything is published as a number, as the API describes it, instead of text.

peskas.kenya.data.pipeline 5.6.1

Landing-site codes match whatever their case

  • FIXED A survey’s landing site is matched to the Airtable landing sites whatever its case, so “Kipini” finds the table’s “kipini”. 148 KEFS trips at Kipini and Ras Ya Mwana had no district for that reason and now count in Garsen.

peskas.kenya.data.pipeline 5.6.0

Alerts point to the species sample that raised them

  • NEW Alerts 5.2 (a sample heavier than its total) and 5.3 (a length over 500 cm) record which species sample they are about (numbered as in the form), its species and the weight or length recorded. The Management Platform shows them, so a reviewer goes straight to the sample instead of checking each one: 2,018 KEFS submissions now say which. No alert or decision changes.

peskas.kenya.data.pipeline 5.5.2

No-catch trips with a price of zero are not flagged

  • FIXED Alert 4.2 flags a trip with no catch only when its total price is above zero. It said “zero or more”, so a no-catch trip with a price of 0 would have been flagged. No trip in the data changes: so far those trips record no price at all.

peskas.kenya.data.pipeline 5.5.1

Each alert shows once

  • FIXED A submission lists each alert code once. An alert used to repeat once per species recorded (“5.2, 5.2”), and each repeat counted as one more flagged submission in the enumerator statistics.

peskas.kenya.data.pipeline 5.5.0

A second estimate of total catch and revenue, by FAO’s ARTFISH method

  • NEW Monthly total catch and revenue are also estimated with the FAO ARTFISH method for each district, split by gear or boat type, beside the GPS tracker method’s estimate so the two can be compared.
  • FIXED Estimated catch and revenue are no longer shown for a district and month with fewer than ten surveyed trips, where a single trip stood for the whole fleet.

peskas.kenya.data.pipeline 5.4.0

Boats are placed by where they land

  • NEW Fleet activity estimates place each tracked boat in the district where its trips land, read from its GPS track, so new and moved trackers count without being linked to a district by hand.
  • NEW Fishing trips longer than two days now count in the fleet activity estimates.

peskas.kenya.data.pipeline 5.3.2

Reviewers’ decisions are kept between runs

  • FIXED KEFS surveys a reviewer approves in the Peskas Management Platform stay in the data, those a reviewer rejects leave it, and reviewers’ decisions are no longer undone by the next run.

peskas.kenya.data.pipeline 5.3.1

KEFS fork lengths are restated as total length

  • FIXED

KEFS measures most fish on total length, but tunas, mackerels, jacks and some snappers on fork length: 360 of 12,746 measured fish. The mean length per species averaged both as if they were one, and the API drops the length type, so the size views compared fork lengths with lengths at maturity given as total length. Each fork-length fish is now restated as total length before the mean, with the FishBase fits coasts uses for the lengths at maturity (coasts::get_tl_conversions()).

Alert 5.3 checks every fish, not the mean

  • FIXED

Alert 5.3 tested the mean length of the fish measured, so one impossible fish among normal ones could stay under 500 cm. It now tests the largest fish. This flags one more landing: two frigate tuna recorded at 106.5 and 669.5 cm (the species reaches about 65 cm), published until now with a mean of 388 cm.

peskas.kenya.data.pipeline 5.3.0

Records say which organization collected them

  • NEW

survey_organization names the organization behind each record — "WCS" or "KEFS" — as the first column. survey_id identifies the form, not the organization, and Kenya publishes two programmes side by side that differ in what they collect and in how catch is sampled.

peskas.kenya.data.pipeline 5.2.0

Length is validated at all

  • NEW

Alert 5.3 flags a catch length above 500 cm. Kenya had no length check of any kind, so a 1,316 cm Siganus sutor — a rabbitfish reaching about 45 cm — was published and stayed there. Unlike Zanzibar and Mozambique there is no morphometrics table here, so this is an absolute ceiling rather than a species check.

peskas.kenya.data.pipeline 5.1.2

landing_date is published as a date, not a timestamp

The cross-country landings schema declares landing_date as a date, and Mozambique, Timor-Leste and Zanzibar all publish one. Kenya published a POSIXct timestamp, and was the last country still doing so — anyone concatenating the four parquet files hit either a type error or a silent coercion, depending on the reader.

The cause was a bind, not a cast. The WCS tables carry landing_date as POSIXct and the KEFS tables carry it as Date; dplyr::bind_rows() promotes the pair to POSIXct, so the WCS type won for the whole column, KEFS rows included.

  • FIXED format_api_wcs() casts landing_date with lubridate::as_date() before the select, so both branches enter the bind as Date. Data impact: the published type only. Verified against trips-validated__20260921154256_ff4fdcf__.parquet: 286,916 rows and the 22-column order are unchanged, no value moves, and the set of dates is identical() to the published one once both are read as dates. Every one of the 258,556 WCS source rows is already at midnight, so the cast discards no time component.
  • The cast is at the API boundary rather than in preprocessing on purpose: the WCS preprocessed parquet is read by merge_landings(), validate_landings() and the price join, which derives year from this column. Retyping it there would reach all of them for no gain, since the schema this fixes governs the export alone.

peskas.kenya.data.pipeline 5.1.1

Structural fix to KEFS v2 preprocessing. The form’s two repeat groups were joined as if they were the same level of the data, producing a cartesian product of every measured fish with every sampled species.

Data impact

Re-running preprocess_kefs_surveys_v2() over the current raw snapshot (20,227 submissions):

before after
preprocessed rows 60,707 36,147
API sum(catch_kg) 535,898 kg 429,391 kg (-19.9%)
API sum(catch_price) 266.4M KSH 159.7M KSH (-40.1%)
duplicate (trip_id, n_catch) keys 20,580 0
rows carrying length_cm 26,525 3,744
tot_catch_kg == sum(catch_kg) failure 53.6% 46.0%

The submission count is unchanged: the removed rows were copies, not records. 2,517 trips had their weight inflated, the worst by a factor of 25, and 56.9% of the rows carrying a length had it on the wrong species. Part of the 54.5% invariant failure reported in 5.1.0 as a property of the sampled-composition design was in fact this defect; the remaining 46% is the design.

Fixes

  • OverallSampleWeight and PrioritySpeciesCatch were joined on submission_id alone (R/preprocessing-surveys.R). They are nested levels: the first is the species composition of the weighed sample, one row per catch item; the second is a length subsample of individual fish drawn from it, one row per fish. Enumerators do measure several species per trip, so the fish cannot be assumed to share a taxon. The new summarise_priority_lengths() collapses them onto the species they belong to, attached by submission_id and species.

  • total_sample_weight read only SampleWeight (R/preprocessing-surveys.R). The Kobo form renamed the field to lowercase around 2025-12-10; the column is now the coalesce() of both, recovering 641 submissions. Validation rules 5.1 and 5.2 had been inert for those and now fire 184 additional genuine alerts, losing none.

Behaviour changes

  • The preprocessed KEFS v2 schema changed. The priority_* columns are gone, since a catch row now carries exactly one taxon. In their place: length_min_cm, length_max_cm, n_measured, measured_weight_kg. length_cm is now the mean over the individuals measured for that row’s species, matching how Timor publishes the field. The 22-column API schema is unchanged.

  • Lengths for a species absent from the composition block are dropped, with a warning naming the count (292 species / 572 fish). Keeping them would publish a catch item that was never weighed.

  • A species appearing in more than one composition row stays as separate rows (276 pairs). These are genuine separate entries with their own weight and price, not duplicates; they share the species’ length summary, which is a mean and so double-counts nothing.

peskas.kenya.data.pipeline 5.1.0

Numeric-integrity release for the peskas-api-prod landings export. Kenya was the only country whose published trips failed the cross-country invariant tot_catch_kg == sum(catch_kg) within trip_id. These changes take the overall failure rate from 24.5% to 6.6% of 141,336 validated trips, with the remainder confined to KEFS (see Known remaining gap below).

Data impact

Numbers below are measured against the production snapshot trips-validated__20260911025624_92d37f9__.parquet.

  • WCS catch rows were being duplicated by the price join. The published validated export drops from 342,607 to 301,737 rows — 40,870 duplicated catch rows removed — with the trip count unchanged at 141,336. No trip and no genuine catch record is lost; only rows that the price join had duplicated.

  • Published WCS weight was over-stated. Total catch_kg across the validated export falls from 4,164,999 kg to 3,639,974 kg (-12.6%). The removed kilograms were double-counted copies of real catch rows, not real catch.

  • WCS trip revenue was over-stated by the same factor. tot_catch_price is summed over the catch rows of a trip, so it inherited the duplication. Summed trip revenue falls from 952,653,752 to 830,447,469 KSH (-12.8%).

  • Reconciliation by survey, tot_catch_kg == sum(catch_kg) failure rate:

    survey_id trips before after
    aSgfgkYbHn5Q4CVD7Lgcu2 (WCS current) 54,304 41.8% 0.4%
    aAy6nzUo7d7xPsH4YaFs4M (WCS previous) 42,252 7.3% 0.9%
    legacy (WCS legacy) 29,277 1.6% 1.6%
    aNKjDfXDKyW3JLtaCdxDp5 (KEFS) 15,503 54.5% 54.5%

  • trip_duration_hrs is no longer fabricated for WCS. 87.7% of published rows previously carried a hardcoded 24 that was indistinguishable from a measured value; those rows now carry NA_real_. Any WCS catch-per-hour figure computed from the old value was meaningless. KEFS durations, which are measured, are unaffected.

  • KEFS catch_price was a rate published as a value. It was mapped straight from sample_price, which is a KSH/kg rate, into a column the schema defines in the same units as tot_catch_price. Multiplying by the row’s weight takes summed KEFS catch_price from 14,278,682 to 87,172,859 KSH against a trip-level tot_catch_price total of 152,027,377 KSH, and raises the share of KEFS trips where tot_catch_price agrees with sum(catch_price) within 1% from 0.7% to 37.1% (66.5% across the 53.9% of trips that sample the whole catch). Row and trip counts are unchanged at 44,468 / 15,503, and KEFS weight reconciliation is deliberately unchanged — see Known remaining gap. Across the whole validated export, summed catch_price moves from 839,184,457 to 787,061,815 KSH, the net of this increase and the WCS de-duplication above.

  • The raw export now covers the same population as the validated export. kenya/raw grows from 20,312 trips (1 survey, from 2024-01-06) to 169,394 trips (4 surveys, from 1995-09-06), and the number of validated trips absent from the raw export goes from 125,833 to 0. Kenya’s status parameter now selects two processing stages of one population, as in the other three country repos, instead of two unrelated populations.

Fixes

  • Duplicate keys in wcs-price_table fanned out the WCS catch rows (merge_prices(), R/merge-landings.R). The table carried 6,130 rows for only 5,776 distinct (date, landing_site, fish_category, size) keys. It is now collapsed to exactly one median per key before upload: 6,130 -> 5,776 rows, 354 duplicate keys removed.

    Two independent sources produced the duplicates, and the collapse is placed after the landing_site recoding so it catches both. 318 keys came from the v1 and v2 price forms overlapping through 2025 — both forms collect at the same sites, each is summarised to a yearly median separately, and bind_rows() kept both. The remaining 36 are created by the recoding itself (rigati -> rigata, kiwayuu_cha_nje -> kiwayuu_cha_inde), which folds a variant spelling onto a key that already exists. The duplicate pairs are not a small/large split that lost its size label: both sizes appear duplicated independently, with the size label intact.

  • The price join in validate_landings() is now declared many-to-one (R/validation.R). dplyr had been warning about an unexpected many-to-many relationship on every production run. Because catch_price is computed per joined row and total_catch_price = sum(catch_price) is computed after the join, the trailing distinct() could not collapse the duplicates. A duplicated price table now aborts the run instead of silently inflating both weight and revenue.

  • trip_duration_hrs was hardcoded to 24 for WCS (format_api_wcs(), R/api.R), now NA_real_. No WCS form — legacy, v1 or v2 — collects trip duration. The only time fields on the Kobo forms are the start/end stamps of the enumerator’s form session, whose median span is 0 minutes; they measure form-filling, not time at sea. The reasoning is recorded in the function’s roxygen so the NA is not later “fixed” back to a constant.

  • export_api_raw() published a different population from export_api_validated() (R/api.R). It formatted KEFS only. It now also reads the WCS merged landings (wcs-surveys-all_landings, the WCS preprocessed stage written by merge_landings()) and formats them through the same mapping path, mirroring the validated export. WCS catch_price and tot_catch_price are NA at this stage because prices are resolved during validation, consistent with the documented raw-stage schema.

  • KEFS catch_price published a price per kilogram, not a value (format_api_kefs(), R/api.R), now sample_weight * sample_price. The evidence that sample_price is a rate: it equals the form’s trip-level PricePerKg in 95.0% of single-species trips (against 2.3% for a value reading), is an exact multiple of 50 KSH in 83.9% of rows, and is essentially uncorrelated with sample_weight (Spearman -0.125, where a value correlates at 0.937). This matches how format_api_wcs() already derives catch_price, as a median KSH/kg multiplied by the row’s weight. The unit is now recorded in the function’s roxygen, alongside a description of the sampled-composition design, so neither is re-derived from the column names.

  • Removed a dead n_catch mutate in format_api_kefs() (R/api.R). The mutate assigned n_catch = as.integer(n_sample) and the following select() immediately overwrote it by renaming n_sample to n_catch, discarding the result. The integer cast is kept, applied to n_sample so it survives the rename.

Internal

  • Added the internal helper read_wcs_api_assets() (R/api.R, not exported) so both export stages share one copy of the Airtable taxa/gear/site/geo lookup that maps WCS surveys, rather than duplicating the block in the raw export.

Known remaining gap

KEFS (aNKjDfXDKyW3JLtaCdxDp5) still fails the invariant on 54.5% of its trips, and this release deliberately does not change that. KEFS is a sampled- composition design: the form records the whole catch (TotalCatchWeight, TValue) and separately identifies the species composition of a sample (OverallSampleWeight, reshaped to sample_weight/sample_price). The sample rows sum to total_sample_weight, not to total_catch_weight — measured at 80.5% exact agreement with total_sample_weight. Forcing them to agree would falsify the survey. A schema-level way to declare sampled composition is needed instead, and is out of scope here.

The catch_price unit error found alongside this is fixed above; it was a genuine defect rather than a property of the design. What remains is the design itself, and one consequence of it:

  • tot_catch_price for KEFS is derived, not summed: it equals total_catch_weight * total_price_kg in 100.0% of trips. It is a whole-catch valuation at a single trip-level price, which is why it cannot fully reconcile against per-species sample rows even with catch_price in the right units. The residual gap is the un-sampled portion of the catch, not an arithmetic error.
  • Declaring sampled composition in the export itself would need a schema addition — a sampled-weight column, or a flag distinguishing census from sample rows — which is out of scope for this release. Until then the design is documented only in format_api_kefs()’s roxygen, and consumers computing species composition from KEFS catch_kg should treat it as a sample, not a census.

peskas.kenya.data.pipeline 5.0.0

Major Changes

  • WCS artifacts moved to dedicated buckets: The entire WCS chain — ingestion, preprocessing, merging, validation and export_summaries() — now reads and writes conf$storage$google$options_wcs (kenya-wcs-dev / kenya-wcs-prod, in the separate peskas-wcs Google Cloud project) instead of the shared kenya-dev / kenya-prod buckets. This isolates WCS data so a WCS-only collaborator can run the full chain with credentials scoped to those two buckets and nothing else. Breaking: the WCS files must exist in the new buckets before the pipeline is run against them; a run pointed at the old bucket will find nothing.

  • Downstream consumers follow the split: export_api_validated() and compute_survey_matches() now read wcs-surveys-validated from the WCS bucket while continuing to read KEFS from the shared bucket. Their combined outputs remain in options_api / options — mixed artifacts are never written back into the WCS bucket.

  • New export_coasts_metrics(): The three cross-country portal artifacts (kenya_fishery_metrics, kenya_monthly_summaries_map, KE_regions) have been split out of export_summaries() into their own exported function, as they were the only outputs of the WCS chain landing outside the WCS bucket. Breaking: export_summaries() no longer produces them, and create_geos() is now called from export_coasts_metrics(). Callers relying on the old behaviour must call both functions; the scheduled pipeline does.

  • New internal helpers load_wcs_summary_inputs() and compute_monthly_summaries(): Extract the shared download, BMU metadata lookup, landing-site filter and monthly gap-filling so export_summaries() and export_coasts_metrics() can run independently without duplication.

Workflow

  • New WCS Pipeline workflow (.github/workflows/wcs-pipeline.yaml): runs the WCS chain on its own, carrying only the WCS Kobo and peskas-wcs credentials. Deliberately not scheduled — data-pipeline.yaml remains the scheduled pipeline and already covers the WCS chain, and a second cron writing the same prefixes would race it. export_summaries() is gated behind the repository variable WCS_ENABLE_EXPORT_SUMMARIES.
  • Extracted build-container.yaml as a reusable workflow, called by both pipelines so the image definition lives in one place.
  • Added export_coasts_metrics() as a step in data-pipeline.yaml, after export_summaries().

Configuration

  • Added storage.google.options_wcs (project peskas-wcs, bucket kenya-wcs-dev), overridden to kenya-wcs-prod in the production profile.
  • No WCS-specific profile was added, deliberately: the WCS chain uses options_wcs in every profile, so a WCS-only collaborator runs the standard profiles with no extra configuration.

Fixes

  • exclude_dashboard_ids silently excluded nothing: the entries were YAML aliases (*kobo_asset, *kobo_kf_asset) pointing at anchors that were never defined, so they resolved to the literal string _yaml.bad-anchor_ and matched no survey_id. Replaced with explicit !expr Sys.getenv() calls. Note: the two Kobo asset IDs are now genuinely excluded from the dashboard summaries, so those outputs will change.
  • read_config() no longer logs the resolved configuration: the previous log_debug("Running with parameters {pars}") printed the service account key, MongoDB connection strings, Kobo passwords and API tokens in plaintext. It now logs only the storage buckets in use. Also fixed the configutation typo in the adjacent info message.

Documentation

  • Added WCS-GUIDE.md, a standalone guide for WCS collaborators covering setup, the bucket boundary, configuration, the ten-step data flow, running locally and on Actions, and troubleshooting.
  • Added a WCS pipeline section to the pkgdown reference, populated by the new wcs roxygen keyword; tagged every function in the WCS chain with it. Functions combining WCS with KEFS or PDS data are deliberately excluded.
  • Added a pointer to the guide from the README.

Repository

  • Removed from version control: inst/reports/ (Quarto sources plus a 16 MB rendered HTML and the lumo extension), inst/model-tracks.R, inst/users_mng.R and inst/kenya_monthly_summaries.geojson. The files remain on disk; reports/ and inst/**/*.R are now gitignored.
  • Added WCS-GUIDE.md to .Rbuildignore.

peskas.kenya.data.pipeline 4.10.0

Major Changes

  • WCS survey support in GPS trip matching: merge_trips() now processes both KEFS and WCS surveys and uploads a single country-level matched-trips dataset to conf$surveys$matched_trips$file_prefix (kenya-surveys-matched-trips). Previously only KEFS was matched; the WCS branch referenced the wrong config key and was never executed.

  • New compute_survey_matches() internal function: Extracts the per-survey load + match + enrich pipeline so it can be called independently for each survey type. The device registry and GPS trips are loaded once and shared across both survey pipelines.

  • merge_trips() signature change: The survey argument has been removed. The function now always runs both KEFS and WCS and returns a unified result. Callers that previously passed survey = "kefs" should drop the argument.

Configuration

  • Added surveys.matched_trips.file_prefix: kenya-surveys-matched-trips as the single country-level output for matched trips.
  • Removed surveys.kefs.v2.merged (replaced by the above). Downstream consumers of kefs-v2-surveys-merged should migrate to kenya-surveys-matched-trips.

Fixes

  • Fixed match.arg(site) bug in merge_trips() (parameter is named survey, not site) that caused an immediate error on every call.
  • Fixed stale {site} glue reference in the registry loading log message.
  • Fixed export_api_raw() config key typo: conf$api$trips$rawfile_prefix → conf$api$trips$raw$file_prefix / conf$api$trips$raw$cloud_path.
  • Corrected export_api_raw() description: function exports KEFS only (WCS is included at the validated stage via export_api_validated()).

peskas.kenya.data.pipeline 4.9.0

Infrastructure & Workflow

  • Delegating to coasts most of the core storage and databse-related functions: Now core and other countries shared storage functions are delagated to central and upgraded features of the coastspacakge for improved standardization and maintainability

peskas.kenya.data.pipeline 4.8.0

Major Changes

  • Adopted coasts as the shared multicountry analytics engine: Aggregated data summarization and dashboard export are now delegated to WorldFishCenter/peskas.coasts (dev branch). This centralizes the logic for producing monthly, taxa, district, and gear summaries — as well as fishery metrics — across all Peskas country deployments (Zanzibar, Kenya, Mozambique), ensuring consistent outputs and a single place to maintain and improve the shared pipeline logic.
    • Added coasts to Imports and Remotes in DESCRIPTION
    • Added remotes::install_github("WorldFishCenter/peskas.coasts", ref = "dev") to both Dockerfile and Dockerfile.prod so the image ships the package
    • Pipeline steps that previously used local summarize_data() and generate_fleet_analysis() now call the equivalent coasts:: functions, passing package = "peskas.kenya.data.pipeline" so they read the country-specific inst/conf.yml

peskas.kenya.data.pipeline 4.7.0

Improvements

  • Standardized configuration structure: Replaced inst/conf.yml with a unified multi-country template harmonized across all Peskas deployments (Zanzibar, Kenya, Mozambique). Key structural changes:
    • Survey credentials moved from surveys.* into a new top-level ingestion.* section
    • Stage keys shortened (raw_surveys → raw, preprocessed_surveys → preprocessed, etc.)
    • Source names shortened (wcs_surveys → wcs, wf_surveys_v1 → wf_v1, etc.)
    • MongoDB structure reorganized: connection strings under connection_strings.*, databases under databases.*, collections key pluralized, portal renamed to dashboard
    • Airtable config moved from top-level airtable.* to metadata.airtable.*
    • All R code updated to use the new config paths

peskas.kenya.data.pipeline 4.6.0

New Features

  • GPS Trip Matching System: Added comprehensive fuzzy matching infrastructure to link catch surveys with GPS trip data
    • Implemented match_surveys_to_gps_trips() for universal two-step matching workflow (surveys -> registry -> trips)
    • Added merge_trips() for end-to-end Kenya KEFS-PDS data integration pipeline
    • Universal matching approach supports both explicit device registries (Kenya) and implicit registry construction (to test with Zanzibar)
    • Supports customizable Levenshtein distance thresholds for registration numbers (default 0.15) and names (default 0.25)
    • Implements conservative one-trip-per-day constraint to prevent ambiguous matches
  • Fuzzy Matching Infrastructure: Created robust text matching system for cross-dataset linking
    • Text normalization functions for cleaning boat identifiers (clean_text(), clean_registration())
    • Field-level Levenshtein distance matching with normalized thresholds (0-1 scale)
    • Multi-field matching strategy (registration number, boat name, fisher name) with per-field thresholds
    • Match quality metrics: n_fields_used, n_fields_ok, match_ok for transparency
    • Handles variant column names across datasets (vessel_reg_number, boat_reg_no, captain_name, etc.)

Workflow Integration

  • GitHub Actions Pipeline:
    • Added merge-trips-kefs-v2 job to automate survey-trip matching
    • Integrated matching step after KEFS v2 validation in production workflow
    • Exports merged dataset (matched + unmatched records) to cloud storage
  • Configuration Support:
    • Added merged trips output path configuration (surveys.kefs.v2.merged)
    • Integrated with existing PDS API credentials and cloud storage settings

Documentation

  • Added comprehensive documentation for 9 new functions with detailed examples:
    • match_surveys_to_gps_trips.Rd: Main workflow function with two-step matching process
    • merge_trips.Rd: Kenya-specific end-to-end pipeline
    • match_surveys_to_registry.Rd: Registry fuzzy matching algorithm
    • match_imei_to_trip.Rd: IMEI-date trip joining with uniqueness constraint
    • build_registry_from_trips.Rd: Implicit registry construction from historical trips
    • standardize_column_names.Rd, clean_matching_fields.Rd, clean_registration.Rd, clean_text.Rd: Helper functions

peskas.kenya.data.pipeline 4.5.0

New Features

  • API KEFS Data Export Pipeline: Added new export_api_raw() function to export raw preprocessed survey data in API-friendly format
    • Exports raw/preprocessed trip data (before validation) to cloud storage
    • Part of a two-stage API export pipeline (raw and validated exports)
    • Transforms nested survey data into flat structure with standardized trip-level records
    • Generates unique trip IDs using xxhash64 algorithm
    • Exports versioned parquet files to kenya/raw/ path for external API consumption
    • Includes comprehensive output schema with 14 standardized fields (trip_id, landing_date, gear, catch metrics, etc.)

Improvements

  • Configuration Enhancements:
    • Added api configuration section for trip data exports with separate raw/validated paths
    • Configured cloud storage paths for API exports (kenya/raw, kenya/validated)
    • Added Airtable base ID and token configuration for metadata management
    • Enhanced options_api storage configuration for peskas-coasts bucket
  • GitHub Actions Workflow:
    • Added new export-api-data job to automated pipeline workflow
    • Configured API export job to run after survey preprocessing step
    • Added production environment configuration for API data exports

peskas.kenya.data.pipeline 4.4.0

New features

  • KEFS V2 Validation Pipeline:
    • Added validate_kefs_surveys_v2() function for comprehensive validation of KEFS catch assessment surveys
    • Integrated KoboToolbox API validation status querying with get_validation_status() and update_validation_status() functions
    • Implemented multi-dimensional validation system with information, trip, catch, and indicator flags
    • Added support for manual validation override to preserve human-reviewed approvals while applying automated checks
    • Exported new validation helper functions: get_trips_flags(), get_catch_flags(), and get_indicators_flags()
    • Added GitHub Actions workflow job validate-kefs-catch-v2 for automated validation processing

Enhancements

  • KEFS Data Preprocessing Improvements:
    • Enhanced preprocess_kefs_surveys_v2() to include submission_date field for tracking when data was submitted
    • Added fishing_per_week field extraction and parsing from survey data
    • Improved catch outcome handling: automatically sets catch_outcome to “yes” when total_catch_weight > 0 but catch_outcome is NA
    • Renamed weight and price columns for clarity:
      • sample_weight → total_sample_weight
      • catch_weight → total_catch_weight
      • price_kg → total_price_kg
      • catch_price → total_catch_price
    • Enhanced enumerator name standardization integration
  • Survey Data Reshaping:
  • Configuration Updates:
    • Added KEFS_KOBO_TOKEN environment variable for KoboToolbox API authentication
    • Added validation flags file prefixes for both KEFS v1 and v2 in config.yml
    • Configured cloud storage paths for validation output files

Documentation

  • Enhanced Function Documentation:
    • Added comprehensive documentation for validate_kefs_surveys_v2() including detailed workflow description, validation limits, and requirements
    • Added documentation for KEFS validation helper functions:
      • get_trips_flags(): Documents trip-level validation for horse power, fishers, duration, and revenue with 6 alert codes
      • get_catch_flags(): Documents catch-level validation for sample weight inconsistencies with 2 alert codes
      • get_indicators_flags(): Documents composite indicator validation (CPUE, RPUE, price/kg) with 3 alert codes
    • Added documentation for KoboToolbox integration functions:
      • get_validation_status(): Retrieves validation status from KoboToolbox for submissions
      • update_validation_status(): Updates validation status in KoboToolbox
    • Updated get_indicators_flags() documentation to correctly reflect the clean_ids parameter
    • All new documentation follows roxygen2 style with comprehensive examples, parameter descriptions, and return value specifications
  • New man pages added:
    • get_catch_flags.Rd, get_indicators_flags.Rd, get_trips_flags.Rd
    • get_validation_status.Rd, update_validation_status.Rd
    • validate_kefs_surveys_v2.Rd

NAMESPACE

  • Exported new validation functions: get_trips_flags, get_catch_flags, get_indicators_flags
  • Exported KoboToolbox integration functions: get_validation_status, update_validation_status
  • Exported validate_kefs_surveys_v2 for KEFS V2 validation workflow

peskas.kenya.data.pipeline 4.3.0

Configuration

  • Environment-based Configuration:
    • Migrated authentication from file-based (auth/) to environment variable approach using .env files
    • Improved security and deployment flexibility by using environment variables for credentials
    • Added .env.example for reference configuration

Fixes

peskas.kenya.data.pipeline 4.2.0

New features

  • GPS Track Processing and Analysis:
    • Added process_fishing_tracks() for comprehensive GPS track processing and fishing activity classification
    • Implemented prepare_gps_data() to convert raw tracking data into GPSMonitoring format
    • Added classify_fishing_activity() using speed thresholds and spatial clustering to identify fishing vs transit activities
    • Created process_trajectories_with_speed() for calculating vessel speeds from GPS positions
    • Implemented calculate_fishing_summaries() to generate effort metrics by trip and spatial grid
    • Added visualize_fishing_track() for mapping fishing activities
    • Created create_exclusion_zones() and create_extent_polygon() for spatial analysis
  • Airtable Integration:
  • Fishery Metrics Expansion:
    • Extended get_fishery_metrics() with normalized long format output for maximum interoperability
    • Added metrics for CPUE and RPUE by gear type
    • Implemented species composition analysis with top 2 species ranking
    • Created fully normalized dataset structure for flexible aggregation and filtering
  • PDS Tracker Report:
    • Added comprehensive GPS tracking visualization report (inst/reports/pds/trackers.qmd)
    • Integrated interactive maps, time series, and network analysis visualizations
    • Implemented spatial heatmaps and trip trajectory analysis

Enhancements

  • Package Quality and Compliance:
  • KEFS Data Integration:
    • Updated configuration (inst/config.yml) for KEFS survey data integration
    • Enhanced inst/kefs_integration.R with improved data processing workflows

Documentation

  • Added 14 new man pages for GPS tracking functions:
    • process_fishing_tracks.Rd, prepare_gps_data.Rd, classify_fishing_activity.Rd
    • process_trajectories_with_speed.Rd, calculate_fishing_summaries.Rd
    • create_exclusion_zones.Rd, create_extent_polygon.Rd, visualize_fishing_track.Rd
  • Added 6 new man pages for Airtable integration:
    • airtable_to_df.Rd, df_to_airtable.Rd, update_airtable_record.Rd
    • bulk_update_airtable.Rd, get_writable_fields.Rd
  • Updated get_fishery_metrics.Rd to document dual function signatures for different use cases
  • Enhanced documentation with comprehensive examples and parameter descriptions

peskas.kenya.data.pipeline 4.1.0

Enhancements

  • Improved MongoDB Data Handling:
    • Enhanced mdb_collection_pull() to properly handle MongoDB ObjectId fields by explicitly including _id in queries and maintaining correct column ordering
    • Updated column reordering logic to prioritize _id field placement in retrieved datasets
    • Improved data consistency and integrity when pulling data from MongoDB collections
  • Code Style and Formatting:
    • Standardized function parameter formatting across storage functions for improved readability
    • Enhanced code consistency in upload_parquet_to_cloud(), cloud_object_name(), and related functions
    • Updated get_metadata() function formatting to follow consistent style guidelines
  • User Management Infrastructure:
    • Added comprehensive user management system for MongoDB dashboard with treatment group support
    • Implemented secure password generation and user creation functionality
    • Added support for BMU-based user access control and role assignment

peskas.kenya.data.pipeline 3.2.1

New features

  • Individual Fisher Metrics Data Extraction:
    • Added get_individual_gear_metrics() to calculates gear key fishery performance metrics at the individual fisher level from validated catch data, including CPUE (Catch Per Unit Effort), RPUE (Revenue Per Unit Effort), price per kg, trip costs, and profit margins for each fisher.
    • Integrated the fish composition distribution at fishers level

peskas.kenya.data.pipeline 3.2.0

New features

  • Individual Fisher Metrics Data Extraction:
    • Added get_individual_metrics() to calculates key fishery performance metrics at the individual fisher level from validated catch data, including CPUE (Catch Per Unit Effort), RPUE (Revenue Per Unit Effort), price per kg, trip costs, and profit margins for each fisher.

peskas.kenya.data.pipeline 3.1.0

New features

  • Individual Fisher Data Extraction:
    • Added get_individual_data() to extract fisher IDs and trip costs from raw survey data, enabling more granular analysis of fishing effort and expenses.
    • Integrated individual fisher and trip cost data into the version 2 preprocessing pipeline (preprocess_landings_v2()), with new columns for fisher_id and trip_cost in preprocessed outputs.

Enhancements

  • Expanded Data Merging:
    • Updated merge_landings() to support merging of legacy, v1, and v2 preprocessed landings, including new fields for individual fisher and trip cost data.
    • Improved column selection and ordering in merged datasets for consistency and downstream compatibility.
  • Export and Summarization Improvements:
    • Enhanced export_summaries() to include new individual-level statistics, such as mean trip expenses per fisher, aggregated by BMU and month (for data collected after June 25, 2025).
    • Improved monthly and distribution summaries to reflect the expanded data model.

Configuration

  • Updated config.yml to support new data paths and fields required for individual fisher and trip cost data in v2 surveys.

Documentation

  • Added new man page for get_individual_data().
  • Updated documentation for all affected functions to reflect new parameters and outputs.

peskas.kenya.data.pipeline 3.0.0

New features

Enhancements

  • Improved Data Processing Pipeline:
    • Refactored ingestion logic to handle multiple survey versions
    • Enhanced preprocessing workflow with version-specific transformations
    • Updated configuration structure to support versioned data paths
    • Improved documentation for all versioned functions

Documentation

  • Added comprehensive documentation for new versioned functions
  • Updated existing function documentation to reflect versioning changes
  • Added new man pages for version-specific functions
  • Enhanced function descriptions and examples

peskas.kenya.data.pipeline 2.1.0

New features

  • Geospatial Export Capabilities:
    • Added create_geos() function to generate GeoJSON files with regional-level fishery metrics
    • Implemented spatial analysis to assign BMUs to nearest coastal regions
    • Exported time series of aggregated metrics (CPUE, CPUA, effort, RPUE, RPUA) at regional level
    • Added regional polygon geometries for spatial visualization

Enhancements

  • Temporal Data Processing:
    • Revised fishery metrics calculation for more accurate temporal representation
    • Implemented daily-average approach instead of aggregate-then-divide method
    • Maintained per-day units (kg/fisher/day, KES/km²/day) consistent with scientific standards
    • Fixed CPUE and other daily metrics by calculating at daily level first
    • Filtered export data to include only records from 2023 onwards

Fixes

  • Data Quality Improvements:
    • Corrected handling of missing information with NA values instead of zeros
    • Fixed dependency logic of the pipeline
    • Improved price table prefix path handling
    • Updated configuration paths for downloading preprocessed price and legacy catch data

peskas.kenya.data.pipeline 2.0.0

New features

Enhancements

  • Improved Error Handling:
    • Added comprehensive input validation in all IQR functions
    • Added consistent NULL/empty data checks
    • Improved NA value handling using dplyr::if_else
    • Added warnings for cases where bounds cannot be calculated

Documentation

  • Added full documentation for all new IQR functions with examples
  • Updated NAMESPACE to export new IQR functions
  • Added proper Roxygen documentation for all new parameters

Other Changes

  • Switched default validation method from MAD to IQR in main pipeline
  • Maintained backward compatibility with existing MAD functions

New features

  • Migration to Google Cloud Storage:
    • Replaced MongoDB storage with Google Cloud Storage (GCS) using Parquet files
    • Added new cloud storage functions for GCS operations:
      • Authentication and connection management
      • Upload and download capabilities
      • Versioned file handling
    • Integrated Apache Arrow for efficient Parquet file processing
    • Updated configuration to support GCS file prefixes and paths

Enhancements

  • Storage Operations:
    • Refactored storage operations to use cloud-native approaches
    • Improved data access performance with Parquet file format
    • Added support for versioned file management

Dependencies

  • Added cloud storage related libraries
  • Added Arrow for Parquet file handling

peskas.kenya.data.pipeline 1.0.0

New features

  • Catch Validation Enhancements:
    • Added validate_fishers_catch() to flag cases where a single fisher reports an excessively high catch.
    • Introduced impute_price() to fill missing fish prices using median values across landing sites and sizes.
    • Integrated price validation in validate_landings() to ensure missing or zero prices are flagged.
  • Revenue Metrics in Summaries:
    • export_summaries() now calculates revenue-based effort metrics:
      • rpue (Revenue Per Unit Effort) as aggregated price per fisher
      • rpua (Revenue Per Unit Area) as aggregated price per square km
  • Extended Workflow Automation:
    • Updated GitHub Actions workflow (data-pipeline.yaml) to include a job for processing and merging price data.
    • Added price-related MongoDB collections (raw_price, preprocessed_price, price_table) to config.yml.

Enhancements

  • Improved Validation and Data Processing:
  • Refined Data Summarization:
    • export_summaries() now includes revenue metrics while maintaining prior CPUE and CPUA calculations.
    • Updated .qmd reports to reflect new price fields in the summary tables.

Fixes

  • Bug Fixes and Consistency Improvements:
    • Fixed potential issues in merge_prices() that could lead to duplicate price records.
    • Ensured impute_price() correctly applies median imputation across all relevant data points.
    • Updated .Rd documentation for all modified functions to reflect new parameters and changes.

peskas.kenya.data.pipeline 0.9.0

New features

  • Price Data Integration: Added support for processing fish price data alongside catch data.
  • Extended Workflow Automation:
    • Added a new GitHub Actions job merge-price-data to automate the processing and merging of price data.
  • Expanded MongoDB Collections:
    • Introduced new collections in config.yml for handling raw, preprocessed, and aggregated price data (raw_price, preprocessed_price, and price_table).

Enhancements

  • Validation and Alert Flagging:
    • Improved handling of validation alert flags by introducing dynamic flag_value parameters.
    • Adjusted anomaly detection in validate_landings() to use more descriptive alert codes.
  • Configuration Updates:
    • Updated .github/workflows/data-pipeline.yaml to include price survey ingestion and processing.
    • Extended config.yml to support price data processing in both local and production environments.
  • Namespace Expansion:
    • Exported new functions (ingest_landings_price, merge_prices, preprocess_price_landings, summarise_catch_price) to make them available for package use.

Fixes

  • Documentation Corrections:
    • Updated .Rd documentation for ingest_landings() to clarify that it processes catch surveys (not all WCS surveys).
    • Added descriptions for new parameters (flag_value) in validation function documentation.
  • Bug Fixes:
    • Resolved inconsistencies in price data processing by ensuring column renaming and format standardization.
    • Fixed logical errors in merging price data to prevent duplicate records.
    • Improved MongoDB push operations to correctly handle newly introduced price data collections.

peskas.kenya.data.pipeline 0.8.0

New features

  • Introduced flag_value parameter to validation functions (validate_dates, validate_nfishers, validate_nboats, and validate_catch) for customizable alert thresholds.
  • Added logical_check and anomalous_submissions workflows to improve anomaly detection in validate_landings.
  • Enhanced data upload pipeline with purrr::walk2 for streamlined MongoDB uploads of validated and flagged data.

Enhancements

  • Updated export_summaries to calculate mean_trip_catch using the median for improved robustness against skewed data.
  • Improved documentation for validation functions to include detailed descriptions and examples for the new flag_value parameter.
  • Refined the logic in validate_landings for clearer alert flagging and better handling of edge cases.
  • Added the validation_flags collection to the configuration file (config.yml) for centralized management of flagging outputs.
  • Updated anomaly alert flags to use descriptive string values instead of numeric codes for better interpretability.

Fixes

  • Fixed inconsistencies in the handling of alert flags during data validation.
  • Resolved potential mismatches in merged datasets by improving join logic in validate_landings.
  • Corrected documentation typos and improved consistency across .Rd files.

peskas.kenya.data.pipeline 0.7.0

New features

  • Introduced get_total_catch_bounds for calculating upper bounds on total catch data grouped by landing site and gear type.
  • Added validate_total_catch function for validating total catch data with improved outlier handling.
  • Updated validate_landings to identify and filter out inconsistent submissions (e.g., mismatched fishers and boats).
  • Enhanced validate_catch to set outlier values in catch_kg to NA and flag them for review.

Enhancements

  • Improved get_catch_bounds to exclude invalid categories and clarified grouping logic for gear and fish categories.
  • Updated documentation across validation and catch bounds functions to include better descriptions, keywords, and return values.
  • Refined data preprocessing in preprocess_landings to ensure unique rows, handle NA values, and streamline workflow.
  • Adjusted configuration parameters in config.yml to enhance validation sensitivity (e.g., increasing k_catch for outlier detection).
  • Updated export_summaries to improve monthly summaries by recalculating effort as fishers per km² per day and refining CPUE and CPUA metrics to include temporal normalization. Simplified mean trip catch calculations for consistency across datasets.

Fixes

  • Corrected plot code formatting in data_report.qmd for CPUE, effort, and CPUA visualizations.
  • Fixed documentation typos and inconsistencies in .Rd files for validation functions.

peskas.kenya.data.pipeline 0.6.0

New features

  • Add merge_landings function for combining data sources
  • Add validation catch function to improve data quality checks
  • Include form consent and total catch fields in merged data
  • Calculate total catch in legacy landings

Enhancements

  • Homogenise gear and fish groups names
  • Update validation code structure
  • Fix mismatch between fish groups sum and total catch
  • Fix landing sites names
  • Improve validation catch function
  • Update export data based on validation
  • Fix legacy submission IDs

peskas.kenya.data.pipeline 0.5.1

New features

  • Add functions to extract effort and CPUE from validated legacy data
  • Implement MongoDB pushing functionality
  • Add storage and metadata functions

Enhancements

  • Index functions by topic and package functional position
  • Update package documentation and website
  • Improve MongoDB storage calls
  • Add configuration file for MongoDB database and collection references

Fixes

  • Drop stale files
  • Update storage calls for better efficiency
  • Fix various typos and syntax errors

peskas.kenya.data.pipeline 0.5.0

Enhancements

  • Improve mongoDB storage functions by adding indexes to improve query performance. Before this fix column order in data was not preserved.

peskas.kenya.data.pipeline 0.4.0

New features

  • Now ingest_surveys() uses Kobotoolbox API directly to download surveys instead of rely on R package

Fixes

  • Fix ingestion functions as it was limited to download max 30,000 submissions. The approach now uses pagination to retrieve large datasets, with a limit of 30,000 records per request

peskas.kenya.data.pipeline 0.3.0

New features

  • Add ingestion function to ingest ongoing data ingest_surveys()
  • Add preprocessing function to preprocess ongoing data preprocess_landings()

peskas.kenya.data.pipeline 0.2.0

  • Add package website with documentation functions
  • Integrate mongodb storage with package functions
  • Optimize configuration file

peskas.kenya.data.pipeline 0.1.0

  • Initial CRAN submission.