Changelog

peskas.mozambique.data.pipeline 2.9.2

Bug Fixes

Three defects found by a cross-country audit of the validated parquet files published to gs://peskas-api-prod. Measured against trips-validated__20260913030758_763060b__.parquet (2,193 rows, 1,605 trips) and trips-raw__20260913030600_763060b__.parquet (2,949 rows, 1,989 trips).

• A raw form code was reaching the API in catch_habitat: 16 published rows across 12 validated trips carried catch_habitat = "3" — a literal 3, sitting in a column whose other values are habitat names. The ADNAP habitat mapping in preprocess_landings_adnap() was a case_when() over the current form’s string codes with a TRUE ~ .data$habitat fallthrough, so any code it did not recognise was published verbatim. Two early versions of the form (vWMxUBnadxLX3ashwjxLKj, vD3vqVYfJGPqEoxLKn4FbR) code the same KoBo choice list dx7qi97 numerically, before it was renamed and extended: 1 reef, 2 FAD, 3 deep sea (the option later renamed opsea), 4 shore, 6 mangrove, 7 seagrass. Those six codes are now mapped. Data impact: validated export, 16 rows across 12 trips change from "3" to "Open sea"; raw export, 19 rows across 15 trips — 18 from "3" to "Open sea" and 1 from "4" to "Shore". No other value changes, and NA still passes through as NA.

• An unmapped habitat code now fails the run instead of being published: the fallthrough was the real defect — it made a habitat code the pipeline had never seen indistinguishable from one it understood, so the next renamed choice would land in production the same way. preprocess_landings_adnap() now stops on any raw group_trip/habitat value with no entry in the mapping, naming the codes and pointing at the KoBo versions endpoint that carries the choice list. preprocess_landings_lurio() keeps its own numeric mapping; its form emits only codes 1 2 3 4 6 7, all of which it covers, and it does not feed the API export.

• n_fishers = 0 now raises alert 11 instead of being published: the API schema declares n_fishers with a minimum of 1, but a trip whose three fisher counts were all zero was published as 0, and every per-fisher metric divided by it into Inf, which survives mean() and median() into the portal. Zero fishers is not a count — every one of these submissions records survey_activity = 1, a fishing_start, a fishing_end and a habitat, so the trip happened and the crew was simply never entered. Alert 11 already covered exactly this, but was narrowed by & catch_outcome == "1", so the identical defect was excluded on a landed catch and published on a no-catch trip: of the 42 preprocessed submissions with all-zero counts, 37 were already dropped and 5 were not. The catch-outcome condition is removed, so the flag now fires on the zero itself. Data impact: 4 submissions leave the validated set (1,605 → 1,601 trips, 2,193 → 2,189 rows), removing all 4 published n_fishers = 0 rows. All four carry catch_outcome = "0" and catch_kg = 0, so no catch is lost. They are not discarded — they surface in the flags-adnap and enumerators_stats-adnap collections for the enumerator to correct at source. validate_landings_lurio() keeps the narrower condition for now; widening it there would newly exclude 570 submissions (1.44% of its validated set), which is a portal-data decision beyond this audit.

• tot_catch_kg was summed before distinct(), not after: both export functions computed tot_catch_kg = sum(catch_kg) grouped by trip_id and only then called dplyr::distinct(). Any row the deduplication removed was still counted in the total, so tot_catch_kg would have exceeded sum(catch_kg) for that trip — the one invariant the API export exists to hold. distinct() now runs before the total is taken, so the sum covers exactly the rows that reach the file. Data impact: none today. distinct() removes 0 rows from both exports (2,949 raw, 2,193 validated), because preprocess_landings_adnap() already deduplicates upstream, and both orderings produce an identical row set. This was latent: it would have bitten the first time two genuinely identical catch rows shared a trip, and it would have failed silently, since nothing downstream rechecks the total.

• A no-op relocate() in both API exports: dplyr::relocate(c("catch_price", "tot_catch_kg", "tot_catch_price"), .after = "catch_price") relocated catch_price after itself. Now .after = "catch_kg", matching Kenya and Zanzibar. Data impact: none — the preceding select() already fixes the order, and both spellings produce the identical 22-column order byte-for-byte against the published file. Latent, not active.

tot_catch_kg == sum(catch_kg) within trip_id still holds at 0 failures across all 1,605 validated trips; none of these changes touch that derivation.

peskas.mozambique.data.pipeline 2.9.1

Refactor

• The total-length restatement now comes from coasts: get_length_conversions() and convert_lw_to_tl() are deleted; get_length_weight_batch() calls coasts::convert_lw_to_tl() instead. The same arithmetic existed here, in Zanzibar and in Timor. coasts passes unconvertible rows through rather than dropping them, so the call site keeps Type == "TL" to preserve the behaviour this pipeline had. Verified over 71 taxon codes at FishBase 25.04 / SeaLifeBase 24.07, area 51 — the alias table’s 28 codes included: the same 24 rows restated for the same 10 taxa, and the resulting table is identical() across all 68 codes that carry coefficients.
• coasts (>= 4.13.0) is now a declared floor in DESCRIPTION.

peskas.mozambique.data.pipeline 2.9.0

Bug Fixes

Ports the FishBase import and coefficient fixes from peskas.zanzibar.data.pipeline 4.9.0. The mechanisms are identical; the species pool, the alias table and the coverage baseline are Mozambique’s own, measured against the live KoBo data for both forms (Lurio and ADNAP).

• FishBase releases are now pinned, and a missing coefficient fails the run: rfishbase reads a remote parquet dataset over the network, so an unpinned "latest" let a new FishBase release reach the pipeline the moment a container was rebuilt, with no code change. Release 26.06 dissolved Caesionidae into Lutjanidae and Scaridae into Labridae — both family names survive with zero species in them — so any taxon named after one expanded to nothing, got no length-weight coefficients, and weighed NA, which sums to zero. Here that is CJX (Caesionidae, 1,683 Lurio rows) and PWT (Scaridae, 6,536 rows). The releases are now pinned per server in inst/config.yml under metadata:fishbase and threaded through all five rfishbase reads in getLWCoeffs(), which previously could mix snapshots within a single run. rfishbase is additionally pinned to 5.0.1 in both Dockerfiles as the last install step, because remotes::install_local(dependencies = TRUE) upgrades it otherwise.

• assert_taxa_coverage() fails the run when a taxon resolves to no coefficients: previously a taxon that matched nothing was dropped in silence and the pipeline stayed green while publishing a hole. Called from both preprocess_landings_lurio() and process_version_data(), after the manual FLY coefficient is pooled in.

• Unmatched taxa are now logged: match_species_from_taxa() dropped any name that matched no species without a warning. That warning is what the alias table below was built from — it reported 30 unmatched names on the first run here.

• SeaLifeBase routing corrected: process_species_list() routed only ISSCAAP groups 57, 45, 43, 42 and 56 to SeaLifeBase, sending sea cucumbers, gastropods, oysters, mussels, scallops and mantis shrimp to FishBase, where they matched nothing and were dropped. Routing is now ISSCAAP >= 40.

• Species names ending in “idae” are no longer read as families: the rank test placed the family suffix before the species test, so a species like Haliotis midae was searched as a family and matched nothing.

• Length-type conversion recovers taxa that weighed NA (get_length_conversions(), convert_lw_to_tl()): FishBase tags every published length-weight pair with the length type the original study measured, and for tunas, billfish and several carangids that is fork length. get_length_weight_batch() kept only Type == "TL", so those taxa got no coefficients at all and every length-measured catch row of them weighed NA. The conversions are published data in FishBase’s POPLL table, which this pipeline never read. Reading it restates a on a total-length basis (a_TL = a * ratio^b, b unchanged) and recovers 18 codes across the two forms: ARQ BAN BET CFX CJC CJZ FLY HDH HES HSU JOA LTQ MLS NGR NXP NXT ZEV AVR.

• Morphology bounds no longer silently disable length validation: min(CommonLength, na.rm = TRUE) returns Inf for a taxon whose matched species all lack that field, and the permissiveness step then computed Inf - 0.75 * Inf = NaN. Every comparison against NaN is NA, which case_when() treats as no-match, so the length alert codes never fired for those taxa — a missing bound was indistinguishable from a passed check. safe_min() now yields NA rather than Inf, and because FishBase populates CommonLength for only 10% of species against 91% for Length, missing values are estimated as 0.625 * Length (common_length_ratio()). All taxa with morphology now have usable bounds (283 on the dev pool, 290 on production). Expect a wave of new length alerts on the first run: those records were never checked before.

• Search-name aliases fix 28 taxa the ASFIS names could not match (taxa_search_aliases(), apply_taxa_aliases()): a handful of ASFIS reference names match nothing in the taxonomic backbone, so the taxon is dropped and every catch row of it weighs NA. Every row was derived by looking the ASFIS name up in the synonym table for the pinned release. Most are genus splits — Carangoides across Ferdauia, Platycaranx, Atropus and Turrum; Sepia across Rhombosepion, Ascarosepion and Acanthosepion — plus spelling drift (ESR, PKT, RPO, SYQ, ZEV, LGE) and two broken ASFIS strings: HES is truncated to Herklotsichthys quadrimaculat. and GRX carries the parenthetical Haemulidae (=Pomadasyidae), whose embedded space made the rank rule read a family as a species. VMX is Valamugil, a genus the backbone no longer carries, so Osteomugil and Moolgarda are both searched. Coverage goes from 237 to 263 of 295 codes, and unmatched names from 30 to 2.

  CRA (“marine crabs nei”, Brachyura) and CUX (“sea cucumbers nei”, Holothuroidea) are deliberately not aliased: both are ranks match_species_from_taxa() cannot search, and choosing a target means deciding which crab or holothurian families Mozambique lands. CUX is the largest single loss in the baseline, at 972 Lurio rows.

• OQC now gets the octopus mantle-length conversion: OCZ (Octopus spp, 5,990 Lurio rows) was special-cased in three places — an ML-only coefficient filter, the arm-span-to-mantle /5.5 conversion, and the min_length floor — but OQC (Octopus cyaneus, ADNAP) was not. OQC resolves to a mantle-length pair, so applying it to arm-span unconverted weighed a single octopus at 264 kg instead of 2.43 kg. This was latent while OQC had no coefficients and would have gone live with the alias above. The conversion is applied in both calculate_catch_adnap() and calculate_catch_lurio(), since the two forms carry one octopus code each.

• Fixed a duplicate join key for FLY: both preprocessing paths appended a hardcoded flying-fish coefficient unconditionally. Now that the conversion recovers Exocoetidae pairs, getLWCoeffs() returns a FLY row of its own, and two rows on the same key would have doubled every flying fish catch record. The manual value now replaces rather than appends, and stays authoritative: changing it is a separate decision.

• Removed a dead fallback in process_version_data(): the tryCatch around getLWCoeffs() read inst/length_weight_params.rds, which is not in the package, so the fallback could only ever fail — while hiding the original error behind it.

Known Issues

Measured 2026-09-06 against FishBase 25.04 / SeaLifeBase 24.07 over the production KoBo data for both forms: 263 of 295 codes resolve length-weight coefficients, up from 237 before this release (ADNAP 230/260, Lurio 52/55). The other 32 form the documented baseline in assert_taxa_coverage(), so any new loss fails the run. CJX and PWT are deliberately not in it — they resolve at 25.04 and are the two codes that break at 26.06, so a release move fails the check.

Measure the baseline against production, not dev. The first CI run failed on LHV, an ADNAP code absent from the dev bucket, whose raw files lagged production by three weeks and 8 codes (BIG BLR FLI LHV NXU OCZ PNQ RRU; the other 7 all resolve).

• Not a taxon (1) — MZZ (Actinopterygii), dropped before the search.
• A rank the matcher cannot search (2) — CRA (infraorder Brachyura) and CUX (class Holothuroidea).
• Wrong reference name (2) — AND, NAI name species absent from FAO 51.
• No published coefficients (18) — ADT CJV CWC ECG EFZ EJX GQT GQV ICZ NUH OCN OIC PEJ PKF RDR TCI UVG YFK occur in FAO 51 but carry no length-weight pair in any length type. Nothing to convert, nothing to alias.
• Only a doubtful pair (2) — LHV (Lethrinus variegatus) and TEC (Pterocaesio chrysozona) each have exactly one published pair, flagged EsQ = "Yes" by FishBase itself, which get_length_weight_batch() drops on purpose. Recovering either means overriding FishBase’s quality flag.
• No usable length type (7) — HMP (SL), PKV and QCY (FL), RMB (disc width), and EFB, EFN, KAK, which record no length type at all.

Not ported

Zanzibar’s retired-code remaps (AHI→BAF, BFL→TEI) and its SR/MAC→ AQX correction come from Zanzibar’s forms; none of those three codes occurs in Mozambique’s data, so no remap was added. The existing TUN→TUS, SKH→CVX and CLP→ANX remaps in preprocess_landings_lurio() are the local equivalents and are unchanged. Because CLP is already remapped at the survey level here, Zanzibar’s CLP→Dorosomatidae alias is not needed.

Two metadata gaps were found while measuring and are not code fixes: FOT (Eleutheronema tetradactylum, 17 ADNAP rows) is a valid ASFIS code missing from the Airtable taxa table, so map_surveys() leaves those rows without a scientific or English name; and Lurio survey_label 27 (43 rows) has no Airtable row at all, orphaning those submissions. Both need an Airtable edit, not a remap — rewriting either code would mislabel real catch.

peskas.mozambique.data.pipeline 2.8.0

Infrastructure & Workflow

• Delegating to coasts most of the core storage and databse-related functions: Now core and other countries shared storage functions are delagated to central and upgraded features of the coastspacakge for improved standardization and maintainability

peskas.mozambique.data.pipeline 2.7.0

Improvements

• Standardized configuration structure: Replaced inst/conf.yml with a unified multi-country template harmonized across all Peskas deployments (Zanzibar, Kenya, Mozambique). Key structural changes:
  • Survey credentials moved from surveys.* into a new top-level ingestion.* section
  • Stage keys shortened (raw_surveys → raw, preprocessed_surveys → preprocessed, etc.)
  • Source names shortened (wcs_surveys → wcs, wf_surveys_v1 → wf_v1, etc.)
  • MongoDB structure reorganized: connection strings under connection_strings.*, databases under databases.*, collections key pluralized, portal renamed to dashboard
  • Airtable config moved from top-level airtable.* to metadata.airtable.*
  • All R code updated to use the new config paths

peskas.mozambique.data.pipeline 2.5.1

New Features

• Trip-Survey Merging Pipeline: Added new merge_trips() function to combine GPS tracker data with survey landings
  • Merges PDS trip data with validated survey submissions based on device IMEI and landing date
  • Smart matching logic joins only when there is exactly one trip and one survey per device per day
  • Records with multiple trips or surveys per day are preserved but not joined to maintain data integrity
  • Outputs merged dataset to cloud storage for downstream analytics
  • New GitHub Actions job runs automatically after validation completes

Improvements

• Device Registry Optimization: Enhanced PDS data ingestion performance
  • Changed from direct Airtable queries to cached device registry stored in cloud storage
  • Reduces API calls and improves pipeline execution speed
  • Device metadata now loaded from versioned RDS files
• Survey Preprocessing Enhancement:
  • Added boat_pds field to ADNAP preprocessing pipeline for better device tracking
  • Enables linking survey data with GPS tracker information during merge operations
• Configuration Updates:
  • Added merged_surveys configuration section with file prefix for merged trip-survey data
  • Enhanced workflow dependencies to ensure proper sequencing of merge operations

Code Quality

• Improved function parameter formatting for better code readability
• Enhanced documentation for PDS ingestion functions

peskas.mozambique.data.pipeline 2.5.0

New Features

• API Data Export Pipeline: Added new export_api_raw() function to export raw preprocessed survey data in API-friendly format
  • Exports raw/preprocessed trip data (before validation) to cloud storage
  • Part of a two-stage API export pipeline (raw and validated exports)
  • Transforms nested survey data into flat structure with standardized trip-level records
  • Generates unique trip IDs using xxhash64 algorithm
  • Exports versioned parquet files to mozambique/raw/ path for external API consumption
  • Includes comprehensive output schema with 14 standardized fields (trip_id, landing_date, gear, catch metrics, etc.)

Improvements

• Configuration Enhancements:
  • Added api configuration section for trip data exports with separate raw/validated paths
  • Configured cloud storage paths for API exports (mozambique/raw, mozambique/validated)
  • Added Airtable base ID and token configuration for metadata management
  • Enhanced options_api storage configuration for peskas-coasts bucket
• GitHub Actions Workflow:
  • Added new export-api-data job to automated pipeline workflow
  • Integrated Airtable authentication with GitHub Secrets (AIRTABLE_TOKEN, AIRTABLE_BASE_ID_FRAME, AIRTABLE_BASE_ID_ASSETS)
  • Configured API export job to run after survey preprocessing step
  • Added production environment configuration for API data exports

peskas.mozambique.data.pipeline 2.4.0

Major Changes

• Lurio landings export pipeline: Added export_lurio_landings() to publish portal-ready collections to MongoDB after validation.
  • Generates monthly metrics, site stats, taxa length distributions, taxa composition per site, gear/habitat CPUE-RPUE metrics, and geo indicators
  • New create_metric_structure() helper builds ApexCharts JSON series for gear/habitat views
  • Added GitHub Actions job to run the Lurio export after validate-lurio

Improvements

• Lurio validation output:
  • Exclude flagged submissions from validated parquet exports
  • Map catch_taxon to alpha3_code and de-duplicate records for cleaner downstream exports
• Landings export metrics:
  • Use medians for CPUE, RPUE, and price per kg summaries to reduce outlier sensitivity
  • Temporarily align landing_date with submission_date in exports pending validation fixes
  • Pause fishery metrics parquet export in export_landings() while outputs are reviewed

Infrastructure & Dependencies

• Added jsonlite dependency for JSON export
• Ignore .geojson artifacts in git

peskas.mozambique.data.pipeline 2.3.1

Major Changes

• Streamlined Validation Workflow: Replaced KoboToolbox API updates with direct MongoDB storage to improve performance.
  • New export_validation_flags() function exports validation flags directly to MongoDB
  • Validation status queries now only identify manually edited submissions, not update them
  • Disabled sync_validation_submissions() workflow steps in GitHub Actions
  • Significantly reduced pipeline execution time by avoiding slow KoboToolbox API calls

Improvements

• Validation System:
  • Validation functions now preserve manual human approvals while updating system-generated statuses
  • Added fetch_error field to get_validation_status() for better error tracking
  • Improved error handling in validation status queries
• Code Quality:
  • Fixed SeaLifeBase API calls by pinning to version 24.07 to avoid server errors
  • Standardized function parameter formatting across validation and preprocessing modules
  • Removed empty R/data.R file
• Pipeline Configuration:
  • Removed survey activity filter in Lurio preprocessing to include all submissions

peskas.mozambique.data.pipeline 2.3.0

Major Changes

• Enumerator Name Standardization: New intelligent name cleaning and matching system to handle data entry inconsistencies.
  • Introduced standardize_enumerator_names() function with fuzzy string matching using Levenshtein distance
  • Automatically removes special characters, numbers, and extra whitespace from enumerator names
  • Matches similar names with typos (e.g., “john smith” and “jhon smith”) and consolidates to a standard form
  • Integrated into Lurio preprocessing pipeline with configurable distance threshold
  • Marks single-word entries as “undefined” to ensure quality control
  • Returns cleaner, more consistent enumerator tracking for performance analysis
• Enhanced Validation System: Improved validation logic with manual approval tracking and new quality checks.
  • Added manual approval tracking from KoboToolbox to distinguish human-reviewed approvals from system approvals
  • Manual approvals by human reviewers now properly bypass automatic validation flags
  • New validation flag 20: Detects landing date after submission date inconsistencies
  • New validation flag 11: Flags zero fishers with positive catch outcome
  • Parallel processing for KoboToolbox validation status queries using furrr for faster bulk validation
  • Improved handling of infinite CPUE/RPUE values in composite indicator validation
  • Enhanced price per kg threshold from 1,875 to 2,500 MZN (~30 EUR) for more realistic outlier detection

Improvements

• Data Structure Enhancements:
  • Added scientific_name field to catch data for better species traceability and validation
  • Renamed tot_fishers to n_fishers throughout codebase for naming consistency
  • Fixed column name inconsistency where survey_label and catch_taxon were swapped in some contexts
  • Improved data pipeline clarity with consistent field naming across preprocessing and validation stages
• Validation Pipeline Improvements:
  • Enhanced validation logic to handle edge cases (infinite values, zero denominators)
  • Better separation of basic quality checks (flags 1-7, 20) from composite indicators (flags 8-11)
  • Improved flag consolidation logic to properly aggregate multiple validation issues per submission
  • More robust handling of submissions with missing or invalid fishers count
  • Enhanced logging for validation status queries with submission counts
• Code Quality:
  • Fixed assignment operators (= → <-) for R style consistency
  • Improved function parameter formatting throughout codebase
  • Added proper roxygen2 documentation for new standardize_enumerator_names() function
  • Enhanced inline comments explaining validation thresholds and logic

Bug Fixes

• Fixed infinite CPUE/RPUE calculations when fishers count or trip duration is zero
• Corrected validation flag order to properly prioritize data quality issues
• Fixed column mapping issue in Lurio validation where survey labels were misaligned
• Resolved issue where submission date validation was not being performed
• Fixed enumerator name column selection to use correct nested field path

Infrastructure & Dependencies

• Added stringdist package dependency for fuzzy name matching capabilities
• Enhanced parallel processing configuration for validation status queries
• Updated NAMESPACE with new exported function standardize_enumerator_names()
• Added man page documentation for enumerator name standardization function

peskas.mozambique.data.pipeline 2.2.0

Major Changes

• Redesigned Length Frequency Processing: Complete rebuild of catch data reshaping with simplified, row-by-row processing architecture.
  • Introduced expand_length_frequency() for processing individual species rows
  • Refactored reshape_catch_data() to use row-wise expansion instead of complex joins
  • Eliminated data loss issues caused by multiple join operations
  • Preserves all metadata (counting_method, species, n_buckets, etc.) throughout transformation
  • Simpler and more maintainable code with clear step-by-step logic

Improvements

• Enhanced Catch Data Processing:
  • Fixed critical bug where counting_method was being lost during length frequency expansion
  • Improved handling of NA values in separate_wider_delim() with too_few = "align_start"
  • Better support for length frequency data (fish under 100cm) with proper regex pattern matching
  • Clearer inline documentation explaining each processing step
  • More robust error handling for empty length bins
• Code Architecture:
  • New expand_length_frequency() function processes one species row at a time
  • Deprecated process_regular_length_groups() in favor of simpler row-by-row approach
  • Retained process_over100_length_groups() for backwards compatibility with large fish data
  • Eliminated complex join logic that was prone to losing metadata
  • Uses rowwise() |> group_split() |> map_dfr() pattern for cleaner row processing
• Documentation Quality:
  • Updated all function documentation to reflect new implementation
  • Added detailed @details sections explaining the row-by-row approach
  • Improved @keywords for better pkgdown organization
  • Clear documentation of deprecated functions
  • Enhanced examples showing length frequency analysis

Bug Fixes

• Fixed counting_method = NA issue where metadata was lost during length data expansion (#issue)
• Corrected regex pattern to match no_individuals_5_10 format (was looking for 5_10 only)
• Fixed separate_wider_delim() failure on NA length ranges
• Eliminated extra length group columns appearing in final output
• Resolved data preservation issues in complex join operations

Technical Details

• Length Frequency Data Flow:
  • Old approach: Extract all length data → Join back → Lose metadata
  • New approach: Process each row → Expand in place → Keep everything
  • Result: 100% metadata preservation with simpler logic
• Performance: Row-by-row processing with purrr::map_dfr() provides clean, functional approach while maintaining good performance for typical survey sizes

peskas.mozambique.data.pipeline 2.1.0

Major Changes

• Enhanced Validation Sync System: Restructured validation synchronization following Kenya pipeline best practices.
  • Added sync_validation_submissions() for bidirectional validation status updates with rate limiting
  • Implemented process_submissions_parallel() helper function for consistent API interactions
  • Rate limiting (0.1-0.2s delays) prevents overwhelming KoboToolbox API
  • Manual approval respect: Human review decisions are never overwritten by system updates
  • Optimized API usage: Skips already-approved submissions to minimize unnecessary calls
  • Fetches current validation status BEFORE making updates for smarter decision-making
  • Automated approval/rejection of submissions in KoboToolbox based on validation results
  • Stores validation metadata in MongoDB for enumerator performance tracking
  • Enhanced error tracking with success/failure logging
• Improved Asset Management: Enhanced preprocessing with form-specific asset filtering.
  • Preprocessing functions now automatically filter Airtable assets by form_id
  • Better separation of metadata between Lurio and ADNAP survey forms
  • Improved handling of shared assets across multiple survey versions
  • More reliable species, gear, vessel, and site mappings
• Optimized GitHub Actions Workflow: Streamlined pipeline execution by combining ingestion and preprocessing stages.
  • Combined ingest and preprocess jobs for each data source (Lurio, ADNAP, PDS) reducing container startups by ~30%
  • Simplified dependency graph from 10 jobs to 7 jobs for faster pipeline execution
  • Maintained separation of validation stages for better error isolation and independent re-runs
  • Aligned workflow structure with Kenya pipeline best practices

Improvements

• Validation System Enhancements:
  • Manual approvals by human reviewers now properly bypass automatic validation flags
  • System-generated approvals are re-validated to ensure data quality
  • Better logging of validation status queries with submission counts
  • Enhanced validation flag preservation for monitoring and reporting
  • Improved handling of catch_taxon field mapping in Lurio surveys
• Configuration Management:
  • Restructured MongoDB connection strings to support separate validation database
  • Added KOBO_TOKEN authentication for ADNAP asset
  • Improved configuration structure for multiple database contexts
  • Enhanced PDS storage configuration organization
  • Added explicit assets configuration for Airtable integration
• Workflow Performance:
  • Reduced overall pipeline execution time through job consolidation
  • Parallel execution of independent data streams (Lurio, ADNAP, PDS)
  • Cleaner job naming for better CI/CD monitoring
  • Maintained robust error handling with granular validation stages
• Code Quality:
  • Added new exported functions: summarize_data(), sync_validation_submissions(), process_submissions_parallel()
  • Enhanced function documentation with proper importFrom declarations
  • Improved variable scoping and data pipeline clarity
  • Better separation of concerns between preprocessing and validation
  • Centralized API interaction logic in reusable helper functions

Bug Fixes

• Fixed asset fetching logic to properly filter by target form_id
• Corrected catch_taxon column mapping in Lurio validation (changed to alpha3_code)
• Fixed validation status query to exclude system approvals from manual approval overrides
• Fixed MongoDB configuration path typo (collection → collections) in enumerators_stats
• Removed redundant asset fetching code in preprocessing functions
• Added missing KOBO_USERNAME configuration for ADNAP asset
• Fixed sync function to never overwrite manual approvals from human reviewers

Infrastructure & Dependencies

• Added MONGODB_CONNECTION_STRING_VALIDATION environment variable for separate validation database
• Enhanced GitHub Actions workflow with combined ingest-preprocess jobs
• Improved parallel processing configuration in validation sync
• Updated NAMESPACE with new imports for future, furrr, and progressr packages
• Maintained compatibility with existing storage and authentication systems

peskas.mozambique.data.pipeline 2.0.0

Major Changes

• Dual Survey System Integration: Full support for both Lurio and ADNAP fisheries surveys with parallel processing workflows.
  • Added ingest_landings_lurio() and ingest_landings_adnap() for separate survey data streams
  • Implemented preprocess_landings_lurio() and preprocess_landings_adnap() with survey-specific transformations
  • Created validate_surveys_lurio() and validate_surveys_adnap() with tailored validation rules
  • Added calculate_catch_lurio() and calculate_catch_adnap() for survey-specific catch weight estimation
  • Introduced survey version detection and adaptive processing via reshape_catch_data()
• Enhanced Validation System for ADNAP: Advanced validation with KoBoToolbox integration.
  • Integrated KoBoToolbox validation status API for manual approval workflow
  • Added get_validation_status() to query submission approval status
  • Implemented parallel processing for validation status queries across multiple submissions
  • Manual approvals in KoBoToolbox now bypass automatic validation flags
  • Maintained two-stage validation (7 basic checks + 3 composite economic indicators)
• Flexible Survey Data Reshaping: New module for handling multiple survey form structures.
  • Introduced reshape_species_groups() for converting wide-format species data to long format
  • Created reshape_catch_data() supporting both version 1 and version 2 survey structures
  • Added preprocess_catch() with automatic survey version detection
  • Implemented preprocess_general_adnap() for ADNAP-specific trip information processing
  • Enhanced handling of nested length groups and fish over 100cm
• Improved GitHub Actions Workflow: Enhanced automation with clearer job naming and parallel execution.
  • Renamed workflow jobs for better visibility (e.g., “Ingest Lurio landings”, “Validate ADNAP landings”)
  • Parallel execution of Lurio and ADNAP pipelines for faster processing
  • Separate PDS data ingestion and preprocessing jobs
  • Clear dependency chains between ingestion, preprocessing, and validation stages

Improvements

• Documentation Overhaul:
  • Corrected all storage backend references from MongoDB to Google Cloud Storage
  • Updated function documentation to accurately reflect Parquet file usage
  • Added specific titles to distinguish Lurio and ADNAP functions
  • Removed misleading unused parameters from ingestion functions
  • Added explicit invisible(NULL) returns to all workflow functions for consistency
  • Enhanced documentation for KoBoToolbox validation integration
  • Improved parameter documentation honesty (noting hardcoded values)
• Code Quality Enhancements:
  • Cleaned up function signatures by removing unused parameters
  • Made return values explicit across all workflow functions
  • Improved consistency between function documentation and implementation
  • Enhanced examples to reflect actual usage patterns
• Survey Processing Pipeline:
  • Added support for multiple species field normalization
  • Improved handling of separate length group structures for large fish
  • Enhanced catch data validation with species-specific thresholds
  • Better integration with Airtable form assets for data mapping

Bug Fixes

• Fixed incorrect package reference in documentation (removed non-existent KoboconnectR package)
• Corrected validation threshold documentation (200 individuals, not 100) for ADNAP surveys
• Fixed duplicate GitHub Actions job names that made debugging difficult
• Corrected storage backend documentation throughout codebase (MongoDB → GCS)
• Updated validation flag numbering documentation for consistency across surveys

Infrastructure & Dependencies

• Maintained compatibility with parallel processing packages (future, furrr)
• Enhanced configuration system to support multiple survey sources
• Improved separation of concerns between Lurio and ADNAP processing pipelines
• Updated documentation generation with roxygen2
• Package maintains clean R CMD check status

peskas.mozambique.data.pipeline 1.0.0

Major Changes

• GPS Tracking Integration with Pelagic Data Systems (PDS): Full support for vessel tracking data ingestion and preprocessing.
  • Added ingest_pds_trips() to retrieve and store GPS trip metadata from PDS API
  • Added ingest_pds_tracks() to download individual trip point data with parallel processing support
  • Implemented get_trips() and get_trip_points() functions for PDS API interaction
  • Added preprocess_pds_tracks() for track data cleaning and feature extraction
  • Introduced preprocess_track_data() with configurable track processing pipelines
  • Created generate_track_summaries() for trip-level metrics calculation
• Airtable Integration Module: Complete suite of functions for two-way synchronization with Airtable.
  • Implemented airtable_to_df() with automatic pagination for full table retrieval
  • Added device_sync() for intelligent device data synchronization (updates existing, creates new)
  • Created sync_device_users() to manage vessel user credentials across Airtable and MongoDB
  • Developed bulk_update_airtable() and df_to_airtable() for batch operations
  • Added get_writable_fields() to identify editable fields and prevent computed field errors
• Comprehensive Data Validation Framework: Implemented multi-stage validation adapted from Peskas Zanzibar pipeline.
  • Redesigned validate_landings() with 10 validation flags across two stages
  • Stage 1: Basic data quality checks (form completeness, catch info, length validation, bucket/individual counts)
  • Stage 2: Composite economic indicators (price per kg, CPUE, RPUE) following Zanzibar thresholds
  • Created modular validation functions: validate_catch_taxa(), validate_price(), validate_total_catch()
  • Validation results exclude flagged submissions from final dataset while preserving flags for monitoring
• Taxa Modeling and Species Intelligence: New module for automated species identification and biological data enrichment.
  • Introduced match_species_from_taxa() using fuzzy matching against FishBase and SeaLifeBase
  • Implemented get_fao_groups() for commercial species categorization
  • Added get_species_areas_batch() for biogeographic validation
  • Created get_length_weight_batch() for length-weight relationship parameters
  • Developed getLWCoeffs() to retrieve stored coefficients from local database
  • Added process_species_list() for batch processing with automatic fallback logic

Improvements

• Storage System Enhancements:
  • Migrated primary storage to Google Cloud Storage with Parquet format
  • Added upload_parquet_to_cloud() and download_parquet_from_cloud() with versioning support
  • Implemented cloud_storage_authenticate() with temporary credential file handling
  • Created cloud_object_name() for version-aware object retrieval
  • MongoDB maintained as secondary storage for legacy compatibility
  • Moved length-weight coefficients from data/ to inst/ directory for better package structure
• Configuration Management:
  • Switched to dotenv package for environment variable management
  • Added load_dotenv() function with configurable .env file paths
  • Updated read_config() to automatically load environment variables
  • Expanded configuration schema to support PDS, Airtable, and multi-cloud storage
  • Added support for separate storage buckets for different data types (surveys vs. tracks)
• Data Preprocessing Pipeline:
  • Enhanced preprocess_landings() with metadata table joins (landing sites, boats, enumerators)
  • Implemented process_species_group() for handling species group disaggregation
  • Added species validation and enrichment with FishBase/SeaLifeBase data
  • Integrated length-weight conversion using local coefficient database
  • Added habitat information from species area data
  • Improved catch weight calculation with multiple estimation methods
• Export Functionality:
  • Expanded export_landings() to generate multiple analytical outputs
  • Added calculate_fishery_metrics() for aggregated statistics
  • Created MongoDB portal collections for dashboard integration
  • Implemented trip-level summarization for GPS track data
  • Enhanced data transformation for consumption by visualization tools
• Workflow Automation:
  • Added GitHub Actions workflow for automated releases (release.yaml)
  • Updated data pipeline workflow with improved error handling and notifications
  • Integrated cloud authentication in CI/CD pipeline
  • Added support for scheduled and manual workflow triggers

Bug Fixes

• Fixed price validation logic that was incorrectly flagging valid entries (#PR/issue reference if applicable)
• Corrected global variable bindings in validation functions to prevent R CMD check warnings
• Removed invalid geo parameter from mdb_collection_push() function call
• Fixed customer_name and submission_id variable scoping issues using .data$ notation

Infrastructure & Dependencies

• Added new package dependencies: furrr, future, glue, readr for enhanced functionality
• Updated .Rbuildignore to exclude development files (.env, .claude, CLAUDE.md)
• Package now passes R CMD check with no warnings or notes
• Improved documentation coverage with 34 new exported functions
• Enhanced type safety and code consistency throughout codebase

peskas.mozambique.data.pipeline 0.2.0

New features

• Updates data ingestion and preprocessing workflows
• Renames ingest_surveys to ingest_landings
• Adds new metadata joins and data transformations in preprocess_landings
• Introduces calculate_catch function for catch weight estimation
• Updates configuration to include Google Cloud Storage and additional metadata tables

peskas.mozambique.data.pipeline 0.1.0

• Initial CRAN submission.