Skip to contents

Peskas Coasts data pipeline

pkgdown

The code that turns boat GPS tracks into fishing maps, prepares the data for the Peskas Coasts and Peskas Tracks apps, and supplies the shared tools every Peskas country pipeline uses.

See the results: Peskas Coasts (regional comparison across countries) and Peskas Tracks (app for fishers). Read more: How the pipeline works and Understanding the models.

What it is

This repository does three jobs for Peskas, WorldFish’s system for monitoring small-scale fisheries. It prepares the data behind Peskas Coasts, which compares fishing in Kenya, Zanzibar, Mozambique and Timor-Leste. It keeps Peskas Tracks up to date, the app where fishers see their trips and log their catches. It is also the shared toolkit that the Kenya, Zanzibar, Mozambique and Timor-Leste pipelines use to download GPS tracks, read surveys, store files and publish dashboard data.

What it produces

  • Maps where boats fish and for how long, by grouping GPS tracks into a standard grid of small hexagons of sea.
  • Estimates catch per hour of fishing in each hexagon for the most-caught species, by linking GPS tracks to landing surveys.
  • Draws the outlines of the main fishing grounds.
  • Combines monthly catch, revenue and price indicators from the four countries, by region and district, with money shown in US dollars.
  • Keeps Peskas Tracks running: fisher accounts linked to each boat’s tracker, trip and catch statistics for each fisher, and a weekly report on how the app is used.
  • Backs up the GPS tracks every day.

Where the data comes from

  • GPS tracks. GPS trackers (Pelagic Data Systems) are small solar-powered devices on boats that record where they travel. The pipeline downloads new trips and their tracks. A statistical model from WorldFish’s SSF AI toolkit then marks which parts of each track were fishing and which were travelling.
  • Landing surveys. A landing is a boat’s return to shore with its catch, recorded by an enumerator (a trained data collector). This pipeline does not read the survey forms itself. It takes the trips that the Kenya, Zanzibar and Mozambique pipelines have already checked and matched to GPS trips, and the monthly summaries that all four country pipelines publish.
  • Reference data. Boat and tracker records kept in Airtable, the boundaries of regions and districts (FAO’s standard administrative boundaries), and species data from FishBase and SeaLifeBase.
  • Peskas Tracks. The catches fishers log in the app.

The main pipeline runs every day. Fisher accounts in Peskas Tracks are updated every three days and the app usage report is made every Monday.

Catch per unit of effort (CPUE) here means kilograms of catch per hour of fishing, calculated for each hexagon from all the matched trips that fished there.

Known limits:

  • Catch per hour is only calculated for trips that have both a GPS track and a landing survey, for the five most-caught species, and only in hexagons fished by at least three such trips.
  • Money is converted to US dollars with fixed exchange rates written in the code, not current market rates.

Who runs it

WorldFish builds and runs this pipeline. The survey data comes from each country’s partners; see the country pipelines below. For questions, write to .

Part of Peskas

Peskas is WorldFish’s open-source platform for monitoring small-scale fisheries (https://peskas.org).

For developers

The code is an R package called coasts. Function reference and articles: https://worldfishcenter.github.io/peskas.coasts/.

Requirements

  • R 4.5 with the GDAL, GEOS and PROJ spatial libraries. Production uses the rocker/geospatial:4.5 image.
  • Python 3 with ssfaitk from the SSF AI toolkit, which predict_pds_tracks() calls through reticulate. Dockerfile.prod shows the exact setup.

Setup 

devtools::install_deps()
devtools::load_all()

Credentials come from environment variables. Create a .env file in the repository root (it is git-ignored; there is no example file). read_config() loads it and reads inst/conf.yml. The variables are:

  • Pelagic Data Systems: PDS_TOKEN, PDS_SECRET, PDS_USERNAME, PDS_PASSWORD, PDS_CUSTOMER_ID
  • Google Cloud: GCP_SA_KEY (the service account JSON on a single line)
  • MongoDB: MONGODB_CONNECTION_STRING_COASTS, MONGODB_CONNECTION_STRING_TRACKS, MONGODB_CONNECTION_STRING_VALIDATION
  • Metadata: AIRTABLE_TOKEN, AIRTABLE_BASE_ID_FRAME, AIRTABLE_BASE_ID_TRACKS_APP

The default profile uses the development buckets and databases. The production profile is switched on only by CI on main.

Main commands 

devtools::document()  # after editing roxygen comments; man/ is committed
devtools::test()
devtools::check()

Each pipeline step is one exported function, for example coasts::preprocess_pds_tracks(). The order of the steps is in .github/workflows/data-pipeline.yaml.

How it runs in production

GitHub Actions runs every job inside a Docker image built from Dockerfile.prod. Runs on main use the production profile; runs on any other branch use default, which writes to the development buckets and databases.

Workflow When it runs What it does
data-pipeline.yaml Daily at 00:00 UTC, and on every push The full pipeline
tracks-backup.yaml Daily at 00:00 UTC, and on push to main Backs up the GPS tracks
sync-devices-users.yaml 03:00 UTC on the 1st, 4th, 7th, … of each month Copies tracker records to Airtable and updates Peskas Tracks accounts
app-usage-report.yaml Mondays at 00:00 UTC, and on push to main Builds the Peskas Tracks usage report
collection-stats.yaml Mondays at 06:00 UTC, main only Writes the weekly collection figures per district to Airtable

The country pipelines install the latest GitHub release of coasts when they build their images, so a change reaches them only after it is released.

Releases

Bump Version: in DESCRIPTION and add a block at the top of NEWS.md headed # coasts X.Y.Z, written for non-technical readers. On push to main, release.yaml turns that block into a GitHub release if the version is new.

Tests

devtools::test() runs the tests in tests/testthat. No workflow runs them automatically yet.

Contributing

Read .github/CONTRIBUTING.md. Code follows the tidyverse style guide; comment /style on a pull request to apply it. New to R packages? See R Packages by Hadley Wickham and Jenny Bryan. For AI-assisted work, see CLAUDE.md.

Licence

GPL-3 or later. See LICENSE.md.