Changelog • peskas.timor.data.pipeline

peskas.timor.data.pipeline 5.2.0

Records say which organization collected them

NEW

survey_organization names the organization behind each record, as the first column — "MAF" here, one programme across all three form versions. survey_id identifies the form, not the organization, and a country can run more than one programme at once, as Kenya does.

peskas.timor.data.pipeline 5.1.2

A recorded crew of zero is a crew that went unrecorded

The cross-country API schema declares n_fishers with a minimum of 1, and the validated export published 343 rows across 273 submissions carrying zero. It is not a crew of zero: 209 of those 273 landed catch and 211 recorded revenue, so the field was simply never filled in. sum_fishers() reports a crew whenever any component is recorded, which is what let the 186 submissions with no_men_fishers blank but women and children entered as 0 through.

FIXED validate_n_fishers() raises alert 18 when the three fisher_number_* columns sum to zero and blanks them, the same way it already handles an outlying crew. n_fishers then derives as NA. Measured on the published trips-validated__20260913022202 export: n_fishers == 0 goes from 343 rows / 273 trips to none, NA from 6,166 to 6,509 rows, and the published minimum from 0 to 1. Row and trip counts are unchanged at 145,022 and 97,828, and sum(catch_kg) is unchanged at 889,070.0 kg — this is a correction to an unrecorded field, not a data loss.
CHANGED 273 submissions gain alert 18 in their published flag string, so the enumerator sees them in the review queue and can enter the real crew. The alert vocabulary is untouched: 18 is already “Anomalous number of fishers” in inst/config.yml.
FIXED Per-fisher metrics no longer divide by zero. On the same export the 247 Inf rows in catch_kg / trip_duration_hrs / n_fishers go to zero, and mean() over it goes from Inf to 1.27427; the median moves 0.6774 -> 0.6755. coasts::summarize_data() and R/format-public-data.R both already discarded these defensively, so the portal is unchanged — the fix is for every other consumer of the API.

The landing-date bound no longer depends on the submission date

CHANGED validate_surveys_time() raises alert 4 on a landing dated after the run, not only on one dated after its own submission. The existing test implies the new one wherever submission_date is present, so this covers only a missing or itself-future submission date — 0 rows change in the current export, where every submission has a submission date and the one future-dated landing is already flagged.
A future landing date is still flagged and kept, not blanked or dropped. landing_date is the merge key and every time aggregation reads it, and the single offending submission is recoverable rather than lost: 818065108 was submitted 2026-09-07 and claims a landing on 2026-09-21, its neighbouring submission ids land between 2026-08-20 and 2026-09-07, and the latest legitimate landing anywhere in the export is 2026-09-11. Blanking it would destroy a correctable day typo whose true value the submission date already bounds.

peskas.timor.data.pipeline 5.1.1

The total-length restatement moves to coasts

get_morphometric_tables() derived its own ratio table from POPLL and rescaled a inline. Zanzibar and Mozambique each had the same arithmetic written out again, with different semantics and nothing comparing them. It now lives in one place.

CHANGED The inline ratios block is replaced by coasts::convert_lw_to_tl(), which does exactly what Timor did — restate every convertible pair, pass the rest through carrying their original Type. Verified end to end against the old code on the full 11,349-row table: identical() is TRUE, 615 taxon codes, zero difference in lw_a or lw_b.
CHANGED coasts (>= 4.13.0) is now a declared floor in DESCRIPTION, so an old hub fails at install rather than part-way through a run.

peskas.timor.data.pipeline 5.1.0

Timor appears on the multi-country coasts portal

Timor’s boundary polygons were already on the coasts portal and already pushed to its MongoDB on every run, but nothing published the metrics behind them, so the country rendered as empty shapes. Timor now runs the same coasts chain as Kenya, Mozambique and Zanzibar, in the same jobs and the same order, rather than a Timor-shaped variant of it.

NEW summarize-model-data (coasts::summarize_data() then coasts::generate_fleet_analysis()) and export-surveys-portal (coasts::export_portal()), mirroring the other three pipelines. export_portal() uploads timor_monthly_summaries_map to the coasts bucket, which is what puts Timor on the portal, and fills the dashboard database.
NEW coasts::preprocess_pds_tracks() in preprocess-pds-data, the step Timor previously skipped. It is incremental, but the first run has no pds-tracks-preprocessed to diff against and so grids every track in the bucket — 101,985 objects in gs://pds-timor measured 2026-09-10. Expect one long run, then normal ones.
NEW config: surveys$summaries, surveys$aggregated, metadata$map_boundaries and storage$mongodb$databases$dashboard — the keys the other three already carry, in the same positions. pds$customers is deliberately not among them: coasts::generate_fleet_analysis() reads it as a device roster while coasts::select_country_trips() reads it as trip selection and refuses to run alongside exclude_customers, which is how Timor selects trips. One key, two meanings — see the note in inst/config.yml.
NEW inst/tinytest/test_coasts_metrics.R, 9 assertions on the published parquet: the seven-column schema export_geos() reads, country as the join key, metrics present and finite, and every named administrative post resolving to a municipality. Runs as a step in export-surveys-portal.
CHANGED Requires peskas.coasts >= 4.12.3, which fixes two things Timor’s data trips over: one undated landing aborting every summary, and zero-fisher trips putting Inf in the published metrics. Both are in summarize_data(), and without them this pipeline cannot complete.

The Timor portal is unaffected. It is still served from versioned portal-*.json in the public bucket by export-trips, which is untouched; storage$mongodb$databases$dashboard is a separate sink that nothing in peskas.timor.portal.v2 reads.

peskas.timor.data.pipeline 5.0.0

This release completes the move to a single source of reference data. Landing site names, municipalities, gears, vessels and species codes now all come from the shared Peskas reference base, the same one Kenya, Mozambique and Zanzibar use, so a site or species means the same thing in every country’s data. The spreadsheets that used to hold some of this are gone.

What changes in the published data

Landing site names now use the shared spellings. Eleven of the forty sites are affected. These names appear in the cross-country dataset, not on the Timor portal, so the portal’s own figures are unaffected.
One landing site was filed under the wrong municipality. Welaluhu was recorded as being in Manatuto and is in fact in Manufahi. It is a single landing out of roughly 95,000, but because municipal figures are averages, correcting it shifts the reported catch and average landing weight for both municipalities. It does not move any coast-level or national total, since both municipalities are on the south coast.
North and south coast are now decided by the landing site, not the municipality. Two municipalities have landing sites on both coasts, so deciding by municipality misfiled some of them. National and coast totals come out identical to before this change, but the rule now has one definition instead of two that had drifted apart.
GPS tracking data is more complete. Around 2,800 fishing trips had been dropped because their tracker had since been reassigned to a different project; they are back. In the other direction, around 1,400 journeys that were not fishing trips at all — bicycle trips from a separate transport project sharing the same tracking account — are now excluded. Both affect reported fishing effort, and municipalities differ in how much.

For anyone running the pipeline

This is a major version because installed function names and stored table layouts changed.

No Google Sheets access is needed. GOOGLE_SHEET_ID is no longer used, and the pipeline step that read the spreadsheets is gone. The one thing they still held — the roster of tracker serial numbers, which is needed because fieldworkers can only note down part of each number — is now kept in cloud storage and maintained by the pipeline itself, so it needs no manual updates.
Functions removed: ingest_metadata_tables(), preprocess_metadata_tables(), get_preprocessed_sheets(), timor_assets(), ingest_assets(). Reference data is read with get_assets(); the shared reference snapshot is now written by the hub package rather than by this pipeline, as it already was for the other countries.
The stored survey table is narrower, 38 columns instead of 98. The removed columns were untranslated copies of columns that already existed, survey-software bookkeeping nothing read, and a field carrying the fieldworker’s phone identifier, which had no reason to be stored.
check_portal_contract() is new and exported. It compares a set of published portal files against a reference and reports any change in structure.

Reliability fixes

Published files are now checked on every run. The website finds its data files by searching for them by name, so a renamed file or a dropped field would have quietly removed a section of the site with nothing failing. Each run now compares what it just published against a reference structure and stops if they differ.
Four checks could not fail. The pipeline’s data checks reported success regardless of what they found, because of how their results were read. They now stop the run. One check had in fact been failing unnoticed — and was itself wrong: it objected to future-dated catches, which the pipeline deliberately keeps and flags for review rather than discarding. It now verifies that no future date goes unflagged.
Two errors that pointed at the wrong cause. A failed download used to surface as an unrelated file-format error, and a missing reference-data setting used to be silently substituted with the wrong region. Both now say what actually went wrong.

peskas.timor.data.pipeline 4.0.0

Alignment of the Timor pipeline to the harmonized Peskas standard shared with the Mozambique, Kenya and Zanzibar pipelines. Timor was the first Peskas pipeline and predates the conventions the others share, so most of this release is structural. The portal contract — the seven portal-*.json objects, their keys, nesting, categories and column types — is deliberately unchanged. The figures inside it move, and that is the first section below.

Published figures change

This release is a bias correction, not a re-skin. The first production run on it republishes the portal with, against the live production set:

catch −18.4%, landing weight −15.3%, estimated tonnage −18.4%;
price per kg +17.8%, revenue −4.5%;
nutrient supply −26.0%, nutrient RDI −20.3%.
North/South Coast revenue redistributed, Lautem now counted North.

Object names, keys, categories, nesting and column types are unchanged — the contract gate reports 0 structural failures — so nothing on the site breaks or empties.

Three corrections to the length-weight path account for it, and they do not all push the same way:

A central estimate per taxon. summarise_lw_coeffs() takes a geometric mean of FishBase’s a and an arithmetic mean of b over every study not flagged questionable, where the previous code selected a per-taxon percentile.
The right species behind each taxon. Coefficients are now restricted to FAO areas 57 and 71, TUN is the tribe Thunnini rather than a common-name search, and CLP searches Dorosomatidae as well as Clupeidae — FishBase moved the tropical sardines there in 2022, so Timor’s second-largest taxon had been priced off Atlantic herring. Taken alone this raised catch 12.6%.
One measurement basis. A published W = a * L^b is fitted on whichever axis the study used, and more than half of the matched pairs are not total length — 990 fork-length and 460 standard-length against 1,824 TL. Pooling them as published and applying the result to a TL measurement overestimates weight, because FL and SL are shorter than TL. The length-length conversions were already being fetched and discarded; 1,363 of 1,648 non-TL pairs are now restated on a TL basis before pooling. Taken alone this lowered catch 10.9%.

The largest per-taxon moves against production are FLY +8%, CLP +3%, CJX −8%, SNA −15%, MZZ −18%, SDX −20%, CGX −21%, TUN −25%, MOO and BEN −37%, LWX −39% and GZP −45%. GZP’s fall is a correction: its coefficients came from a common-name lookup whose pool was topped by driftfishes and scads at 185–328 g while the actual garfish sit at 17–21 g.

Measured 2026-09-06 against the live production set, on FishBase release 25.04 (see the next section).

The FishBase release is pinned

peskas.coasts read FishBase over the network with no pinned release, so the reference data moved whenever a container was rebuilt. rfishbase 5.0.3 changed the data host from HuggingFace (latest release 25.04) to Source Cooperative (latest 26.06), and in 26.06 the families Caesionidae and Scaridae survive with zero species attached — their genera having moved to Lutjanidae and Labridae. CJX and PWT therefore resolved to no coefficients and weighed NA, which sums to zero. CJX is 5% of landed weight and one of the 13 modelled taxa, and it went missing from portal-taxa_aggregated entirely on two runs, with no error.

metadata.fishbase.db_version: "25.04" in inst/config.yml, read by coasts::resolve_db_version() and resolved once per call so a run cannot mix snapshots. This requires coasts >= 4.10.0, which is now a hard floor. conf is passed to both get_taxa_morphometrics() call sites — without it coasts resolves "latest" from its own configuration and the pin does nothing.
assert_taxa_coverage() fails the run when any taxon but the two documented exemptions resolves to no coefficient pair, so a vanished taxon cannot reach the portal silently. This is what caught the above.

Every figure in the section above was measured on 25.04. Moving the key re-baselines the portal and should be done deliberately, with the portal contract check run against the change.

Breaking changes

The taxa and weight path no longer reads Google Sheets, and is filtered to Timor’s FAO areas. Taxon codes still come from the PESKAS | FRAME snapshot; their scientific names now come from the FAO ASFIS list in the country bucket, joined on Alpha3_Code, as Mozambique does — measured behaviour-neutral, 55 of 56 names identical. Length-weight coefficients are restricted to FAO areas 57 and 71 via the new metadata.fishbase.fao_areas key, which must be set: coasts falls back to the Indian Ocean pair c(51, 57) otherwise. get_morphometric_tables() loses its manual_table argument.
Storage is delegated to the shared peskas.coasts hub. R/cloud-storage.R and R/google-drive.R are deleted; every call site is coasts::{cloud_storage_authenticate, upload_cloud_file, download_cloud_file, cloud_object_name, cloud_object_names, upload_parquet_to_cloud, download_parquet_from_cloud}. coasts ≥ 4.6.0 is a hard floor.
Secrets move from an auth/ directory to .env. The local: config environment is gone, so a local run and CI resolve the same config branch and differ only by R_CONFIG_ACTIVE. .env.example documents every variable.
inst/conf.yml → inst/config.yml, derived from the cross-country configuration template: country, ingestion, surveys.landings.{v1,v2,v3}, api, pds, metadata, storage.google.options_{coasts,api}, storage.mongodb, validation. Through the migration the file carried every legacy key beside the harmonized one that replaced it, marked # [legacy]; that half is now deleted, each key after its last reader.
Interchange format is flat long parquet, one row per (submission, catch, length bin), from raw through validated. No .rds artefact is left on the survey path; get_validated_landings() re-nests the parquet on read for the portal path, which is why that path did not change.
KoBo retrieval, PDS ingestion and the Airtable frame are delegated. ingest-pds-data.R, retrieve-pds-data.R, retrieve-survey-data.R, preprocess_pds_trips() and Timor’s own air_* client are deleted; the pipeline calls coasts::get_kobo_data(), coasts::ingest_pds_trips(), coasts::ingest_pds_tracks() and coasts::ingest_assets().
Validation flags go to MongoDB, one surveys_flags-<asset_id> collection per live form in the shared validation-{dev,prod} database, replacing the Google Sheets sink. This is what puts Timor into the cross-country validation UI.
The v1 form is frozen (last submission 2020-08-28). It is neither ingested nor preprocessed; merge_landings() reads a snapshot produced once per environment by inst/freeze-landings-v1.R, which also converted v1’s fork lengths to total length.
Reference data is now the shared PESKAS | FRAME Airtable base wherever it overlaps the Google Sheets metadata tables — taxa, gears, vessels, landing sites, districts and PDS devices. The Sheets shrink accordingly.
The resolved configuration is named conf throughout, not pars.

New features

Timor publishes the cross-country API parquet. export_api_raw() and export_api_validated() write the 22-column trips table Kenya, Mozambique and Zanzibar already publish, to peskas-api-{dev,prod}/timor/{raw,validated}. The export is gated exactly as those three gate theirs — the shared “Set env to production” step and nothing else — so this release is Timor’s first write to peskas-api-prod, and the API gains its fourth country.
data-raw/compare-portal-json.R — a gate over the seven portal-*.json objects: object names, then keys, nesting, column sets and column types, then per-column numeric summaries. Run it before touching the export path.

Improvements

read_config() no longer logs the resolved configuration. It was printing the service-account private key and every token into each CI job log; it now logs key names only, and every coasts:: workflow call passes log_threshold = logger::INFO for the same reason.
DESCRIPTION declares ggchicklet in Remotes:. It is a GitHub-only package used by inst/report/data_report.Rmd and installed in both Dockerfiles, but it was declared nowhere pak could see it, so any dependency resolution outside the container failed to solve.
Reference sections in _pkgdown.yml are keyword-driven rather than name-pattern driven, so a rename can no longer silently move a function between sections.
CI is on current templates: R-CMD-check.yaml (replacing check-standard.yaml), pkgdown.yaml, test-coverage.yaml and pr-commands.yaml from the r-lib v2 examples, checkout@v5, build-push-action@v6, ubuntu-latest, and a release.yaml that cuts a GitHub release from this file. Four workflows that could only ever build through the retired docker.pkg.github.com registry were retired or rebuilt.
The export emits the seven portal-*.json objects the portal actually consumes. portal-indicators_grid and portal-label_groups_list, which the portal excludes and which were rebuilt on every run from a 2024-07-27 artefact, are no longer written.

Bug fixes

validate_catch_params() assigned length_individuals positionally into a separately-derived frame, and an isTRUE() on a vector made alerts 12–15 unreachable. Both are alert-identical to the intended behaviour on current data.
The coast rule put Lautem in the South. export_files() classified Lautem as a South Coast municipality; it is on the north coast. North and South Coast revenue in portal-summary_data change accordingly.
registered_boats came from a stale Google Sheet. It now reads geo.total_boats from the PESKAS | FRAME Airtable base, the same source the rest of the pipeline uses, which moves Manatuto from 283 boats to 213 and Viqueque from 213 to 207. The other ten municipalities are unchanged.
A duplicate get_preprocessed_metadata() shadowed the correct definition.
The local: config environment inherited a non-existent development environment.

Removals

None of the following changed a published number; each was verified to have no caller or no reader before it went.

25 unreferenced functions, ~2,900 lines. The second, uncalled glmmTMB estimator model_indicators() and its 674-line subgraph; the four PDS map products (ingest_pds_map(), ingest_kepler_tracks(), kepler_mapper(), ingest_complete_tracks()) with inst/kepler_mapper.py; four cloud accessors; and twelve scattered helpers.
Six Google Sheets metadata tables. Five whose last reader had gone — vms_installs, centro_pescas, boats, fishing_vessel_statistics and registered_boats — plus morphometric_table, whose 559 curated length-weight rows are now package data (inst/extdata/morphometric-coefficients.csv). That was the weight path’s last Google Sheet. Six tables remain, each annotated in inst/config.yml with what blocks moving it.
rescue_by_common_name(), which reached three taxa by rfishbase::common_to_sci() on the literal strings "Tuna", "Shark" and "Garfish". Being a substring match on common names, SKH’s pool contained a Mekong catfish and an aquarium bala shark. Replaced in the same change by an explicit alias table — it was the only source of coefficients for TUN, 56% of landed weight, and removing it alone would have halved published catch.
The KoBoToolbox validation-status client, upstreamed to coasts 4.7.0 and now called from there. A package’s own definitions win over its imports, so the delegation was only real once the local copy was gone.
glmmTMB, httr2 and reticulate leave Imports and ggplot2 moves to Suggests. The glmmTMB GitHub install leaves both Dockerfiles with it — a TMB compile on every image build, for a package nothing imports.

peskas.timor.data.pipeline 3.3.0

Breaking changes

Now kobo surveys are retrieved according the new v2 API. The function retrieve_survey_data() has been replaced by get_kobo_data updated to retrieve data from the new API. The function retrieve_survey_metadata() has been deprecated.

peskas.timor.data.pipeline 3.2.0

Improvements

Align variables to the Aquatic Food Ontology (AQFO)

peskas.timor.data.pipeline 3.1.0

Improvements

Estimations are now weighted based on the number of records in each monthly bin. This improve the accuracy of estimations and reduce outliers effect.

peskas.timor.data.pipeline 3.0.0

New features

Integrated the ingestion of the new survey form “Peskas 2”. The new form was deployed with the aim to be more manageable for enumerators. It also includes some more questions regarding the fish handling and trading.

peskas.timor.data.pipeline 2.0.0

Breaking changes

Validation step in now implemented using google sheets using the googlesheets4 package.

peskas.timor.data.pipeline 1.4.0

Improvements

Improve national and municipal estimates combining packags Amelia and mice for missing and outliers data imputation.
Integrating price per kg into export data

Bug fixes

Improved revenue outliers identification based on empirical information

peskas.timor.data.pipeline 1.3.0

New features

Integrated taxa selection and filtering in tracks file deemed to the leaflet map in the portal

peskas.timor.data.pipeline 1.2.0

New features

Added a new feature in the function ingest_pds_map. The function process and upload a data frame containing the number of trips, CPE (catch per unit effort) and RPE (revenue per unit effort) splitted by grids to produce leaflet maps in the web portal.

peskas.timor.data.pipeline 1.1.0

Improvements

Improve the estimation of some fish groups’ catches by calculating the weight per individual case by case.

Bug fixes

-Fix error in cleaning of legacy landings: the columns indicating the number of individuals and fish length for catches > 60cm were exchanged

peskas.timor.data.pipeline 1.0.0

Improvements

Adding option to produce Timor map filtered by fishing trips

peskas.timor.data.pipeline 0.20.0

Improvements

Improve the validation step by flagging those observation characterized by having positive revenue (individuals) despite 0 or null individuals (revenue).
Use a log model to identify abnormal weight-revenue relationships on Cook’s distance estimation.
Replace NA catch code with catch code “0” and ensure these observations having non positive individuals and revenue.
Use total length for weight calculation of MOO in all the landings (weights calculated with FL seem quite unrealistic)
Use 95° quantile instead of median to summarise weight parameters for catch types, it seems to return more realistic weight estimations by single individuals.

Bug fixes

Split landings preprocessing in two jobs to avoid to run out of memory (docker exit code 137)

peskas.timor.data.pipeline 0.19.0

New features

Added a new function ingest_pds_matched_trips to ingest matched pds tracks and survey landings in a zip folder on a monthly scale.

peskas.timor.data.pipeline 0.18.0

New features

Added nutritional values for each catch by the function get_nutrients_table. The function links to the repository https://github.com/mamacneil/NutrientFishbase and join the estimated nutrients values with the FishBase species data.

peskas.timor.data.pipeline 0.17.0

New features

Added the folder report containing the scripts useful to generate a pdf report of Peskas. Such report is downloadable from Peskas portal.
Added a series of functions (get_sync_tracks(), get_full_tracks() ,get_full_trips()) useful to retrieve the complete file of pds tracks.
Generate and retrieve the pds track map with ingest_pds_map() and get_tracks_map().

peskas.timor.data.pipeline 0.16.0

Breaking changes

The Docker file now run rocker/r-ver 4.1.1 instead of rocker/geospatial:4.0.3

Bug fixes

Fixed the bug when calling ingest_rfish_table() in the main pipeline.

peskas.timor.data.pipeline 0.15.0

New features

Added a series of functions (get_catch_types(), get_fish_length() ,retrieve_lengths()) useful to retrieve morphometric conversion factors from catch types names in metadata tables.
Added a new metadata table in airtable (morphometric_table) containing length-weight and length-length conversion factors.
Added join_weights() which integrates morphometric data with merged landings.

peskas.timor.data.pipeline 0.14.0

New features

We test validated data to ensure its integrity
Added merge_trips() which integrates data from the landings and tracking together
Added format_public_data() to format and export data to be used for analytics

peskas.timor.data.pipeline 0.13.0

New features

Added functions to process and validate pds trips. Namely, preprocess_pds_trips() and validate_pds_trips(). These functions make sure the data types are appropriate, check for trip duplicates and perform basic checks for trip duration and distance.

peskas.timor.data.pipeline 0.11.0

New features

Added retrieve_pds_trips_data(), retrieve_pds_trips(), retrieve_pds_tracks_data() and retrieve_pds_tracks() to download trips and tracks from Pelagic Data System API.
Added function get_pds_res() to convert Pelagic Data System API responses to data frames and merge trips and tracks data in a unique file.
Added ingest_pds_trips() and ingest_pds_tracks() to upload Pelagic Data System data into cloud.
Created a new bucket (pds-timor-dev) to store pds data.

Bug fixes

cloud_object_name() returns an empty vector when the bucket is empty

peskas.timor.data.pipeline 0.10.1

Bug fixes

Fixed the bug in survey retrieve process. Integrating a server response status check in retrieve_survey_data()

peskas.timor.data.pipeline 0.10.0

New features

Added merge_landings() to merge and upload pre-processed recent and legacy landings data.

peskas.timor.data.pipeline 0.9.1

Improvements

Simplified jobs in the pipeline workflow

Bug fixes

Fixed workflow so that runs that are supposed to run in production actually do so

peskas.timor.data.pipeline 0.9.0

New features

Added clean_catches(), coalist() and clean_legacy_landings() to restructure legacy landings to recent landings.
Added preprocess_legacy_landings() to clean and ingest preprocessed legacy data.

peskas.timor.data.pipeline 0.8.0

New features

Added validate_landings() and ingest_validation_tables() to get validation data, check the ladings and upload flags to Airtable.
Added air_tibble_to_records() and air_upload_records() to create and update records in Airtable
Added logic to validate IMEIs in the landing data
Added a script in inst/airtable/edit-submission-link.js which can be used in Airtable to generate a link to the editable submission form

Improvements

Deactivated check of uniqueness of IMEI per boat in pt_

Breaking changes

Landings pre-processing now renames a few columns in the data: specifically submission_id (used to be *_id) and landing_date* (used to be date)

peskas.timor.data.pipeline 0.7.0

New features

Added ingest_legacy_landings to retrieve data from legacy data (SFF landings)

peskas.timor.data.pipeline 0.6.0

New features

Added air_get_records() and air_records_to_tibble() to retrieve and process records from Airtable
Added pt_validate_boats(), pt_validate_devices(), and pt_validate_vms_installs() to perform basic data validation from the metadata tables

Improvements

cloud_object_name() can now also match files by exact name and not just by prefix

Breaking changes

ingest_metadata_tables() and preprocess_metadata_tables() now use logic to use Airtable instead og Google Sheets

peskas.timor.data.pipeline 0.5.0

New features

Added ingest_metadata_tables() to ingest data about boats, species, municipalities, etc.
Added preprocess_metadata_tables() to preprocess the data from the metadata ingestion.
Added pt_get_devices_table() and pt_validate_flags() as helper functions for the metadata preprocessing.

Improvements

We use a single function to read the config.file and load the environment parameters
Added an skeleton pipeline to provide guidance to future development. This is now visible in the README.

peskas.timor.data.pipeline 0.4.0

Breaking changes

Renamed ingest_timor_landings() to ingest_landings() for brevity and because all functions relate to Timor anyways.

Improvements

Renamed the job names in the github workflow as the functions that the job calls.
Improved documentation of the package: particularly the readme and the function reference.

Bug fixes

We now skip RCurl from windows tests

peskas.timor.data.pipeline 0.3.0

New features

The preprocessing of East Timor landings is implemented in preprocess_landings()
Added pt_nest_attachments() to group all attachment columns into a nested column containing data frames.
Added pt_nest_species() to group all attachment columns into a nested column containing data frames.
Added cloud_object_name() as a complement to add_version() to return the latest or an specified version of an object in an storage location.
Added download_cloud_file() to download files from cloud storage providers.

Improvements

Now using cloud_storage_authenticate() to internally authenticate to cloud storage instead of authenticating separately in each cloud functionjj. This simplifies authentication and ensures authentication is not attempted when credentials have been already validated.

peskas.timor.data.pipeline 0.2.0

Breaking changes

download_survey_data(), download_survey_metadata(), and download_survey() have been renamed to retrieve_survey_data(), retrieve_survey_metadata(), and retrieve_survey(). This is to avoid confusion with planned functions that download data from cloud locations.
The suffix raw and metadata that is appended to the prefix when retrieving survey information is now separated using “_” rather than “-”. This is to more easily distinguish between information encoded in the file name.

New features

The prefix name of surveys is not hard-coded and can be specified in the config file (file_prefix field).

peskas.timor.data.pipeline 0.1.0

Adds infrastructure to download survey data and upload it to cloud storage providers and implements the ingestion of East Timor landings.

New features

The ingestion of East Timor Landings is implemented in ingest_timor_landings().
The functions download_survey_data() and download_survey_metadata() which download data and metadata for an electronic survey hosted by kobo, kobohr, or ona.
download_survey() can be used as a wrapper to download data and metadata in a single call.
upload_cloud_file() can be used to upload a set of files to a cloud storage bucket. Currently only Google Cloud Services (GCS) is supported.
add_version() is an utility function that appends date-time and sha information to a string and is used to version file names.
get_host_url() is an utility function that gets the host url of an electronic survey provider API.

Pipeline

The data pipeline is implemented and run in GitHub Actions on a schedule.