Skip to contents

Preprocess Landings Data

preprocess_landings.Rd

This function imports, preprocesses, and cleans legacy landings data from a MongoDB collection. It performs various data cleaning and transformation operations, including column renaming, removal of unnecessary columns, generation of unique identifiers, and data type conversions. The processed data is then uploaded back to MongoDB.

Usage

preprocess_landings(log_threshold = logger::DEBUG)

Arguments

log_threshold

Logging threshold level (default: logger::DEBUG)

Value

This function does not return a value. Instead, it processes the data and uploads the result to a MongoDB collection in the pipeline database.

Details

The function performs the following main operations:

  1. Pulls raw data from the raw data MongoDB collection.

  2. Removes unnecessary columns and standardizes list columns.

  3. Renames and reorganizes columns for clarity and consistency.

  4. Generates unique 'survey_id' field.

  5. Merges and consolidates data from various form versions.

  6. Processes gear-specific effort data.

  7. Handles GPS coordinates.

  8. Converts data types and fixes nested column fields.

  9. Recode variables and drop trailing spaces to ensure consistency and clarity.

  10. Uploads the processed data to the preprocessed MongoDB collection.

Note

This function requires a configuration file to be present and readable by the 'read_config' function, which should provide MongoDB connection details.

Examples

if (FALSE) { # \dontrun{
preprocess_landings()
} # }