Preprocess metadata tables

Takes ingested metadata tables, validates them, and processes them into artefacts (devices information, flags) to be used further down in the pipeline.

Usage

preprocess_metadata_tables(log_threshold = logger::DEBUG)

Arguments

log_threshold

The (standard Apache logj4) log level used as a threshold for the logging infrastructure. See logger::log_levels for more details

Details

Specifically this function:

• Validates the flags table containing the columns: flag_id, flag_category, flag_message

• Constructs a devices table containing the columns: imei

This function requires no arguments because it retrieves the parameters from conf.yml. The fields required are:

metadata:
  spreadsheet:
    name:
  version:
    preprocess:
storage:
  storage_name:
    key:
    options:
      project:
      bucket:
      service_account_key: