Documentation

Add table name prefix to all managed tables

table_prefix:
 value: development # allows to set default, preprocessor will fail if the value has not been provided
 override_with_arg: table_prefix # allows to override argument name used to set table prefix
 separator: _ # will insert that separator if prefix is not empty, defaults to empty
 scope: managed # optional, all | managed (default) | unmanaged

Example usage: > table_prefix: > override_with_arg: developer > separator: _

Add a dataset prefix to all the tables in the spec, which doesn't already have it.

bigquery_defaults:
 dataset: dataset_name # Name of the dataset

Move all managed tables into schema (postgres) or dataset (BigQuery)

table_namespace:
 value: development # allows to set default, preprocessor will fail if the value has not been provided
 override_with_arg: table_namespace # allows to override argument name used to set table namespace
 scope: managed # optional, all | managed (default) | unmanaged
 on_existing: overwrite # optional, overwrite (default) or keep_original

Rewrite logical schemas (postgres) or datasets (BigQuery) into "physical" to facilitate multiple environments

logical_dataset:
 default_dataset: someproject.somedataset # default, it the table was referenced without dataset
 datasets:
   input: someotherproject.raw_analytics_data
   derived: production_analytics
   billing: someotherproject.billing
 allow_unknown: false # will fail when a table from unknown dataset is referenced to prevent typos (default)

Both datasets and default_dataset can be overridden with meta arguments.

Validate spec against references to unknown datasets

validate_datasets:
 allowed_datasets:
   - input
   - derived
   - billing

Define tables automatically in a SQL dir.

autodiscover