Writing custom filters

You can extend the list of pre-built filters and add your own custom filter to DataCater.
At startup time, DataCater loads all filter definitions from the directory configured in datacater.filters.path.
You can extend the set of available filters by adding a new folder to this directory. DataCater expects two files in this folder: filter.py and spec.yml.
Please note that, at the moment, you need to build your own Docker images for datacater/datacater and datacater/python-runner after adding new filters.
filter.py
The file filter.py must provide a Python function filter that takes three parameters, value, row, and config:
value is the value of the attribute that the filter is applied to. The data type of value depends on the attribute.
row is a Python dict and provides access to all other attributes of the record. You can address these attributes by their name.
config is a Python dict and provides access to the configuration of the filter.
Please see the following code listing for the filter.py of the filter reg-exp:
def filter(value, row, config):
    import re
    return bool(re.match(config["value"], value))
spec.yml
The file spec.yml provides documentation for the filter. It provides the following options:
name is the name (or label) of the filter.
key is an internal, unique identifier of the filter.
description describes the filter.
license specifies the license of the filter.
website links to the website or repository of the filter.
author provides information about the author of the filter. It is an object and consists of the keys name and email.
labels can be used to attach information to the filter.
config provides information about the available configuration for the filter.
version defines the version of the filter.
Please see the following code listing for the spec.yml of the filter reg-exp:
---
name       : Matches regular expression
key        : reg-exp
description: Returns whether the attribute matches a provided regular expression.
license    : BSL
website    : https://github.com/DataCater/datacater/filters/reg-exp
author     :
  email: [email protected]
  name: DataCater GmbH
labels     :
  input-types: string
config     :
-   name: value
    label: Value
    type: text
version    : 1.0.0
Last modified 2mo ago