Writing custom filters
You can extend the list of pre-built filters and add your own custom filter to DataCater.
At startup time, DataCater loads all filter definitions from the directory configured in
datacater.filters.path
.You can extend the set of available filters by adding a new folder to this directory. DataCater expects two files in this folder:
filter.py
and spec.yml
.Please note that, at the moment, you need to build your own Docker images for datacater/datacater and datacater/python-runner after adding new filters.
The file
filter.py
must provide a Python function filter
that takes three parameters, value
, row
, and config
:value
is the value of the attribute that the filter is applied to. The data type ofvalue
depends on the attribute.row
is a Python dict and provides access to all other attributes of the record. You can address these attributes by their name.config
is a Python dict and provides access to the configuration of the filter.
Please see the following code listing for the
filter.py
of the filter reg-exp
:def filter(value, row, config):
import re
return bool(re.match(config["value"], value))
The file
spec.yml
provides documentation for the filter. It provides the following options:name
is the name (or label) of the filter.key
is an internal, unique identifier of the filter.description
describes the filter.license
specifies the license of the filter.website
links to the website or repository of the filter.author
provides information about the author of the filter. It is an object and consists of the keysname
andemail
.labels
can be used to attach information to the filter.config
provides information about the available configuration for the filter.version
defines the version of the filter.
Please see the following code listing for the
spec.yml
of the filter reg-exp
:---
name : Matches regular expression
key : reg-exp
description: Returns whether the attribute matches a provided regular expression.
license : BSL
website : https://github.com/DataCater/datacater/filters/reg-exp
author :
email: [email protected]
name: DataCater GmbH
labels :
input-types: string
config :
- name: value
label: Value
type: text
version : 1.0.0