Integrating existing Apache Kafka topics
Self-managed installations of DataCater (Open-Core or Enterprise) allow you to integrate existing Apache Kafka topics. You can add a new stream and connect it with a Kafka broker and topic. Depending on whether you want to publish data to or consume data from the stream (or topic), you can set the according properties of the official Apache Kafka consumer and producer API:

In addition to the name of the topic and the connection properties, you must let DataCater know the format of the data in the topic. To this end, you can define deserializers and serializers for both the key and the value of the records. By default, DataCater expects data to be formatted in JSON.
DataCater provides serializers and deserializers for handling JSON data:
- If you want to consume JSON data from a stream, please choose the io.datacater.core.serde.JsonDeserializer for the
key.deserializer
and/orvalue.deserializer
- If you want to publish data to a stream in the JSON format, please choose the io.datacater.core.serde.JsonSerializer for the
key.serializer
and/orvalue.serializer
DataCater provides serializers and deserializers for handling JSON data:
- If you want to consume AVRO data from a stream, please choose the io.datacater.core.serde.AvroDeserializer for the
key.deserializer
and/orvalue.deserializer
- If you want to publish data to a stream in the AVRO format, please choose the io.datacater.core.serde.AvroSerializer for the
key.serializer
and/orvalue.serializer
Please note that you need to provide a schema when using AVRO. To this end, either point DataCater to a schema registry using the property
schema.registry.url
or provide an inline schema using key.deserializer.schema
, value.deserializer.schema
, key.serializer.schema
, or value.serializer.schema
.DataCater can work with the (de)serializers provided by Apache Kafka for handling raw strings:
- If you want to consume Strings from a stream, please choose the
org.apache.kafka.common.serialization.StringDeserializer
for thekey.deserializer
and/orvalue.deserializer
- If you want to publish data to a stream in the String format, please choose the
org.apache.kafka.common.serialization.StringSerializer
for thekey.serializer
and/orvalue.serializer
Last modified 8mo ago