Integrating existing Apache Kafka topics

Self-managed installations of DataCater (Open-Core or Enterprise) allow you to integrate existing Apache Kafka topics. You can add a new stream and connect it with a Kafka broker and topic. Depending on whether you want to publish data to or consume data from the stream (or topic), you can set the according properties of the official Apache Kafka consumer and producer API:
In addition to the name of the topic and the connection properties, you must let DataCater know the format of the data in the topic. To this end, you can define deserializers and serializers for both the key and the value of the records. By default, DataCater expects data to be formatted in JSON.
Integrating topics holding JSON
DataCater provides serializers and deserializers for handling JSON data:
If you want to consume JSON data from a stream, please choose the io.datacater.core.serde.JsonDeserializer for the key.deserializer and/or value.deserializer
If you want to publish data to a stream in the JSON format, please choose the io.datacater.core.serde.JsonSerializer for the key.serializer and/or value.serializer
Integrating topics holding AVRO
DataCater provides serializers and deserializers for handling JSON data:
If you want to consume AVRO data from a stream, please choose the io.datacater.core.serde.AvroDeserializer for the key.deserializer and/or value.deserializer
If you want to publish data to a stream in the AVRO format, please choose the io.datacater.core.serde.AvroSerializer for the key.serializer and/or value.serializer
Please note that you need to provide a schema when using AVRO. To this end, either point DataCater to a schema registry using the property schema.registry.url or provide an inline schema using key.deserializer.schema, value.deserializer.schema, key.serializer.schema, or value.serializer.schema.
Integrating topics holding Strings
DataCater can work with the (de)serializers provided by Apache Kafka for handling raw strings:
If you want to consume Strings from a stream, please choose the org.apache.kafka.common.serialization.StringDeserializer for the key.deserializer and/or value.deserializer
If you want to publish data to a stream in the String format, please choose the org.apache.kafka.common.serialization.StringSerializer for the key.serializer and/or value.serializer

Last modified 8mo ago