Inspecting streams
In addition to CRUD operations for managing Stream objects, DataCater offers the Inspect operation.
The Inspect operation allows you to view the most recent records of the underlying Apache Kafka topic and is very useful for exploring your data and debugging pipelines.
You can either call the API endpoint for retrieving the raw records of a Stream or inspect a Stream in our UI. The latter lets you explore the values of your records in an interactive grid:
Inspecting the stream external.companies using DataCater's UI.
DataCater offers two methods for sampling the records from a Stream.
This is the default sampling method. When retrieving
X records from a Stream, we fill up the sample set one partition at a time until x records are reached, moving from the first partition, partition 0, to the second partition, partition 1, and so on. This could mean that only records from partition 0 will be returned when inspecting the Stream. On the other hand, it is guaranteed, that X records will be returned, provided that the Stream contains at least X records.The sampling method
uniform tries to create a sample set that holds an equal number of records from each partition of the Apache Kafka topic. When creating a sample set of X records for a Kafka topic with N partitions, this sample method consumes up to X / N records from each partition. Please note that this sampling method does not guarantee to return X sample records. If at least one of the N partitions contains less than X / N records, the sampling method uniform returns less than X records. If you do not require an equal distribution of the sample records across the partitions of the Kafka topic , please consider using the sampling method sequenced.