When testing client performance, an important benchmark is per-record latency. This includes ingestion latency (ingestion timestamp - event timestamp), consumer latency (consumer wallclock timestamp - ingestion time);

Whilst there are tools like kafka.tools.TestEndToEndLatency for running such tests with producer/consumer primitives, testing a higher order producer like Kafka Connect source connector is a challenge. Many users rely on manual, ad-hoc observations for benchmarking how long it took for a source connector to write a record into Kafka.

In this session, we shall cover kafka-latency-analyzer, a script that can use configurable timestamps to produce quantile reports of latency (focused on average and tail) at a topic level and optionally relay such results downstream to another kafka topic for analysis ans also includes features such as sampling; This approach is useful for providing latency SLAs, as well as for performing exploratory benchmarks & tuning of clients/connectors;

What we shall cover:

1. Understanding ingestion latency: CreateTime vs LogAppendTime (and why additional timestamps need to be injected for true event time in sources)

2. Understanding other connector bound latencies involved (such as SMT chains with connectors)

3. Understanding consumer / processing latency, as seen by a consumer: ie, WallClockTime (although we shall also consider how to introduce timestamp injection for sink-side write)

4. Analysis of results: Optimization based on latency distribution (mean and tail quantiles: p90, p95, p99)

5. Managing results of test runs with your observability tooling: OpenMetrics, Prometheus, Datadog integration & more