What I covered
I built the whole talk around one Swiggy order, from tapping “Place order” to the restaurant, rider and notifications all knowing about it. A single map of the pipeline came back on screen as each topic started, with that part highlighted, so the room always knew where we were. Every piece opened with the naive way of doing it and what breaks, before showing how Kafka solves it.
- Kafka core: why Kafka exists, and how it differs from a regular message queue.
- Kafka Connect: how data moves in and out of Kafka without integration code.
- Schema Registry: how it stops a small data change from quietly breaking other teams.
- Kafka Streams: computing live results straight from the stream of events.
- Confluent Cloud: running the same pipeline without managing servers.
Live demo
- Insert an order into Postgres; a Debezium source connector picks it up.
- The order lands on the
order-eventstopic within seconds, its shape checked by Schema Registry. - A Kafka Streams app updates that customer’s running total live.
The demo ran on a local stack, so there’s no public repository.
Gotchas I shared
- Size the partition count for peak load up front. Changing it later moves keys and breaks the ordering you relied on.
- Add new fields as optional. Don’t rename or remove fields without coordinating with every reader.
- Always configure a dead-letter topic on Connect, so one bad record can’t stop the pipeline.
- If you alert on one thing, make it consumer lag.
How it landed
People said following one order made the concepts easy to grasp, and the live demo made the pieces click.
