Topic Creation & Partition Management
In Apache Kafka, a Topic is a named feed or category where records are published. Topics in Kafka are partitioned, which allows data to be split and scaled across multiple brokers.
Topic Log Structure
- Partition: An ordered, immutable sequence of records. Each partition is hosted on a broker.
- Offset: A unique, sequential ID assigned to each record inside a partition.
- Replication Factor: The number of copies of partition data maintained across different brokers for fault tolerance.
Since this is a single-node sandbox running in KRaft mode, you will create a topic with multiple partitions and a replication factor of 1.
Step 1: Create the Inventory Purchases Topic
Your supermarket company wants to track consumer transaction events. Create a topic named inventory_purchases with 3 partitions and a replication factor of 1.
Execute the following command in the terminal to create the topic:
# Create the topic with 3 partitions and a replication factor of 1
kafka-topics --create --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1 --topic inventory_purchases
Step 2: List Active Topics
Verify that your topic was created successfully by listing all active topics in the cluster:
# List all topics in the cluster
kafka-topics --list --bootstrap-server localhost:9092
You should see inventory_purchases in the terminal output.
Step 3: Describe Topic Details
To view detailed configuration, partition leadership, and replica distribution for your topic, run:
# Describe the metadata of the inventory_purchases topic
kafka-topics --describe --bootstrap-server localhost:9092 --topic inventory_purchases
Note the PartitionCount, ReplicationFactor, and the leader mapping for each partition.
Click Verify step below once you have created the topic to proceed.
Hint
Use `kafka-topics --create --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1 --topic inventory_purchases`.
Producing and Consuming Events
Now that your topic is created, you can publish (produce) and read (consume) data records.
Definitions
- Producer: An application or client that publishes records to a Kafka topic.
- Consumer: An application or client that subscribes to a topic and reads its records.
In this step, you will use Kafka's command-line utilities to simulate a supermarket checkout publishing inventory purchase events, and a backend service consuming them.
Step 1: Start the Console Producer
Start the console producer client to write messages to the inventory_purchases topic:
# Start the interactive console producer
kafka-console-producer --bootstrap-server localhost:9092 --topic inventory_purchases
Once the producer starts, you will see a blank cursor line. Type the following three test messages, pressing Enter after each:
product: apples, quantity: 5
product: lemons, quantity: 7
product: oranges, quantity: 12
Once finished, terminate the producer session by pressing Ctrl + C.
Step 2: Consume Messages from the Beginning
Start the console consumer to retrieve and display the records you just published:
# Start the console consumer to read all messages from the beginning
kafka-console-consumer --bootstrap-server localhost:9092 --topic inventory_purchases --from-beginning
You should see your checkout messages print to the terminal. Press Ctrl + C to close the consumer.
Step 3: Stream Ingestion to a Local Log File
To verify and record your consumption, start a consumer session that writes its output directly to a file:
# Stream the topic contents into a local output file
kafka-console-consumer --bootstrap-server localhost:9092 --topic inventory_purchases --from-beginning --max-messages 3 > /tmp/consumed_purchases.txt
Verify that the file contains your checkout events:
# Print the contents of the file
cat /tmp/consumed_purchases.txt
Click Verify step below to check your log file and proceed.
Hint
Check that the file `/tmp/consumed_purchases.txt` contains your published test messages.
Scaling with Consumer Groups
When multiple consumers belong to the same Consumer Group, they share the consumption load. Kafka dynamically assigns each topic partition to exactly one consumer within the group.
If a topic has 3 partitions:
- 1 Consumer in the group: Receives messages from all 3 partitions.
- 3 Consumers in the group: Each receives messages from exactly 1 partition.
- 4 Consumers in the group: 3 consumers get 1 partition each; 1 consumer remains idle as a hot standby.
Step 1: Open a Second Terminal Tab
To observe group scaling in action, you need a new terminal tab. Click the + button at the end of the workspace tabs to open a new terminal (Terminal 2).
Step 2: Start a Consumer in a Specific Group
In Terminal 1, start a console consumer specifying a consumer group named supermarket-group:
# Start a consumer in the supermarket-group
kafka-console-consumer --bootstrap-server localhost:9092 --topic inventory_purchases --group supermarket-group
Now, switch to Terminal 2, and start a second consumer inside the same group:
# Start a second consumer in the same supermarket-group
kafka-console-consumer --bootstrap-server localhost:9092 --topic inventory_purchases --group supermarket-group
Step 3: Produce New Messages to Observe Load Balancing
Open Terminal 3 (click the + tab button again) and publish more test events:
# Start the console producer
kafka-console-producer --bootstrap-server localhost:9092 --topic inventory_purchases
Type these lines:
product: grapes, quantity: 22
product: bananas, quantity: 15
product: cherries, quantity: 30
Press Ctrl + C to close the producer.
Now, inspect Terminal 1 and Terminal 2. Because both consumers belong to supermarket-group and read from different partitions, the incoming messages are split between the two terminals!
Step 4: Verify Consumer Group States
In Terminal 3, run the following command to check active consumer group metadata and partition lags:
# Describe the supermarket-group details
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group supermarket-group
Observe the current offsets, log end offsets, partition lags, and active consumer IDs.
Click Verify step below to complete the lab.
Hint
Verify that you have run a console consumer inside a specific group (e.g. `--group supermarket-group`).