> For the complete documentation index, see [llms.txt](https://tingreavinash.gitbook.io/the-tech-toolbox/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://tingreavinash.gitbook.io/the-tech-toolbox/notes/apache-kafka.md).

# Apache Kafka

### Introduction

Kafka is a distributed streaming platform used to build real-time data pipelines and streaming applications. It is designed to be fault-tolerant and horizontally scalable, and provides high-throughput and low-latency data delivery.

### Core Concepts

#### Topic

A topic is a category or feed name to which records are published. A topic is identified by its name, and a topic can have multiple partitions. A partition is a unit of parallelism in Kafka.

#### Partition

A partition is a unit of parallelism in Kafka. A topic can have one or more partitions, which allows for messages to be distributed across multiple brokers.

#### Offset

An offset is a unique identifier assigned to each message in a partition. It represents the position of a message in a partition.

#### Producer

A producer is a process that writes data to a Kafka topic. It sends records to a Kafka broker, which then writes the records to the appropriate topic and partition.

#### Consumer

A consumer is a process that reads data from a Kafka topic. It subscribes to one or more topics and partitions, and reads messages from them.

#### Broker

A broker is a Kafka server that receives messages from producers and sends messages to consumers. Each broker can handle multiple partitions and topics.

#### ZooKeeper

ZooKeeper is a distributed coordination service used by Kafka to maintain configuration information and provide distributed synchronization. Kafka brokers use ZooKeeper to discover each other and to elect a leader for each partition.

#### Consumer Group

A consumer group is a group of consumer processes that collectively consume messages from one or more topics. Each consumer in a group is assigned a subset of the partitions for the topics it is consuming.

#### Replication

Replication is the process of copying data from one broker to another for fault tolerance. Each partition has a leader replica, which is responsible for handling read and write requests. The leader replica replicates data to one or more follower replicas for fault tolerance.

#### Leader Replica

The leader replica is the replica in a partition that is responsible for handling read and write requests. It is the only replica that can receive data from producers and send data to consumers.

#### Follower Replica

A follower replica is a replica in a partition that receives replicated data from the leader replica. It is used for fault tolerance, and can take over as the leader if the current leader fails.

#### ISR (In-Sync Replica)

The ISR is the set of replicas that are currently up-to-date with the leader replica. Messages are only committed when they are written to all replicas in the ISR. If a replica falls out of sync with the leader replica, it is removed from the ISR.