Building Event-Driven Microservices with C# and Kafka

Event-driven architecture has become increasingly popular for building scalable, loosely-coupled microservices. In this post, we'll explore how to implement event-driven patterns using C# and Apache Kafka.

Why Event-Driven Architecture?

Traditional request-response patterns create tight coupling between services. Event-driven architecture allows services to communicate asynchronously through events, improving scalability and resilience. When a service publishes an event to Kafka, multiple consumers can react independently without blocking the producer.

Setting Up Kafka with C#

First, install the Confluent.Kafka NuGet package:

bash

dotnet add package Confluent.Kafka

Implementing a Producer

Here's how to create a simple producer that publishes order events:

csharp

using Confluent.Kafka;

public class OrderEventProducer
{
    private readonly IProducer<string, string> _producer;

    public OrderEventProducer(string bootstrapServers)
    {
        var config = new ProducerConfig
        {
            BootstrapServers = bootstrapServers,
            EnableIdempotence = true,
            Acks = Acks.All
        };
        _producer = new ProducerBuilder<string, string>(config).Build();
    }

    public async Task PublishOrderCreatedAsync(string orderId, string orderData)
    {
        var message = new Message<string, string>
        {
            Key = orderId,
            Value = orderData
        };

        var result = await _producer.ProduceAsync("order-events", message);
        Console.WriteLine($"Published to {result.Topic} partition {result.Partition} at offset {result.Offset}");
    }
}

Implementing a Consumer

csharp

using Confluent.Kafka;

public class OrderEventConsumer
{
    private readonly IConsumer<string, string> _consumer;

    public OrderEventConsumer(string bootstrapServers, string groupId)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = bootstrapServers,
            GroupId = groupId,
            AutoOffsetReset = AutoOffsetReset.Earliest,
            EnableAutoCommit = false
        };
        _consumer = new ConsumerBuilder<string, string>(config).Build();
    }

    public void StartConsuming()
    {
        _consumer.Subscribe("order-events");

        while (true)
        {
            var consumeResult = _consumer.Consume();

            try
            {
                ProcessOrder(consumeResult.Message.Value);
                _consumer.Commit(consumeResult);
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error processing message: {ex.Message}");
                // Implement retry or dead letter queue logic
            }
        }
    }

    private void ProcessOrder(string orderData)
    {
        // Process the order event
        Console.WriteLine($"Processing order: {orderData}");
    }
}

Key Considerations

When building event-driven systems with Kafka, keep these principles in mind: ensure idempotency in your consumers, implement proper error handling with retry policies, use schemas for event contracts, monitor consumer lag, and design for eventual consistency.

Event-driven architecture with C# and Kafka provides a powerful foundation for building scalable microservices. Start small, validate your design with real traffic, and iterate based on operational metrics.

Appearance

Language & Time

Desktop app