I’ve been thinking about building systems that handle massive scale without breaking a sweat. Recently, while designing an order processing platform, I hit performance walls with traditional approaches. That’s when I revisited event-driven architecture with modern tools. Let me show you how combining Spring Boot, Apache Kafka, and Java 21’s virtual threads creates systems that scale effortlessly while keeping code clean.

Our journey starts with the core setup. For Kafka integration in Spring Boot, include these essentials:

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>
</dependencies>

Now, how do we define our events? Clear models prevent headaches later:

public record OrderEvent(
    String orderId, 
    String customerId,
    List<OrderItem> items,
    BigDecimal amount
) {}

For high-throughput producers, virtual threads change everything. Notice the @Async annotation:

@Service
public class OrderPublisher {

    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    @Async
    public void publishOrder(OrderEvent event) {
        kafkaTemplate.send("orders", event.orderId(), event)
            .whenComplete((result, ex) -> {
                if (ex != null) {
                    log.error("Delivery failed for {}", event.orderId(), ex);
                }
            });
    }
}

Why struggle with thread pools when virtual threads handle thousands of concurrent tasks with minimal overhead? Java 21’s lightweight threads integrate seamlessly with Spring’s @Async. Configure them in your application.yml:

spring:
  threads:
    virtual:
      enabled: true

On the consumer side, we need robust processing. Ever wonder what happens when messages arrive faster than you can handle them? Virtual threads prevent consumer lag:

@KafkaListener(topics = "payments")
public void handlePayment(PaymentEvent payment) {
    paymentService.process(payment); // Runs in virtual thread
}

For resilience, implement a dead letter queue pattern:

@Bean
public ConsumerFactory<String, PaymentEvent> paymentConsumerFactory() {
    return new DefaultKafkaConsumerFactory<>(
        consumerConfigs(),
        new StringDeserializer(),
        new ErrorHandlingDeserializer<>(new JsonDeserializer<>(PaymentEvent.class))
    );
}

@Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, PaymentEvent>> 
    kafkaListenerContainerFactory() {
    
    ConcurrentKafkaListenerContainerFactory<String, PaymentEvent> factory =
        new ConcurrentKafkaListenerContainerFactory<>();
    factory.getContainerProperties().setObservationEnabled(true);
    factory.setCommonErrorHandler(new DefaultErrorHandler(
        new DeadLetterPublishingRecoverer(template),
        new FixedBackOff(1000L, 3L))
    );
    return factory;
}

Performance tuning is crucial. These producer settings balance throughput and reliability:

spring:
  kafka:
    producer:
      batch-size: 16384
      linger-ms: 20
      compression-type: snappy
      buffer-memory: 33554432

For monitoring, expose Kafka metrics via Spring Actuator:

management:
  endpoint:
    metrics:
      enabled: true
  endpoints:
    web:
      exposure:
        include: metrics

Now, what about testing? Use embedded Kafka for reliable integration tests:

@EmbeddedKafka
@SpringBootTest
class OrderProcessingTests {

    @Autowired
    private KafkaTemplate<String, OrderEvent> template;

    @Test
    void shouldProcessOrderWithinTimeout() {
        template.send("orders", testOrder);
        // Assertions
    }
}

When deploying, remember these production essentials:

• Set auto.create.topics.enable=false in Kafka brokers
• Always use idempotent producers
• Monitor consumer lag with Prometheus
• Configure proper retention policies

I’ve seen teams struggle with thread exhaustion under load. Virtual threads eliminate that. One client handled 10x more orders without adding servers. The key? Letting virtual threads manage concurrency while Kafka handles message durability.

What problems could this solve in your systems? Share your thoughts below. If this helped you, pass it to someone facing scaling challenges. Your comments fuel deeper explorations of these patterns.