Kafka + WebSocket System

Overview

• Kafka messaging system integrated with WebSocket.
• Node.js runtime.
• Varying configurations of resources and message loads.

Configurations

• Number of connections: 100-1000
• Messages per connection: 1-500
• CPU Cores: 2-4
• Memory: 1GB-4GB

📈 Scalability

• The system demonstrated good scalability, handling up to 1000 concurrent connections.
• Peak performance: 5,356 messages sent per second with 1000 connections and 50 messages per connection.

⚡️ CPU and Memory

• 4 CPU core, 4GB Memory: memory overhead significantly reduced performance (~16% compared to 2 CPU cores and 1GB memory)
• 2 CPU cores 1GB Memory: reduced performance by 1.5x compared to 4 CPU cores and 4GB memory
• 4 CPU cores 1GB Memory: Most Optimal and Best Performance

Message Batching:

• Larger message batches per connection improved overall throughput. Reason being, system was able to process multiple messages at once by utilizing kafka's batching mechanism.
  • 1 message/connection: 344.13 messages/second
  • 2 messages/connection: 985.28 messages/second
  • 100 messages/connection: 1,282.69 messages/second

🕙 Latency:

• 99th percentile connection duration increased with load but not proportionally.
• Light load: 2.89 seconds
• Heavy load: 7.75 seconds

🤌🏻 Conclusion:

• The system shows robust performance and good scalability, handling increased loads efficiently.
• Four cores provide optimal performance, suggesting effective parallel processing.
• Message batching significantly improves throughput, indicating efficient use of kafka resources.
• Consistent message delivery across various loads demonstrates system reliability.
• While latency increases with load, the non-linear growth suggests good system design for handling concurrent connections.