RabbitMQ Retry Architecture

Event-Driven architecture services communicate by messages using Message brokers like RabbitMQ. The entire transaction will be completed if the message will get processed successfully. While processing messages from Queue there will be a chance of failure due to invalid data or resources unavailable.

If data is invalid, then the message will get failed while processing. In this case, we can reject the message from the Queue and needs to send a notification to the corresponding services about the invalid data. Resource unavailability will be based on completing the entire transaction in the distributed system. The availability time will vary from milliseconds to seconds in distributed systems.

Once Resource is available, we need to process the message again. We need to retry the message in case of failure that will increase our system reliability and accuracy.

Retry Mechanism

We can retry the message from RabbitMQ using two approaches.

  1. Rollback
  2. Retry with Dead Letter Exchange

Rollback

While using Rollback the message will get requeued again. The time taken to process the requeued message will be based on the existing message count in the queue. We can also requeue the message after a period X. In this case, we need to keep the thread open until the X period. In case there a lot of messages are getting failed then there will be a lot of threads that will wait to complete the X period for each message. It's not recommended to requeue the message after the X period. If your resource will be available after requeuing the message without any waiting period, then we can requeue the message otherwise we need to add it to Dead Letter Exchange.

But due to fault tolerance, service will not be available for certain periods of time or other reasons. In this case, we need to keep the message in the queue certain period. We can achieve this using Dead Letter Exchange.

Dead Letter Exchange

Each queue can configure a Dead Letter exchange. If Queue is configured with Dead Letter exchange, then rejected or expired message will be moved into the Dead Letter Exchange otherwise it will be get removed from the Queue. Dead Letter Exchange will push those messages into the corresponding bounded Queues.

Queue Configuration

Dead Letter Configuration

Bind Dead Letter Exchange

Retry Messages

Here we need to retry the message from DeadLetterQ, In this case, we need to bind the DeadLetterQ with notification Exchange.

I have bound notification exchange as a Dead Letter exchange for DeadLetterQ, so the message will be there in DeadLetterQ for the period. Once it's expired it will be pushed into notification exchange. From notification exchange, it will flow into the SMS queue. The flow is getting repeated. We can set the threshold for message retry. Once it reached the threshold, we can complete the message.

If you want to add some processing on failure messages, then we can connect to DeadletterQ and we can process the message from the Queue. It will be based on the system architecture and business needs.