Unraveling the Magic of Message Queues and How To Use Them
What role do they play in modern software systems? Let's find out.
Time and again, you’ve probably wondered what role do message queues play in today’s software infrastructure. Well, they play a pivotal role.
Imagine a world without message queues where every task was done synchronously.
Imagine a scenario where a payment processor such as Stripe only used synchronous means of handling “payment succeeded” checks.
The system would have to wait for the payment verification process to complete before moving on to the next payment request. This could result in a backlog of payment requests, causing delays and potentially frustrating customer experiences.
For instance, suppose a popular e-commerce website experiences a surge in sales during a holiday season and relies on a payment processor that only uses synchronous means of handling payment checks. In that case, the payment verification process may take longer to complete, leading to slower transaction times and potentially causing some customers to abandon their purchases.
So, what do we need to solve this havoc on customer experience?
The payment processor may need to implement message queues to handle payment verification asynchronously. With message queues, payment verification requests can be queued up and processed in the order they are received, even if there is a backlog of requests. This would help reduce delays and improve the customer experience.
So what happens in a message queue?
In simple terms, a message queue acts as a post office for our applications — it collects messages from the senders (producers) and delivers them to the receivers (consumers).
And guess what? All of this happens asynchronously, allowing our apps to work on other tasks while the message delivery is in progress!
How do they work?
The workings of a message queue are pretty straightforward:
1. The producer sends a task to the queue and updates the user about the task’s status.
2. A consumer picks the task from the queue, processes it, and indicates that it’s done.
Quite simple, innit?
The messages stay in the queue until they are processed and removed.
And the best part? Each message gets the undivided attention of a single consumer and is processed only once.
The Superpowers of Message Queues
Here are some fantastic benefits of using message queues:
Scalability: Message queues give you the power to scale your system exactly where it’s needed.
Decoupling: They help in breaking down dependencies between components, simplifying the implementation of decoupled applications.
Performance: Message queues enhance performance by enabling asynchronous communication. Producers can add requests to the queue without waiting for them to be processed. This is very important!
Reliability: They make our data persistent and reduce errors when different parts of our system go offline.
Some key types of message delivery to keep in mind
Push or Pull Delivery: Most message queues provide both push and pull options for retrieving messages.
FIFO (First-In-First-Out) Queues: In these queues, the first message that comes in is the first one to get processed.
At-Least-Once Delivery: Message queues may store multiple copies of messages and resend them in the event of failures to ensure they are delivered at least once.
Exactly-Once Delivery: FIFO message queues ensure that each message is delivered exactly once by filtering out duplicates automatically.
Schedule or Delay Delivery: Many message queues support setting a specific delivery time for a message.
A practical example of using Message Queues
Let’s design an efficient email sending architecture using RabbitMQ as a message queue.
1. Overview:
The system design will consist of three main components: the sender service, RabbitMQ message queue, and the email service.
The sender service will be responsible for sending email requests to the message queue, where they will be queued up for processing.
The email service will then retrieve the email requests from the message queue and send them out.
2. The Sender Service:
The sender service will be responsible for sending email requests to the message queue.
It will receive email requests from the frontend (like a React app) or backend application ( a Python or NodeJS API) and publish the requests to the RabbitMQ message queue.
The sender service will also ensure that each email request includes all the necessary information, such as the recipient’s email address, subject, and content (can be either or both text or html).
3. The RabbitMQ Message Queue:
The message queue will store the email requests in a queue until they are ready to be processed by the email service.
The message queue will ensure that the email requests are processed in the order they are received, and it will also handle any errors that occur during the email sending process.
4. The Email Service:
The email service will then be responsible for retrieving email requests from the message queue and sending them out.
It will constantly monitor the RabbitMQ message queue for new email requests and retrieve them as they become available.
The email service will also handle any errors that occur during the email sending process, such as invalid email addresses or network connectivity issues.
This can simply be a backend API within the system that calls an external Email API (such as from SendGrid or Mailgun) that you chose use with your project.
So what are the benefits of this system really?
The use of a message queue for the email sending architecture provides several benefits.
Firstly, it helps to improve the overall performance and scalability of the system by decoupling the email sending process from the frontend or backend application.
Secondly, it helps to ensure that email requests are processed in the order they are received, improving the reliability and consistency of the system.
Finally, it provides a better way to handle errors and exceptions that may occur during the email sending process, improving the overall resiliency of the system.
A few parting words…
We looked at how message queues are an essential part of modern, large-scale systems and provide a robust and efficient method for handling asynchronous communication and ensure the smooth functioning of various components.
If you learned something from this post, please do tell a friend about it! :)
I’ll see you in the next one.