Message queue and task queue

Consider that you are preparing for a party at your home. There are several tasks to be done. Walls are to be decorated, plates are to be washed, food is to be made ready and you may need an extra shoe rack for guests to leave their footwear. Most probably, you do not do everything alone. You get the help of the members of your household. In fact if the load is too much, then you even ask your friends for help.

When more people work on the tasks for organising a party, you get multiple things done parallely. But with more participation comes the need for co-ordination and communication. Some of the tasks may be related to each other, e.g. someone who has promised to wipe washed dishes and stack them on your shelf will need the one who washes dishes to complete his/her task first. Usually, one of the participating persons needs to stay on top of what gets done and who does what. He/she is like the captain of a sports team. When everyone smoothly communicates the status of his/her tasks and one or two persons are aware of each task’s status, things go on quite well despite the complexity.

Modern computers work the same way. They have multiple processors capable of executing multiple things at the same time. Even a single modern processor performs time slicing, i.e. divides its attention among several processes / threads at the same time. Processes need to communicate among each other. For this, they use a system called a message queue.

Whenever one process wants to communicate with another, the sender process sends out a piece of communication to the receiver. This piece is called a message. The receiving process needs to have a system to receive and process incoming messages, just like you have an inbox for email and a postbox for your home. This system is called a message queue.

Here is how we can describe an application driven by a message queue.

Here is what the message queue consists of.

The following systems have the concept of message queues built in:

The above systems support message queues out of the box, so there is no special third-party software that you need to install to unleash communication between processes running in parallel.

Message queues can be of two types: centralised and distributed. In a distributed message queue, each process has its own message queue and every other process needs to know the address of or have a reference to that queue in order to send messages to it. Each process can only read from its own queue. It has no idea what is going on in all other queues.

But what if your messaging needs were more complicated. What if you want to broadcast a message to multiple processes so that they all get to work on the same signal. E.g. a hard disk running out of space is of concern to all processes who wish to write to the hard disk. What if processes want to opt in and opt out of receiving messages based on certain conditions? E.g. a user will either want to or not want to receive notifications from an app.

In this case, a single message queue is used. All processes use this queue for their messaging. Such as system is called a centralised queue. Here are some features of a centralised queue.

A centralised queue is made to solve the following problems

We mentioned 5 frameworks in which message queues are built-in. What if we want to use them in other languages? E.g. a standard Java or a Python program? We have two types of solutions.

So far we have talked about message queues where one process sends a message in the form of a small instruction plus some data to another process. It is upto the receiving process to set up an algorithm that actually uses that instruction and data. This is akin to you asking your computer graphics savvy friend to convert a TIFF image to a JPEG image for you. He has the know-how. All you gave him was an instruction ‘convert’ and the data, i.e. your TIFF file. He’ll know how to open Photoshop and use the Save As menu and the array of options that allow us to set the image quality. He’ll message you back when he’s done.

But what if you have hired 5 new trainees who do not know how to use Photoshop? In this case, you will have to tell them how to open a TIFF image, which menus to navigate through to bring up the option to save as JPEG and how to control the quality options to get the desired quality of output. Instead of giving them just an instruction, you’d be giving them detailed steps, like an algorithm. What’s the use of hiring trainees if you have to tell them detailed steps when you can do those steps yourselves? Well, what if you have 100 images to convert from TIFF to JPEG? I am sure your work hours are more valuable than converting images from one form to another. You may need to document and tell your trainees those steps once, but they will continue to use those instructions as long as they are trainees with you.

Computer applications can work similarly. Instead of an instruction, the application’s main process can hand out entire algorithms to processes and threads. These processes will do the hard work while our application’s main process can stay responsive to the user.

Similar to message queues, processes can have their own task queues. Other processes can fill up the task queue with algorithms, which the process with the task queue will pick up and execute as is. This is similar to the trainees executing your detailed instructions to the dot.

To get things done at the double, programming languages offer threading and multi-tasking. But this makes it necessary for multiple processes to communicate with each other. Messaging queues and task queues offer the solution by allowing processes to send short instructions, status messages or even entire algorithms between each other.

Related posts:

Making sure that your efforts are worth it can be difficult in the current society, even more, considering COVID-19 and the whole quarantine situation. A lot of things have changed, lots of people…

Today I can feel myself crack-crack-cracking beneath the weight of the Ford/Kavanaugh hearing, workplace trauma that re-surfaced to bite me in the ass, many sleepless nights, and medical unknowns…

Do you feel stuck or need more motivation? Your environment might be to blame. In this blog, we explore the power of your surroundings to influence your mood, mindset, and inspiration. Discover how to create an environment that inspires you to achieve your goals and live your best life.