RabbitMQ Series - Part 1

Learning about Publisher and Exchange

Dikshit Maheshwari

Aug 27, 2023

Imagine you are Dwight(the intern). Post his failure at Netflix, he joined Amazon as an intern(again, oh Dwight).

25 "The Office" Memes And Gifs That Need Little To No Explanation

At Amazon, he was given the task of removing

Checkout service(finalize the order and process the inventory),

History service(update customer’s order history) and

Shipping service(ship the order to the customer) from a Monolith structure and create each of them as a separate micro-services.

in your face, Dwight | The office show, The office characters, Office quotes

That’s tough. How is he gonna handle the flow of message from one service to another. How will be handle the processing of messages in order. How will he make sure that all finalized orders are shipped or that the order history is updated.

Should he run away? How is he gonna do it? Well Dwight, let me teach you about RabbitMQ and help you out.

Where, Dwight?' I think this makes for a better Anakin/Padme meme template if you need the sides swapped : r/DunderMifflin

RabbitMQ is an open-source message broking system that helps in deploying a message queuing mechanism between two applications. Think of a post office. You want to deliver a letter to some place, you deposit it in a post box, which then goes to the post office, and from there, the postman delivers it to your place. A RabbitMQ is everything here, - the post box, the post office and the post man. It takes in a message, routes to it to the right place, and then it gets delivered there. RabbitMQ is often used in event-driven system architecture.

Formal definition done. Let’s learn with Dwight’s problem above and help him. There are two different kinds of events here :
Checkout service → History Service [Update customer’s order history in their Amazon account].

Checkout service → Shipment Service [Process the shipping of the customer’s item].

Let’s learn with an example. But before that,

Basic concepts of RabbitMQ :

Producer - The creator of the event that needs to be processed.
Consumer - Subscribes to the queues and receives messages and acts on those messages.
Queue - A sequential data structure to store and transfer the messages.
Binding - A binding is a “connection” between a queue and an exchange.
Routing Key - A message attribute that the exchange uses when deciding how to route a message.
Exchange - An exchange is like the post office, responsible for the distribution of messages. Unlike traditional post offices, an exchange can deliver messages based on the type of Exchange.
Different types of Exchanges in RabbitMQ :
- Direct Exchange : Uses a message routing key to transport messages to queues. The producer adds the routing key to the message header, and the exchange sends the message to the queue with the binding key that exactly matches the message’s routing key.
- Topic Exchange : Sends messages to queues based on wildcard matches between the routing key and the queue binding’s routing pattern. The direct exchange is great, but it has a lot of limitations - it doesn’t routing based on multiple criteria.
  With topic exchange, you can use a combination of wildcards and strings to deliver a message to multiple queues. Topic exchange is like a parent to Direct exchange, where it uses the functionality of Direct exchange, while also adding to that using two special cases :
  - *(star) - can substitute for exactly one word.
  - #(hash) - can substitute for zero or more words.
  - In the example above, the routing key consists of three words(two dots). <W1, W2, W3> is a generalization of how the routing key will look here.
  - How would the three routing-keys given above behave
    - *.orange.* - Any string where the middle word is orange, and has a first and third word, would match this key. Eg : start.orange.end, start.orange.fox. However, if we have a string like orange, rabbit.orange, orange.blue, these would not work.
    - *.*.rabbit - The last word in a three word set separated by dots should be rabbit. Eg : any.string.rabbit, hi.bye.rabbit etc. However, string like orange.rabbit, rabbit.blue will not work here.
    - lazy.# - Any number of words after lazy. will fall under this category.
  - Topic exchange is powerful and can behave like other exchanges. If a queue is bounded with '#(hash)’ binding, it acts like a fanout exchange. When it doesn’t use any of the special characters like star or hash, it acts as a direct exchange.
- Fanout Exchange : Duplicate and route messages to any/all queues, irrespective of the routing keys or pattern matching. This type of exchange gets useful when the event is not specific to a particular consumer, for eg : storm alerts etc.

Basic RabbitMQ Flow

Coming back to our problem, let’s start with the Checkout Publisher.

import pika
import sys

class CheckoutPublisher:
    @staticmethod
    def create_connection():
        connection = pika.BlockingConnection(
            pika.ConnectionParameters('localhost')
        )
        return connection
    
    def publish_message(self, routing_key, message):
        connection = CheckoutPublisher.create_connection()
        channel = connection.channel()
        # Checks for that exchange or else creates it
        channel.exchange_declare(
            exchange='AmazonCheckout',
            exchange_type='topic'
        )
        channel.basic_publish(
            exchange='AmazonCheckout',
            routing_key=routing_key,
            body=message
        )
        connection.close()

publisher = CheckoutPublisher()
publisher.publish_message("Event.HistoryConsumer", "Message for history task processing")
publisher.publish_message("Event.ShippingConsumer", "Message for shipping task processing")

In RabbitMQ, a message can never be directly sent to a queue. It needs to go through an exchange. A topic exchange works best for our use case here, since we use a single exchange to send events to different queues and consumers based on the routing key in the message.

Run the above code :