Often you and yourself in a situation where you just need to push a message to remote system to do work and you don’t wait to block the downstream system completing its tasks
There are two types
- Point-To-Point
- Publish-Subscribe
Point-To-Point
One of the easiest way and one of the most common uses in Microservices architecture to more to asynchronous messaging. The calls can replace traditional RESTful calls between services where the response is not needed or can be received an out-of-band process.
- There is single producer that create message and puts on the message broker
- There is also single consumer who responds or listen to the message and does some action on it
- Its a send and forgot model
- The producer create a message , dispatches it and goes on
- As such kind of response is usually another point-to-point message
Some of the Use cases
- No response needed:
More common when no response needed. is the client of the services does not really need to know that the tasks is done there are more no valid reason to wait for it to finish. you can produce a message and fire it of and essentially forget about it
For Example : Place used with Audit record that are not are mission critical if the audit record it self is not a hard fast system requirement. there is no reason to wait for it get written. you can fire message to the message broker intend for the audit system and just expect it to get written
You can’t guarantee that its is written or even done so in timely manner in this module but with proper tuning you will get the record written with high degree of success and save your time for some processing time
- Admin Tasks
When your have admin tasks that need to happen
For Example: Used for clean up operations on aggregated data that was denormalized for efficiencies in an RDMS system. when i rolled that data to a new view, i had no need for the original pieces of data any longer. so fired off a message that triggered the cleanup operation to remove those individual pieces. this not only allow my aggregated data queries to be more efficient but it also let this single process move faster since it just aggregated and left the cleanup to another process.
If tasks did not execute for some reasons because the data was time series in nature, the next admin task execution would clean up after the fail attempts. And this is the case where ignored DLQ. In this case, Only cared missed X number of attempts because that was indication that something was really wrong
- Out of Band
There are other out of band processes that may need to be run as well. that can benefit from point to point asynchronous messaging
For Example: Email itself is asynchronous and not guaranteed for delivery. As such there is not reason to wait. for the building of the email and sending it out since after all,it was just a be an empty result at the end
The producer would put a message on a queue and communication service would pick it up and send the email. In the end nothing guaranteed any way so the improved efficiencies were well worth it
- Scaling
Once You start offloading the processing and blocking nature of calls that would qualify for asynchronous messaging it become easier to scale the system. when start scaling base on the true blocking calls instead of everything being a wait-and-see system that hinder scaling, by isolating theses jobs You can scale them individually and with solid data while leaving the client side system to be scale based on the real needs and not unnecessary waiting
Point-To-Point, this system is build of several components with distinct line of communication between them we start our system with Message Broker. we then create our producer. it will call by some external system, the RESTful message will come and start the process. Now Producer knows about the Message Broker and how to communicate with it. with in need some downstream action is take place and this is where the consumer comes in to play. its responsible for all of the activity that occurs once the message is produced. it also knows about the Message Broker, but in little bit different way. Usually consumer attached it to directly to the Message Broker and actively listen or polls the Message Broker for a new message.it then goes into a, qoute-unquote, idle state. until the next polling time triggers for message to be listen for
- Now the original service makes a call to the producer and blocks, waiting for response.
- The producer creates a message based on the contract for the system and puts the message on the Broker.
- At this point, the Producer can return its response to the original service, usually, with an accepted status. Now the message will move to the Message Broker and be put on the appropriate queue.
- The Message Broker will then present the message. The consumer receiving message which is now cleared from the message Broker one the act is received and its done.
The act time is determined by the consumer. It can wait to acknowledge the message until its done or it can do so immediately. It will then do its own processing and communication is now complete.
Considerations
- Wire time:
If the act of building the message, setting up the connection with the Message Broker, and sending the message takes as long or more just making the downstream call you may want to consider that in your design are you really saving time. Now, Because it is either a RESTful call or in a Message Broker call, setup and tear-down is often similar
- Extra Component:
Along same lines, building extra components isn’t always worth it and specially if the consumer is executing code that is not out of place for the producer’s main flow. if the total time to make the call is not worth it and the code fits in the producer you should take that in to consideration. If however you code’s out of place or the complexity of more components is easily replaced by the reduced processing time, this is the great tradeoff
- Failure scenario:
You must plan for a failure scenario. usually in this model you will build an adjoining dead letter queue. use that dead letter queue but more importantly don’t just ignore. Build in processing to inspect the DLQ regularly and ensure that you aren’t getting message there . Better yet alert when you do get them.
Publish-Subscribe
This actually becomes a very powerful pattern in distributed computing as we will discuss. we usually have single producer who creates the message and dispatch it to the message broker. in this model one or more subscribers who listen the message and act on their own accord
its send and forgot model where the producer isn’t waiting for any kind of response before continuing its work. if a subscriber isn’t there it wont get a message however if the subscriber is durable there is guarantee that the message will be delivered at some point once the subscriber available again. this is the specific registration process that allows this durable subscription
But it very critical when messages must be acted on even in the process have been terminated in the system
Some of the Use cases
- Multiple responders:
Especially in a distributed computing often time we see cases when the flow of traffic between sites is limited on purpose to enforce good constrains across the data centers we often allow data to stay in sync between sites what happen when you need more than data in sync
For instance you may allow your databases stay in sync across the data centers but other triggered actions are data centers independent. In pub-sub model you confederate the message broker across multiple data centers and allow triggered action with in each data center to occur an isolation those triggered actions are often admin tasks.Like triggered to clean up or update search
In this model the central hub can publish on message in a worker in each data center an subscribe to the message. The beauty of this model is a when a new data center comes online there is no code changes needed as you would see in direct push through in asynchronous process
As your new data centers just create the new consumer and subscribe to the channel to get the messages every other data centers gets
- Multiple Tasks:
For Example: We took on GDPR for the company design the heart of the right of GDPR around the pub-sub model. when a consumer request a copy of his or her data or asked to forgotten, we would publish a message to the message broker. each department with in listen for those messages and perform their own series of actions to handle the consumer’s request.
When they were done they would put a point-to-point message on the queue we will then wait until everyone else responded before moving on to the next step.
This pub-sub model allow the company to be compliant early on with a smaller subset of department and then roll new departments online as soon as we will ready again with no code changes in the pub-sub side we didn’t nothing we continued putting the message on the queue.
- Consumer Choice
The consumer can decide if they want to listen to messages or not. don’t have to call API’s have them act the message ignore it they simply don’t have to subscribe. there is you can create message that everyone or no one will listen to without any code changes if there more consumers they just subscribe
Durable Subscription
Their are cases when your consumer must get the message in a pub-sub model even if there are offline at the moment the message is produced these are the key benefits of using durable subscribers
- Always get the Message:
The guarantee that they get the message; the message broker will learn of the durable subscribers and keep the message around until that subscribers acknowledge it without impacting other subscribers. The message becomes targeted at that point without impacting duplicated processes of other subscribers that are online
- Powerful in mission critical operations:
All of our consumers were durable subscribers on purpose. we had legal obligation to process the data but did not want to block or retry. if the remote system was down for some reason. By employing this model we did not block we just allowed the message broker to handle the sending of these messages when the come back online
- Unregister if needed:
If the consumer no longer needs to consume the message the must unregister to ensure the message broker does not hang on the messages that are no longer needed, as opposed to just falling off
- Producer agnostic:
All of this is done in the producer agnostic fashion, the producer does not know anything about the consumers if done correctly. they don’t know if the consumer is durable or not they are completely decoupled
Publish-Subscribe model, we start our system with message broker, we then create a producer. it will called by some external system. the rest will message will come in and start the process. Now the producer knows about the message broker and how to communicate with it. we then will have some downstream actions that need to take place. for example we have to two different consumer processes called subscribers that need to act on the message. they can be durable or not. Each of these subscribers also knows about the message broker. Each one subscribes to the message and starts the pulling the process.
- The message brokers knows that currently there are two subscribers for the message. Now the original service makes a call to producer and once again blocks waiting for a response.
- The producer creates message based on the contract for the system and puts the message on the broker. once again, the producer can respond to the original service with accepted.
- Now the message will move to the message broker once again, the producer can respond to the original service with accepted.
- Now the message will move to the message broker and be put on the appropriate queue. However, it will present the message to all available subscribers.
- The subscribers during their own independent listen phase will receive the message in its original format usually.
- Each subscribers must act the message on its own or the message broker will still hold the message for the individual subscriber. At this point they can each do their own thing and execute the actions needed on the acceptance of the message