Introduction to Amazon Simple Queue Service (SQS)
The beauty of electronic mail (or plain old snail mail for that matter) is that as the sender, you didn’t need to bother yourself with the the details of how your message reaches it’s destination. After composing the message, all you need to do is just to hand it off to your trusted mailman who’s entire job is to handle the logistics and challenges of mail delivery for you.
This mechanism offers such level of abstraction, that you as a sender are not only not responsible for the actual delivery of the message - you don’t even need (or want) to know how it got to it’s destination (by plane, by truck, by carriage etc.).
Message Queuing (MQ) takes this same concept of relieving the sender from the responsibility of actually delivering the message to it’s destination by offloading it to a dedicated component that specializes in doing exactly that - reliably delivering messages to their destination. It is also the technological cornerstone of SOA (Service Oriented Architecture), but that’s a story for another post.
My applications already communicate just fine, what’s in it for me?
TCP/IP Sockets are natively available in every programming language, and using them in applications is both common practice and common knowledge.
With both the technology and the know-how already in place - it is so very tempting to just use it. After all, why bother to invest in bring something new into the mix (together with the associated overhead of skills and other costs) when we already have a viable way of doing business “for free”?
But, just like many other things in life - “free” often times comes with a hidden price tag:
- It places the burden of validating successful delivery (network-wise as well as logically in an application-specific manner) on the sending applications base code, requiring greater programmatic effort to successfully implement the program as a whole.
- More code = more bugs. Plain and simple.
That’s not to say that an “off the shelf” middleware is bug-free, but leveraging existing and proven technologies is always safer and easier than writing and maintaining custom code produced in-house, period. - Sockets are opened between two computers: meaning that transferring a message to multiple recipients requires additional coding effort and more importantly - requires the sender to know where to send the message - a requirement that doesn’t always co-exists well with the dynamic nature of Cloud Computing.
- Sockets are immediate - meaning that in order to pass on a given message the recipient must be ready and willing to receive it right now.
If the recipient suddenly goes offline or simply unable to immediately process the messages (like an electronic trading system outside trade hours) it is the responsibility of the application to take necessary actions such as buffering the messages until the right time to process them.
Message Queuing is designed to address all of the above and much more - it is a proven concept widely used in the most demanding environments and enjoys numerous implementations such as the stand-alone IBM Websphere MQ, Microsoft Message Queuing (MSMQ) which is bundled with Windows Server, as well as open source implementations like Apache Active MQ.
So how does it work?
Message Queuing revolves around the concept of Queue: a logical FIFO list of messages that share the same logical destination (such as incoming messages from clients to application X, or outgoing messages from application X to application Y). In essence, the Queue’s logical function as a pipeline, enabling messages (read information) to freely and easily flow from one application to another.
Note that a Queue doesn’t explicitly specify servers, as it is a detached and independent entity from any single server - more on that in just a bit.
All we need to do from our application(s) is to invoke the Queue’s “send” method in the sending application (called “Producer Application” in SOA terminology) and similarly invoke the “Receive” method in the receiving application (again, in SOA terminology this application will be called “Consumer Application”) - and that’s literally all there is to it (application-wise).
This two-step process in which one application’s send followed by other application’s receive has two very important properties:
- It is asynchronous: meaning that any period of time can pass between the send and the receive - in which time the message will exist inside of the queue, waiting for delivery.
This is a very convenient and “free” way of implementing resiliency into our applications as messages aren’t lost when the receiver is down, only queued until the receiver finally does return to normal operation (think of messages in your mailbox while you are away for vacation or sick-leave). - It is also loosely coupled: meaning that the sender does have to be specifically address the messages to any particular recipient - instead they could be independently fetched by the receiver, or load balanced across multiple receivers (think of call-centers, where your call is placed in a queue and will be answered by the first representative who becomes available).
Together, these two properties are well tailored to Cloud-Computing environments where there is a constantly shifting number of senders and receivers that using this method don’t need to actually know of each other - rather they just pivot around this one central Queue who handles both the load balancing and fault tolerance.
So how does AWS do Message Queuing?
Similarly to S3, SQS is an independent HTTP-based service happily living in the AWS cloud, not attached to any instance in particular.
True to it’s name SQS is very simple to set-up and use (at the expense of being feature-poor compared to other MQ implementations), and can be configured and used by any Internet connected computer (inside and outside of EC2).
Below is a working example, taken from the SQS API Reference for sending a message to a Queue via invoking the following HTTP URL:
http://queue.amazonaws.com/123456789012/testQueue/
?Action=SendMessage
&MessageBody=This+is+a+test+message
&Version=2009-02-01
&SignatureMethod=HmacSHA256
&Expires=2009-04-18T22%3A52%3A43PST
&AWSAccessKeyId=0GS7553JW74RRM612K02EXAMPLE
&SignatureVersion=2
&Signature=Dqlp3Sd6ljTUA9Uf6SGtEExwUQEXAMPLE
Similarly, in order to receive my message from from my queue will invoke the following HTTP URL:
http://queue.amazonaws.com/123456789012/testQueue/
?Action=ReceiveMessage
&MaxNumberOfMessages=5
&VisibilityTimeout=15
&AttributeName=All;
&Version=2009-02-01
&SignatureMethod=HmacSHA256
&Expires=2009-04-18T22%3A52%3A43PST
&AWSAccessKeyId=0GS7553JW74RRM612K02EXAMPLE
&SignatureVersion=2
&Signature=Dqlp3Sd6ljTUA9Uf6SGtEExwUQEXAMPLE
And that’s all there is to it, no servers needed to support the Queue and no dedicated client needed needed to tap into it.
Simple.
Conclusion and Further Reading.
Message Queuing is a proven technological concept that is heavily leveraged both independently and as the foundation of larger concepts (such as SOA, or J2EE) to great success. Oftentimes, Message Queuing is the perfect bridge in the gap between the need to keep applications simple (read: “Single Minded”) and yet sophisticated enough to keep track of what’s going on in highly dynamic Cloud-Computing environments.
Stay tuned, in future posts we will drill down the pros, and cons, of SQS.