22 Jan 2020

Intro to sqs

Amazon Simple Queue Service (SQS) is a system for queuing messages, designed to be the communication bridge between the internal components of a web application. SQS quickly and reliably processes messages created by one component in an application (a producer) and holds them for processing in other places (a consumer). This approach allows application architecture to be more distributed, as all components can feed into--and take from--a global message queue without needing to maintain their own independent resources.

Without a distributed system, each producing component could otherwise have to maintain its own collection of consumers. If those consumers are over burdened, the system may have no way of tapping into under utilized resources associated with other producers. The global queue in a distributed approach brings tremendous efficiency into that scenario. It also brings speed and bandwidth. SQS queues sit on top of very capable AWS servers and the service can handle any volume weather it is 1 message or 1 million messages per second. And each message is stored redundantly on different servers, removing any single point of failure. As a result, the message communication is an app becomes more efficient, more reliable and more scalable.

SQS lets you turn this:

Into this:

A note on cost: SQS free tier covers the first 1 million messages. After that, it costs $0.40 for the next 1 million with a standard queue and $0.50 for FIFO queue. SNS pricing model.

Basic Terminology

Producers: Components of a system creating messages or requests.

Consumers: Components of a system processing requests.

Queue: A system for holding and directing the flow of messages within a distributed application.

Standard Queue: The default queue type in SQS. Provides unlimited throughput, but does not guarantee the order in which messages will be consumed.

FIFO Queue: An alternative queue type. With FIFO, message order is guaranteed to be first in, first out. Throughput can still be very high, but is limited to 3,000 messages per second. Messages are also guaranteed to only be processed once, whereas with a standard queue its occasionally possible for a message to delivered more than once.

Message Life Cycle: Once a message is sent to a queue it is visible for a designated period of time. When a consumer is ready to process that message.

Creating a Queue

Via Web Console

- On the SQS Console homepage, choose Create New Queue
- For this example, we will create a Standard Queue. Provide it a name and press Quick-Create Queue.
- Your queue will now be created. The bottom portion of the screen will show the details of the queue's configuration. Make note of the queue URL and ARN.

Via CLI

  aws sqs create-queue --queue-name

Sending Messages to a Queue

In SQS, a message can be up to 256 characters long and in addition to the message body can contain up to 10 structured message attributes. These can be string, binary, or number formatted.

In a normal implementation, most messages would be sent to a queue from users or from an independent source within the application. However, the SQS console does allow for individual messages to be crated through the console using the Send a Message command in the Queue Actions menu.
The Send a Message window will allow you to construct a message body along with any message attributes. Write a short test message and press "Send message".
When a message is sent, the queue will return a Message Identifier. The Message ID is a system assigned value that uniquely marks each message and is sent back as confirmation that it has been received.

Checking our Queue status

At this point there should be at least one message in the queue. This can be confirmed by looking at the count in the Messages Available column.
View the individual messages currently available in the queue by going to the Queue Actions menu and clicking the View/Delete Messages .
The list will initially be empty. You can populate it by pressing the Start Polling for Messages button. This will show messages from the messages at the front of the queue, ready to be processed by a consumer. What's actually happening is the SQS message temporarily consumes the messages, making them unavailable to other consumers while the polling is underway. Once the polling concludes, the messages are returned to the queue.

We can also delete messages from here by clicking the check mark and pressing the red "Delete Message" button in the lower right corner.
There is a More Details link next to every message. Pressing it will show the full message content including message body, attributes, Message ID and any attached metadata.

Note: In a real production environment there may be consumers reading each new message from the queue shortly after they arrive. However, the queue we created in this tutorial does not have any consumer applications. As a result, an messages sent to the queue so far will still be there when we poll for them.

Bonus Trivia: Launching back in 2004, SQS happens to be the first publicly available AWS service, predating even the AWS brand itself.

Additional SQS Concepts

Consuming Messages from a Queue / Message Polling

There are two mechanisms through which SQS will consume messages: Short Polling and Long Polling.

The purpose of Short Polling is to log the request for reading messages as quickly as possible. To achieve this, short polling takes somewhat of a shortcut. For large queues it may not be able to directly check all the servers, and typically checks just a few servers based on a weighted random distribution and returns messages only from those servers. The benefit is a near instantaneous read of what's in the queue. The drawback is a read request may not return any messages. Learn More

Long Polling, on the other hand, continues polling for a specified duration known as "Receive Message Wait Time". This value can be up to 20 seconds. The Long Polling call will continue checking message servers for that stated time before returning with a count or an empty response. Learn More

The "Receive Message Wait Time" can be set by going to Queue Actions > Configure Queue.

Visibility Timeout

Even after a consumer successfully reads a message from an SQS queue the message is not automatically deleted. Messages read by consumers remain in the queue. That opens the possibility for a message being picked up by multiple consumers. The solution in place to prevent this is the Visibility Timeout, which makes a message invisible for that specified period of time after a consumer has picked it up.

The Visibility Timeout period defaults to 30 seconds, but it is adjustable. The idea is to set it for a duration long enough to allow your consumer to fully process the message under all normal circumstances. And if the consumer is unable to process the message due to a failure or any error, it will become visible for other consumers to try processing it.

Because messages read by the queue are not automatically deleted, a consumer which fails to successfully process a message will not result in a lost message. The message will return to the queue safely. However, there may be good reason why the message failed. It could be that the message itself was corrupt, or was too large, or for any other reason is not fit to be processed by any consumer. You may not want it around forever, because if it were, it would potentially remain at the front of your queue, moving around like a hot potato from consumer to consumer, never being able to be processed successfully. In this circumstance you would want an unsuccessful message to be removed eventually. SQS has a feature just for this situation.

Dead Letter Queue

SQS allows you to configure a special type of queue to hold messages that cannot be consumed successfully by other queues. This is known as a Dead Letter Queue (DLQ), and it is a place to hold all your problematic messages.

Dead Letter Queues allow you to specify a value for Maximum Receives. Any message which exceeds that value is automatically transferred to the DLQ.

Maximum Receives and other Dead Letter Queue settings can be enabled and configured in Queue Actions > Configure Queue.

Once a DLQ is enabled, an easy way to test the dead letter queue is to go to the Queue Actions Menu and use the "View/delete messages" command to view a message enough times to exceed the Maximum Receives value.

SQS Best Practices

Use a DLQ

DLQs can keep your queue healthy and provide a way to isolate messages that can't be processed successfully, which you can later analyze.

Protect Queues with Server-Side Encryption (SSE)

SQS provides the ability to encrypt the content of your messages using AWS KMS service. This can be enhanced further by selecting AWS KMS Customer Master Key (CMK) to generate data keys required for the encryption/decryption process of SQS messages. Both settings can be found in: "Queue Actions" > "Configure Queue"

Monitor the number of visible messages in the queue

Within your application, develop an understanding for the normal message flow. Set a threshold for which the count of Messages Available (Visible) should not exceed. This tells you how many unprocessed messages you have. If you have too many unpressed messages it means you have a backlog of messages that are not consumed. This means you don't have enough consumers or the current batch of messages are unable to be processed.

SQS In the Real world

SQS is a low cost alternative to database storage for high volume of requests that could otherwise overwhelm a system. But SQS uses polling rather than full and through counts of messages. This is less precise, and in some cases, less ideal than the equivalent database operations. Its advantage however is in cost and scalability.

SQS is often useful in applications dealing with extremely high incoming message volume, as the number of messages per second in SQS is virtually unlimited.

If a mobile app is peppering for a huge influx of 100 million new users as they launch in a new region. The company could provision the necessary fleet of servers to keep up with the influx of users, but it would be expensive. And if they over estimate the traffic, it results in wasteful spending. Under estimating the traffic would tax the systems.

SQS would automatically scale to the required 100 million volume with no problem. And it would not require the company to predict traffic beforehand.

In that scenario SQS would be the ideal queuing mechanism to decouple the incoming message queue from the rest of the application.