From Amazon's SQS FAQ:
Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components.
You can scale the amount of traffic you send to Amazon SQS up or down without any configuration.
Amazon SQS is FIFO (first-in-first-out) queues and preserves the exact order in which messages are sent and received. "Standard" SQS attempts to preserve the order of messages. However, receiving messages in the exact order they are sent is not guaranteed. FIFO has limited throughput: 300 msg/s without batching, 3000 msg/s with. FIFO sends message exactly once.
FIFO queue names must end in .fifo
Read more on Amazon.
Basic walk thru setting up a simple SQS in the console.
-- Goto the SQS console.aws.amazon.com/sqs/v2/home.
-- Click 'Create Queue'.
-- Leave 'Standard' selected.
-- Type a name into the name field. Set any configuration options.
Visibility timeout: Immediately after a message is received, it remains in the queue. To prevent other consumers from processing the message again, Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consumers from receiving and processing the message. The default visibility timeout for a message is 30 seconds. The minimum is 0 seconds. The maximum is 12 hours. More
Delivery delay: Delay queues let you postpone the delivery of new messages to consumers for a number of seconds, for example, when your consumer application needs additional time to process messages. If you create a delay queue, any messages that you send to the queue remain invisible to consumers for the duration of the delay period. More
Receive message wait time: When this is set to 0 seconds, then this is called as short polling. This means that AWS will keep on polling the queues to see if messages are present. There is also the concept of long polling wherein you don’t need to poll the queue every time. For this you need to set the value of this parameter to a value greater than 0 and less or equal to 20.
Long polling decreases the number of api calls made to SQS while increasing the efficiency and latency of your application. Long polling can be enabled at the queue level or at the api level using WaitTimeSeconds. More
Message Retention period: anything message older than this setting is deleted.
Maximum message size: self- explanatory. You can use SQS Extended Client to make use of larger file sizes. These larger files will reside in S3.
-- Access policy
Default is the the queue owner can send/receive messages, but you can specify AWS accounts, IAM users and roles.
-- Optional settings
Redrive and Dead letter queue, Encryption, and Tags.
-- Click on 'Create queue'.
Test out message. On the next screen, top right, click on Send and Receive Messages. Enter a Message body and hit enter. Bottom right, hit Poll for messages. View the message. To keep further polling and retrieval from happening, delete the message. You can click Purge in the top right to delete all messages.
More on access policy
Specify the Principal with an aws account id to allow other AWS accounts to access your SQS.
{
"Version": "2012-10-17",
"Id": "Queue1_Policy_UUID",
"Statement": [{
"Sid":"Queue1_SendMessage",
"Effect": "Allow",
"Principal": {
"AWS": [
"111122223333"
]
},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-2:444455556666:queue1"
}]
}If you want an S3 bucket to trigger a message in SQS, you'll need to give the bucket permission in the SQS access policy. Update the policy with your account and bucket info and save it.
{
"Version": "2012-10-17",
"Id": "example-ID",
"Statement": [
{
"Sid": "example-statement-ID",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "SQS:SendMessage",
"Resource": "arn:aws:sqs:us-east-1:111122223333:sqs-s3-same-account",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "123456789"
},
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:hellobucket"
}
}
}
]
}{
"Version": "2012-10-17",
"Id": "example-ID",
"Statement": [
{
"Sid": "example-statement-ID",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": [
"SQS:SendMessage"
],
"Resource": "SQS-queue-ARN",
"Condition": {
"ArnLike": {
"aws:SourceArn": "arn:aws:s3:*:*:awsexamplebucket1"
},
"StringEquals": {
"aws:SourceAccount": "bucket-owner-account-id"
}
}
}
]
}More examples. Another. Receive S3 event notifications to an Amazon SQS queue that uses SSE.
Set up your S3 bucket to use SQS
Go to your bucket. Click on properties and scroll down to Event notifications.
Click on Create notification. Enter options. Specify at least one event type. For Destination, select SQS queue. In the drop down labeled, SQS queue, select the SQS function you created and save changes.
Change Message Visibility by the consumer
If the consumer needs more than the alloted visibility timeout, it should call ChangeMessageVisibility api to extend time. More.
Dead Letter Queue
DLQ's are helpful for debugging. You can create a SQS and use it as a consumer of DLQs. Copy the arn of the SQS you created for use as a DLQ and in the SQS that you want to use as a producer of DLQs, select Enabled under the Dead-letter-queue heading and enter the arn of the DLQ you copied previously.
If a consumer fails to process a message within the Visibiilty Timeout, it goes back to the queue. However, a threshhold, MaximumReceives, can be set of how many times a message can go back to the queue before it is sent into a DLQ.
Redrive
A feature to help consume messages in the DLQ to understand what is wrong with them. With it, manual inspection and debugging can be done and the message can be resent and received by the consumer. More
SQS API
CreateQueue, DeleteQueue. MessageRetentionPeriod can be used with CreateQueue.
PurgeQueue
SendMessage (DelaySeconds), ReceiveMessage, DeleteMessage
MaxNumberOfMessages: default 1, max 10 (for ReceiveMessage api)
ReceiveMessageWaitTimeSeconds
ChangeMessageVisibility: change the message timeout
Batch APIs help decrease cost.
Advanced FIFO
While sending message from FIFO queue found that when content based deduplication is enabled , it is the combination of deduplicationId + content that works as a common denominator not just the content .
Content-based deduplication instructs Amazon SQS to use a SHA-256 hash to generate the message deduplication ID using the body of the message—but not the attributes of the message.
Each message is delivered once and remains available until a consumer processes it and deletes it. Duplicates are not introduced into the FIFO queue.
The tag that specifies that a message belongs to a specific message group.
When content-based deduplication is enabled, the message deduplication ID is optional.
