A solutions architect is designing a payment processing application that runs on AWS Lambda in private subnets across multiple Availability Zones. The application uses multiple Lambda functions and processes millions of transactions each day.
The architecture must ensure that the application does not process duplicate payments.
Which solution will meet these requirements?
To ensure the application does not process duplicate payments, the solution must involve exactly-once processing. The Amazon Simple Queue Service (SQS) FIFO queue offers this capability by ensuring that each message is delivered once and is not duplicated in the queue. Additionally, FIFO queues maintain the order of processing, which aligns with the requirement of processing millions of transactions without duplication. Publishing due payments to an SQS FIFO queue and having another Lambda function poll the FIFO queue for processing ensures the application meets the requirement for exact, non-duplicate transaction processing.
Standard queues provide at-least-once delivery, which means that each message is delivered at least once. FIFO queues provide exactly-once processing , which means that each message is delivered once and remains available until a consumer processes it and deletes it. Duplicates are not introduced into the queue. OPTION C
Option D DynamoDB Streams helps ensure the following: Each stream record appears exactly once in the stream. For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item. DynamoDB Streams writes stream records in near-real time so that you can build applications that consume these streams and take action based on the contents.
Option D...If you need to handle millions of transactions each day, you might need to consider other approach instead of SQS FIFO. And amongst the given options, we have DynmamoDB that maintains order in the streams.
I'm not sure if the answer is DynamoDB as well, but answering your question, SQS Fifo can handle 300 messages/second without batching, 3,000 messages/second with batching. Assuming we're using the 300/sec option, with 86,400 seconds in a day, that gives you 25,920,000 messages, so in short, yes SQS can handle millions of requests each day. Not to mention DynamoDB doesn't provide the exactly-once processing the SQS offer and clearly requested in the question. That's just my train of thought, I'm happy to be corrected.
This calculation limits the number of transactions to 25 million a day. What if there are transactions exceeding this limit? As question say .... millions of transactions a day; that could be 70,80 or 90 millions also. In that case how SQS FIFO would perform? Happy to be corrected with more convincing facts
Dynamodb streams with partition key can be used to implement exactly once processing. There are many options with dynamodb to check for already processed item, and can be filtered out so that they are processed only once.
This calculation limits the number of transactions to 25 million a day. What if there are transactions exceeding this limit? As question say .... millions of transactions a day; that could be 70,80 or 90 millions also. In that case how SQS FIFO would perform? Happy to be corrected with more convincing facts
Dynamodb streams with partition key can be used to implement exactly once processing. There are many options with dynamodb to check for already processed item, and can be filtered out so that they are processed only once.
I'm not sure if the answer is DynamoDB as well, but answering your question, SQS Fifo can handle 300 messages/second without batching, 3,000 messages/second with batching. Assuming we're using the 300/sec option, with 86,400 seconds in a day, that gives you 25,920,000 messages, so in short, yes SQS can handle millions of requests each day. Not to mention DynamoDB doesn't provide the exactly-once processing the SQS offer and clearly requested in the question. That's just my train of thought, I'm happy to be corrected.
This calculation limits the number of transactions to 25 million a day. What if there are transactions exceeding this limit? As question say .... millions of transactions a day; that could be 70,80 or 90 millions also. In that case how SQS FIFO would perform? Happy to be corrected with more convincing facts
Dynamodb streams with partition key can be used to implement exactly once processing. There are many options with dynamodb to check for already processed item, and can be filtered out so that they are processed only once.
This calculation limits the number of transactions to 25 million a day. What if there are transactions exceeding this limit? As question say .... millions of transactions a day; that could be 70,80 or 90 millions also. In that case how SQS FIFO would perform? Happy to be corrected with more convincing facts
Dynamodb streams with partition key can be used to implement exactly once processing. There are many options with dynamodb to check for already processed item, and can be filtered out so that they are processed only once.
It seems that DynamoDB Streams deduplication is not guaranteed - "A Lambda consumer for a DynamoDB stream doesn't guarantee exactly once delivery and may lead to occasional duplicates. Make sure your Lambda function code is idempotent to prevent unexpected issues from arising because of duplicate processing." https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.Lambda.BestPracticesWithDynamoDB.html
"The application does not process duplicate payments" is the key point, which leads us directly to SQS FIFO
Option C Fifo
C over D, because https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html Processing dynamo streams with lambda can cause duplication. SQS FIFO can be configured for High Throughput to exceed the 3000/s (batched) limit https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/high-throughput-fifo.html I previously worked with payments and would argue that either option doesn't fully solve duplications. Events might be sent multiple times from source, you definitely want to perform de-duplication and have some sort of idempotent processing for them, instead of just blindly processing each thing you're given.
AnswerC SQS FIFO was created for such tasks Unlike standard queues, FIFO queues don't introduce duplicate messages. FIFO queues help you avoid sending duplicates to a queue. If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn't introduce any duplicates into the queue. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html
Option c
SQS can have duplicate messages in case of problems with the timeout window.
Option C: FIFO queues Exactly-Once Processing – A message is delivered once and remains available until a consumer processes and deletes it. Duplicates aren't introduced into the queue. First-In-First-Out Delivery – The order in which messages are sent and received is strictly preserved.
c is correct
Option B