To migrate 70 TB of video files from on-premises storage to Amazon S3 while using the least possible network bandwidth, using AWS Snowball Edge is the most suitable option. AWS Snowball Edge is a physical data transport solution that can transfer large amounts of data securely and quickly without utilizing internet bandwidth. The process involves receiving the Snowball Edge device at the premises, transferring the data to the device using the Snowball Edge client, and then shipping the device back to AWS, where the data is imported into Amazon S3. This approach minimizes network bandwidth usage and ensures a rapid transfer given the volume of data.
The solution that best meets the company's requirements of ingesting and quickly consuming up to 100,000 messages each second while decoupling the solution and increasing scalability is to publish messages to an Amazon Simple Notification Service (Amazon SNS) topic with multiple Amazon Simple Queue Service (Amazon SQS) subscriptions. This configuration provides a scalable and decoupled architecture as Amazon SNS can handle high message throughput and enables multiple consumer applications to process the messages in parallel through SQS. This approach ensures that the system can handle drastic variations in message volume efficiently.
To address the variable workloads and maximize both resiliency and scalability, the best approach involves using Amazon Simple Queue Service (Amazon SQS) as a buffer for the jobs. This decouples the primary server from the compute nodes, making the system more resilient. By implementing the compute nodes with EC2 instances managed by an Auto Scaling group and configuring scaling based on the size of the SQS queue, the system can dynamically adjust the number of instances according to the workload. This ensures that the infrastructure scales out to handle peak loads and scales in during periods of lower activity, providing cost efficiency and enhancing responsiveness to workload changes.
Create an Amazon S3 File Gateway to extend the company's storage space. Create an S3 Lifecycle policy to transition the data to S3 Glacier Deep Archive after 7 days. This solution provides seamless integration with existing SMB file servers and allows for the extension of storage capacity using Amazon S3. Additionally, the S3 Lifecycle policy ensures that data is automatically transitioned to a cost-effective storage class after 7 days, managing storage efficiently while maintaining low-latency access for recently accessed files due to the gateway's local caching capabilities.
To ensure that orders are processed in the order they are received, the company should use an Amazon Simple Queue Service (SQS) FIFO (First-In-First-Out) queue. SQS FIFO queues are specifically designed to guarantee the order of message processing and exactly-once delivery. By integrating the API Gateway to send messages to an SQS FIFO queue and configuring the queue to invoke an AWS Lambda function for processing, the company can ensure that order processing maintains the sequence in which the orders were received.