A company maintains multiple extract, transform, and load (ETL) workflows that ingest data from the company's operational databases into an Amazon S3 based data lake. The ETL workflows use AWS Glue and Amazon EMR to process data.
The company wants to improve the existing architecture to provide automated orchestration and to require minimal manual effort.
Which solution will meet these requirements with the LEAST operational overhead?
AWS Step Functions tasks are ideal for orchestrating multiple AWS services, including AWS Glue and Amazon EMR. Step Functions offer a state machine-based workflow orchestration service that can sequence various tasks while handling errors and retries, thus reducing operational overhead. This service integrates directly with AWS Glue to manage ETL tasks and can initiate workloads on Amazon EMR, ensuring streamlined and automated workflows with minimal manual effort.
Glue Workflow only orchestrate crawlers and glue jobs
For me it's B because I did not found a possibility how Glue can trigger/orchestrate EMR processes OOTB. But with StepFunction there is a way: https://aws.amazon.com/blogs/big-data/orchestrate-amazon-emr-serverless-jobs-with-aws-step-functions/
Since it seems to me that this pipeline is complex, with multiple workflows, I would go for Glue workflows.
orchestrating = step function
Orchestrating different AWS services is a typical use case for Step Functions: https://docs.aws.amazon.com/step-functions/latest/dg/connect-emr.html https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html
There is no way for Glue Workflow to trigger EMR
Option A, AWS Glue Workflows, seems to be the best solution to meet the requirements with the least operational overhead. It offers a seamless integration with the company's existing AWS Glue and Amazon EMR setup, providing a managed and straightforward way to orchestrate their ETL workflows without extensive additional setup or manual intervention.
Can you provide an example of Glue initiating an EMR job? Or somewhere in the documents? AFAIK, Glue workflows are only to be used for Glue related things e.g. pull data, transform it, and store it somewhere else (ETL). Executing commands on behalf of other services can be done using boto in glue, but it feels weird using Glue like that when you have step functions which are designed for orchestrating different services.
Yo me voy por la D) Amazon MWAA porque Glue Workflows solo admite Jobs de Glue y Step Function puede fucionar pero no son workflows de datos. Amazon MWAA son workflows de datos y esta integrado tanto con Glue como EMR: https://aws.amazon.com/blogs/big-data/simplify-aws-glue-job-orchestration-and-monitoring-with-amazon-mwaa/
EMR in workflows , i dont think so
Glue Work flows
Here's an example of how you can use AWS Glue to initiate an EMR (Elastic MapReduce) job: Let's assume you have an AWS Glue job that performs ETL tasks on data stored in Amazon S3. You want to leverage EMR for a specific task within this job, such as running a complex Spark job. 1. Define a Glue Job: Create an AWS Glue job using the AWS Glue console, SDK, or CLI. Define the input and output data sources, as well as the transformations you want to apply. 2. Incorporate EMR Step: Within the Glue job script, include a section where you define an EMR step. An EMR step is a unit of work that performs a specific task on an EMR cluster. Code follows in the next entry...
https://aws.amazon.com/blogs/big-data/orchestrate-an-etl-pipeline-using-aws-glue-workflows-triggers-and-crawlers-with-custom-classifiers/
We have both Glue job and EMR job, so we need Step Functions to connect those. Airflow can do it, but required more dev work.
Orchestrating different AWS services is a typical use case for Step Functions: https://docs.aws.amazon.com/step-functions/latest/dg/connect-emr.html https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html
B - because AWS Glue can't trigger EMR
AWS Glue Workflows are specifically designed for orchestrating ETL jobs in AWS Glue. They allow you to define and manage complex workflows that include multiple jobs and triggers, all within the AWS Glue environment.Integration: AWS Glue workflows seamlessly integrate with other AWS Glue components, making it easier to manage ETL processes without the need for external orchestration tools.Minimal Operational Overhead: Since AWS Glue is a fully managed service, using Glue workflows will reduce the operational overhead compared to managing separate orchestrators or building custom solutions.While D. Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is also a good choice for more complex orchestration, it may involve more management overhead compared to the more straightforward AWS Glue workflows. Thus, AWS Glue workflows provide the least operational overhead given the context of this scenario.
Glue workflows are managed services and best for considering least operational overhead.
Answer A, Glue workflows
glue workflows is part of the glue ecosystem so its provides seamless integration with minimal changes
it is interesting to choose A for minimum effort but only step functions can trigger the work both on EMR and on GLUE jobs
The company wants to improve the existing architecture so A cannot be the right choice
Why Not the Other Options? B. AWS Step Functions More flexible but requires manual setup of states and transitions for Glue & EMR. Higher operational overhead than Glue Workflows. C. AWS Lambda Lambda is not ideal for long-running ETL workflows. Best suited for lightweight data transformations or event-driven tasks. D. Amazon MWAA (Apache Airflow) More control but requires cluster management and custom DAGs. Higher maintenance than Glue Workflows.