A company uses Apache Airflow to orchestrate the company's current on-premises data pipelines. The company runs SQL data quality check tasks as part of the pipelines. The company wants to migrate the pipelines to AWS and to use AWS managed services.
Which solution will meet these requirements with the LEAST amount of refactoring?
Migrating the existing Airflow orchestration configuration into Amazon Managed Workflows for Apache Airflow (Amazon MWAA) will meet the requirements with the least amount of refactoring. Amazon MWAA is a fully managed service for running Apache Airflow, making it easy to transfer the existing workflow orchestration with minimal changes. Data quality checks can continue to be implemented as SQL tasks within the Airflow environment, similar to the current on-premises setup. This approach allows the company to leverage AWS managed services while maintaining their current workflow, ensuring a smooth migration process.
Amazon MWAA is a managed service for running Apache Airflow. It allows migrating existing Airflow configurations with minimal changes. Data quality checks can continue to be implemented as SQL tasks in Airflow, similar to the current setup.
Amazon MWAA - becuase we already uses Apache Airflow
The solution that will meet these requirements with the least amount of refactoring is Option C: Migrate the existing Airflow orchestration configuration into Amazon Managed Workflows for Apache Airflow (Amazon MWAA). Create the data quality checks during the ingestion to validate the data quality by using SQL tasks in Airflow. Amazon Managed Workflows for Apache Airflow (MWAA) is a fully managed service that makes it easy to run open-source versions of Apache Airflow on AWS. It allows you to build workflows to design and visualize pipelines, automate complex tasks, and monitor executions. Since the company is already using Apache Airflow for orchestration, migrating to Amazon MWAA would require minimal refactoring.