DP-201 Exam - Question 166

Question

HOTSPOT -Which Azure service and feature should you recommend using to manage the transient data for Data Lake Storage? To answer, select the appropriate options in the answer area. NOTE: Each correct selection is worth one point. Hot Area:.

Examice · Accepted Answer

Scenario: Stage inventory data in Azure Data Lake Storage Gen2 before loading the data into the analytical data store. Litware wants to remove transient data from Data Lake Storage once the data is no longer in use. Files that have a modified date that is older than 14 days must be removed.

Service: Azure Data Factory -

Clean up files by built-in delete activity in Azure Data Factory (ADF).

ADF built-in delete activity, which can be part of your ETL workflow to deletes undesired files without writing code. You can use ADF to delete folder or files from

Azure Blob Storage, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, File System, FTP Server, sFTP Server, and Amazon S3.

You can delete expired files only rather than deleting all the files in one folder. For example, you may want to only delete the files which were last modified more than 13 days ago.

Feature: Delete Activity -

Reference:

https://azure.microsoft.com/sv-se/blog/clean-up-files-by-built-in-delete-activity-in-azure-data-factory/

Design data processing solutions

AhmedReda · Answer

The question asked to remove files older than 14 days which i think ADF & Delete could not do it, so the answer might be = (1) Azure Storage (2) Lifecycle management rule

bansal_vikrant · Answer

The files are stored in ADLS Gen2 which supports Life cycle management rules

syu31svc · Answer

https://azure.microsoft.com/en-us/updates/lifecycle-management-for-azure-data-lake-storage-is-now-generally-available/
Azure storage and lifecycle management rule are the answers

dcpavelescu · Answer

Answer shall be: Azure Storage & Lifecycle management

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-lifecycle-management-concepts?tabs=azure-portal

kempstonjoystick · Answer

Would lifecycle management not fulfill this as well?

BungyTex · Answer

I just tested this in my ADL Gen 2, can set a rule to delete files last modifed more than 14 days ago.

davita8 · Answer

Azure storage
 lifecycle management

mohowzeh · Answer

Seems to me that there are two valid combinations: (Azure Data Factory, delete activity) and (Azure storage, Lifecycle management)

felmasri · Answer

Azure Data Lake Storage lifecycle management is now generally available
https://azure.microsoft.com/en-us/updates/lifecycle-management-for-azure-data-lake-storage-is-now-generally-available/

runningman · Answer

"Litware wants to remove transient data from Data
Lake Storage once the data is no longer in use."  wouldn't that make B Azure Storage a better answer?

Ash666 · Answer

ADLS GEN 2 doesn’t yet support delete blob action
https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues

NikP · Answer

Now, Lifecycle management is supported for accounts that have a hierarchical namespace for General-purpose V2.
With this, you can reduce the delete activity (less cost even it is negligible for a pipeline).
However, I would prefer to use delete activity in ADF to make sure that they got deleted after I load them to database. Better than auto delete through lifecycle.
For me, given answer is correct based on requirement.

ThijsN · Answer

Both ADF with delete or storage with lifecyle will work. I literally build the last one this week. I think that is the best solution as this is the cheapest and easiest. Doesn't cost anything to run, to build or to maintain.

NasRim · Answer

lifecycle management is available in ADLS from July 31, 2020
https://azure.microsoft.com/en-us/updates/lifecycle-management-for-azure-data-lake-storage-is-now-generally-available/

Needium · Answer

The prefered option should be Az Storage and life cycle management rule

kn_shn · Answer

From older comments, ADF + Delete and Azure Storage + Lifecycle management rule seem to have similar functionality to remove files. However there is a difference: Liftcycle is defined based on the creation of the file, and in this question and context, it says:" Files that have a modified date that is older than 14 days must be removed". i.e. the file removal is based on the modified date. As BungyTex confirmed below, ADF + Delete can achieve this objective and the answer is correct.

PhuVu · Answer

ADLS still not support lifecycle management. so that answer is correct
https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues

KasiaK · Answer

Lifecycle management policies (delete blob):	Generally available in Premium,	Generally available	 in Standard  https://docs.microsoft.com/pl-pl/azure/storage/blobs/data-lake-storage-supported-blob-storage-features

lky17 · Answer

The correct answer should be Az Store and Lifecycle ... because ADLSG2 lets delete any file, the unique exception is "If you use the Delete Blob API to delete a directory, that directory will be deleted only if it's empty. This means that you can't use the Blob API delete directories recursively." and support all operation in lifecycle management except "Lifecycle management policies with premium tier for Azure Data Lake Storage.
You can't move data that's stored in the premium tier between hot, cool, and archive tiers. However, you can copy data from the premium tier to the hot access tier in a different account."

Ref https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues

savin · Answer

Azure storage
lifecycle management should be easier option

Dymize · Answer

The way i see this, if the inventory data is coming from a microsoft SQL server, it is being ingested by ADF and not in  Azure Storage, and if using ADF then the delete activity should be used. As per other comments this is proven to work

hoangton · Answer

Given answer is correct
(1)ADF
(2)Delete activity

DP-201 Exam - Question 166

Discussion