Certified Data Engineer Professional Exam

Exam Certified Data Engineer Professional All QuestionsBrowse all questions from this exam

Question 113

A Delta Lake table in the Lakehouse named customer_churn_params is used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources.

Immediately after each update succeeds, the data engineering team would like to determine the difference between the new version and the previous version of the table.

Given the current implementation, which method can be used?

Execute a query to calculate the difference between the new version and the previous version using Delta Lake’s built-in versioning and lime travel functionality.

Parse the Delta Lake transaction log to identify all newly written data files.

Parse the Spark event logs to identify those rows that were updated, inserted, or deleted.

Execute DESCRIBE HISTORY customer_churn_params to obtain the full operation metrics for the update, including a log of all records that have been added or modified.

Use Delta Lake’s change data feed to identify those records that have been updated, inserted, or deleted.

Correct Answer: A

Delta Lake’s built-in versioning and time travel functionality allows you to query the state of the data at different points in time. By executing a query that compares the state of the table between two different versions, you can accurately determine the difference between the new version and the previous version. This method directly leverages Delta Lake's capabilities for maintaining historical data versions and simplifies the process of identifying changes made during the update.

Discussion

Deb9753Option: E

Answer : E

c00ccb7Option: A

Answer A