Certified Data Engineer Professional Exam QuestionsBrowse all questions from this exam

Certified Data Engineer Professional Exam - Question 15

A table in the Lakehouse named customer_churn_params is used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources.

The churn prediction model used by the ML team is fairly stable in production. The team is only interested in making predictions on records that have changed in the past 24 hours.

Which approach would simplify the identification of these changed records?

Apply the churn model to all rows in the customer_churn_params table, but implement logic to perform an upsert into the predictions table that ignores rows where predictions have not changed.

Convert the batch job to a Structured Streaming job using the complete output mode; configure a Structured Streaming job to read from the customer_churn_params table and incrementally predict against the churn model.

Calculate the difference between the previous model predictions and the current customer_churn_params on a key identifying unique customers before making new predictions; only make predictions on those customers not in the previous predictions.

Modify the overwrite logic to include a field populated by calling spark.sql.functions.current_timestamp() as data are being written; use this field to identify records written on a particular date.

Replace the current overwrite logic with a merge statement to modify only those records that have changed; write logic to make predictions on the changed records identified by the change data feed.

Show Answer

Correct Answer: E

To simplify the identification of changed records in the Lakehouse table customer_churn_params, it is most effective to replace the current overwrite logic with a merge statement. This way, only the records that have changed will be modified. The change data feed can then identify which records have been altered, allowing the machine learning team to make predictions only on those records. This approach minimizes unnecessary computations and processes only the relevant data, ensuring efficiency.

Discussion

5 comments

EertyyOption: E

Aug 27, 2023

E is right answer

sturcuOption: E

Oct 16, 2023

E is Correct

leopedroso1Option: E

Feb 18, 2024

E is the correct one. By removing overwrite with merge, this will lead to an UPSERT causing updating only the data needed ( When Matched Upate + When not mached insert clauses). Then, with the CDC the capability of identifying is also satisfied.

kz_dataOption: E

Jan 10, 2024

E is correct

AziLaOption: E

Jan 21, 2024

correct ans is E