A junior data engineer seeks to leverage Delta Lake's Change Data Feed functionality to create a Type 1 table representing all of the values that have ever been valid for all rows in a bronze table created with the property delta.enableChangeDataFeed = true. They plan to execute the following code as a daily job:
Which statement describes the execution and results of running the above query multiple times?
The provided code uses Delta Lake's Change Data Feed (CDF) functionality to read and filter changes from the beginning (startingVersion, 0) each time it runs. It filters for update postimage and insert changes only and appends the filtered results to the target table. Due to starting from version 0 every time, the entire history of inserted or updated records will be read and appended to the target table on each execution, leading to many duplicate entries.
Answer is A, since the df is filtering on updated records using update_postimage filter
it's B: Reading table’s changes, captured by CDF, using spark.read means that you are reading them as a static source. So, each time you run the query, all table’s changes (starting from the specified startingVersion) will be read.
there is also insert in the filter.
sorry, answer is correct B.
B is the right answer, sorry.
B is correct
B is correct
("startingVersion", 0) that means the entiry history of table will be read so B.
answer is A, because there is a filter as asmayassineg said. Filter filters only existing records from change feed
correct
why is it Not E. It gets newly inserted or updated records
I'm with you, follow the reference: https://docs.delta.io/latest/delta-change-data-feed.html#:~:text=_change_type,update_preimage%20%2C%20update_postimage
Notice .option ("startingVersion", 0), which will bring all changes from begining. Hence Answer is B.
Considering that we are talking about Change Data Feed and the code is filtering by[ "update_postimage", "insert" ] the column "_change_type", I would go with the option E. Reference: https://docs.delta.io/latest/delta-change-data-feed.html#:~:text=_change_type,update_preimage%20%2C%20update_postimage
Notice option ("startingVersion", 0), which will bring all changes from begining. Hence Answer is B.
correct answer is B.
Correct B