You have a Fabric warehouse named DW1 that contains a Type 2 slowly changing dimension (SCD) dimension table named DimCustomer. DimCustomer contains 100 columns and 20 million rows. The columns are of various data types, including int, varchar, date, and varbinary.
You need to identify incoming changes to the table and update the records when there is a change. The solution must minimize resource consumption.
What should you use to identify changes to attributes?
The correct answer is: A. a hash function to compare the attributes in the source table. Here's why: Context: You are dealing with a Type 2 SCD table with 100 columns and 20 million rows. You need to detect changes to update the dimension appropriately, while minimizing resource consumption. Option A: A. a hash function to compare the attributes in the source table. This is the most efficient and scalable approach. You compute a hash (e.g., MD5, SHA-256) over a concatenation of all relevant (non-SCD metadata) columns in both the source and existing dimension table. By comparing the hashes, you avoid comparing each column directly. This minimizes CPU and memory usage, especially with many columns and large datasets. Commonly used in modern ETL/ELT tools for change detection in large datasets.
why don't use hash function on source side? the question lost many details. I just can guess they try to use incremental loading method. Normally we need keep Hask key values in target table column and use hash function to get changed imported rows from source side if their hash values cannot match with target hash value column.
A. a hash function to compare the attributes in the source table.