A Delta table of weather records is partitioned by date and has the below schema: date DATE, device_id INT, temp FLOAT, latitude FLOAT, longitude FLOAT
To find all the records from within the Arctic Circle, you execute a query with the below filter: latitude > 66.3
Which statement describes how the Delta engine identifies which files to load?
The Delta engine identifies which files to load by scanning the Delta log for min and max statistics for the latitude column. Delta Lake captures statistics for each data file, including minimum and maximum values in each column, and uses these statistics to optimize query execution by determining which files may contain the data that matches the query filter.
Answer D: In the Transaction log, Delta Lake captures statistics for each data file of the table. These statistics indicate per file: - Total number of records - Minimum value in each column of the first 32 columns of the table - Maximum value in each column of the first 32 columns of the table - Null value counts for in each column of the first 32 columns of the table When a query with a selective filter is executed against the table, the query optimizer uses these statistics to generate the query result. it leverages them to identify data files that may contain records matching the conditional filter. For the SELECT query in the question, The transaction log is scanned for min and max statistics for the price column
I checked the delta log, and it dose store stat, stats":"{\"numRecords\":1,\"minValues\":{\"id\":1,\"name\":\"one\",\"age\":11},\"maxValues\":{\"id\":1,\"name\":\"one\",\"age\":11},\"nullCount\":{\"id\":0,\"name\":0,\"age\":0}}"
D https://www.databricks.com/discover/pages/optimize-data-workloads-guide#:~:text=Delta%20data%20skipping%20automatically%20collects,to%20speed%20up%20the%20queries.
_delta_log contains the max and min of each column for the first 30 odd columns in a table for each partition. Also there is nothing called parquet file footers. Correct answer is D.
I think the correct answer is D
D for sure
correct ans is D
D is correct one
D is the answer
D is the right answer
Statistics on first 32 columns of a table are computed and written in the Delta Log by default.
Delta log first
Based on Docu is D I don't know why here is showing B
D is the answer
Delta log collects statistics like min value, max value, no of records, no of files for each transaction that happens on the table for the first 32 columns (default value)
D. The Delta log is scanned for min and max statistics for the latitude column
No explanation needed, this is where the information is stored.