A data engineer is attempting to drop a Spark SQL table my_table. The data engineer wants to delete all table metadata and data.
They run the following command:
DROP TABLE IF EXISTS my_table -
While the object no longer appears when they run SHOW TABLES, the data files still exist.
Which of the following describes why the data files still exist and the metadata files were deleted?
The reason the data files still exist while the metadata files were deleted is that the table was external. In Spark SQL, an external table indicates that the data itself is managed outside of Spark, and only the metadata is managed within Spark. Thus, issuing a DROP TABLE command on an external table will remove only the table's metadata from the catalog, leaving the underlying data files intact. Conversely, for managed tables, Spark manages both metadata and data files, and dropping such tables would indeed delete both.
C. The table was external The reason why the data files still exist while the metadata files were deleted is because the table was external. When a table is external in Spark SQL (or in other database systems), it means that the table metadata (such as schema information and table structure) is managed externally, and Spark SQL assumes that the data is managed and maintained outside of the system. Therefore, when you execute a DROP TABLE statement for an external table, it removes only the table metadata from the catalog, leaving the data files intact. On the other hand, for managed tables (option E), Spark SQL manages both the metadata and the data files. When you drop a managed table, it deletes both the metadata and the associated data files, resulting in a complete removal of the table.
Option C
C is the correct answer. For external tables, you need to go to the specific location using DESCRIBE EXTERNAL TABLE command and delete all files.
C is correct
THE QUESTION SHOULD BE "Which of the following describes why the metadata files still exist and the data files were deleted?"