You are managing a Dataplex environment with raw and curated zones. A data engineering team is uploading JSON and CSV files to a bucket asset in the curated zone but the files are not being automatically discovered by Dataplex. What should you do to ensure that the files are discovered by Dataplex?
To ensure that files are automatically discovered by Dataplex, you need to enable the auto-discovery feature for the curated zone. This feature, when enabled, allows Dataplex to automatically discover and catalog data assets within a zone, including JSON and CSV files. Without enabling this feature, Dataplex will not be able to detect the presence of new files in the curated zone.
Should be A. Curated zone need Parquet, Avro, ORC format not CSV or JSON. Check the ref - https://cloud.google.com/dataplex/docs/add-zone#curated-zones
- Auto-Discovery Feature: Dataplex has an auto-discovery feature that, when enabled, automatically discovers and catalogs data assets within a zone. - Appropriate for Both Raw and Curated Zones: This feature is applicable to both raw and curated zones, and it should be tailored to the specific data governance and cataloging needs of the organization.
I'd go for Option B, auto-discovery is enabled by default for any zone, including curated ones, so if a file is not automatically discovered it's due to the disabled auto-discovery
In this case, it would be because of invalid data format in curated zones (data not in Avro, Parquet, or ORC formats).
I will go with A, check the ref. Curated zones only store Parquet, Avro, and ORC in CS, and well-defined schema and Hive-style partitions in the BigQuery: https://cloud.google.com/dataplex/docs/add-zone#curated-zones
Discovery raises the following administrator actions whenever data-related issues are detected during scans : Inconsistent data format in a table. For example, files of different formats exist with the same table prefix. Inconsistent data format in a table. For example, files of different formats exist with the same table prefix.
https://cloud.google.com/dataplex/docs/discover-data#invalid_data_format
According to this URL: https://cloud.google.com/dataplex/docs/discover-data, the auto-discovery can support CSV and Json in both Raw-Zone and Curated-Zone. I also open a console the verify it, both Raw and Curated zone can set up csv&json auto-discovery.
While none of the original options (A, B, C, or D) directly address the issue, the closest solution is: Move the JSON and CSV files to a raw zone. (This was previously marked as the most voted option, but it's not ideal due to data organization disruption) Here's why this approach might be necessary (but not ideal): Dataplex curated zones currently don't support native processing of JSON and CSV formats. They are designed for structured data formats like Parquet, Avro, or ORC.
A. Move the JSON and CSV files to the raw zone.
Dataplex does not allow users to create CSV files within a “curated zone”
The answer can be found reading a common config of Dataplex in this URL: https://medium.com/google-cloud/google-cloud-dataplex-part-1-lakes-zones-assets-and-discovery-5f288486cb2f
GCP001 agree with him
Its B guys, i encounter this in my job, and I had to do B to make it work
Actually I did this in a Raw zone, not Curated.
Its A :)
Option A https://cloud.google.com/dataplex/docs/add-zone#raw-zones Raw zones are the only zones that support CSV & JSON
While JSON and CSV can technically be stored in curated zones, it is not a common practice due to the reasons mentioned above. no where in the mention link its mention that there is a restriction.