Your company uses a proprietary system to send inventory data every 6 hours to a data ingestion service in the cloud. Transmitted data includes a payload of several fields and the timestamp of the transmission. If there are any concerns about a transmission, the system re-transmits the data. How should you deduplicate the data most efficiency?
The most efficient way to deduplicate data in this context is to compute the hash value of each data entry and compare it with all historical data. This ensures that duplicate entries, even with different timestamps, are identified and eliminated based on the contents of the payload alone. Using a hash value for comparison is computationally efficient and requires less storage compared to other methods such as storing entire payloads or using GUIDs. Additionally, maintaining just the hash value circumvents issues with variations in timestamps while still ensuring data integrity.
The best answer is "A". Answer "D" is not as efficient or error-proof due to two reasons 1. You need to calculate hash at sender as well as at receiver end to do the comparison. Waste of computing power. 2. Even if we discount the computing power, we should note that the system is sending inventory information. Two messages sent at different can denote same inventory level (and thus have same hash). Adding sender time stamp to hash will defeat the purpose of using hash as now retried messages will have different timestamp and a different hash. if timestamp is used as message creation timestamp than that can also be used as a UUID.
If the goal is to ensure at least ONE of each pair of entries is inserted into the db, then how is assigning a GUID to each entry resolving the duplicates? Keep in mind if the 1st entry fails, then hopefully the 2nd (duplicate) is successful.
A - In D, same message with different timestamp will have different hash, though the message content is the same.
Strong Answer is A - in another question on the gcp sample questions: the correct answer to that particular question was "You are building a new real-time data warehouse for your company and will use BigQuery streaming inserts. There is no guarantee that data will only be sent in once but you do have a unique ID for each row of data and an event timestamp. You want to ensure that duplicates are not included while interactively querying data. Which query type should you use?" This means you need a "uniqueid" and timestamps to properly dedupe a data.
U need a uniqueid but in this scenario, there is none. So u have to calculate by hashing w/ some of the fields in the dataset. A is assigning guid in processing side will not solve the issue. Cause u will assign diff. ids...
Answer - D Key statement is "Transmitted data includes a payload of several fields and the timestamp of the transmission." So the timestamp is appended to message while sending, in other words that field is subject to change if message is retransmitted. However, adding a GUID doesn't help much because if message is transmitted twice you will have different GUID for both messages but they will be the same/duplicate data. You can simply calculate a hash based on not all data but from a select of columns (with the payload of several fields AND definitely by excluding the timestamp). By doing so, you can assure a different hash for each message.
agreed, the key here is "payload of several fields and the timestamp"
"payload of several fields and the timestamp of the transmission"
Hi Max, I also think that the hash value would be worng because the timestamp is part of payload and is not written that the hash value is generated without the ts; but it also not written if GUID is linked or not with sending. Often this is a point where the answer is vague. Because don't specify if the GUID is related to the data or to the send.
Answer is D. Using Hash values we can remove duplicate values from a database. Hash values will be same for duplicate data and thus can be easily rejected. Obviously you won't check hash for timestmp. D is better thatn B because maintaning a different table will reduce cost for hash computation for all historical data
Why can't it be A, where the GUID is a hash value? Why do we need to store the hash with the metadata in a separate database to do the deduplication?
If you add a unique ID aren't you by definition not getting a duplicate record. Honestly I hate all these answers.
You can add a function or condition that verifies if the global unique id already exists or just do a deduplication later
Answer: D Description: Using Hash values we can remove duplicate values from a database. Hashvalues will be same for duplicate data and thus can be easily rejected.
Hash values for same data will be the same, but in this case data contains also the timestamp
While calculating Hash value we exclude the timestamp.
Hard question. It's a *proprietary* system. Who guarantees we can even add a GUID? But if you can, it's definitely more efficient than calculating hashes (ignoring timestamp).
Answer is B - although the time stamp is diff for each transmission - the hash value is computed for the payload, not for the timestamp - which is just an added field for transmission. So, has val remains the same for all transmissions of the same data - which is what we can use for comparision. So, much more efficient to just directly compare the hash values with the historical data - to check and remove duplicates - instead of again wasting space storing stuff - in option D
B. Compute the hash value of each data entry, and compare it with all historical data. Explanation: Efficiency: Hashing is a fast and efficient operation, and comparing hash values is generally quicker than comparing the entire payload or using other methods. Space Efficiency: Storing hash values requires less storage space compared to storing entire payloads or using global unique identifiers (GUIDs). Deduplication: By computing the hash value of each data entry and comparing it with historical data, you can easily identify duplicate transmissions. If the hash value matches an existing one, it indicates that the payload is the same.
As Dg63 wrote.
A is best choice. D doesn't make sense.
A is incorrect. how can you find duplicates if you assign a unique id to every record? The answer is either B or D. I first selected B, but reading through the answers D may be better.
This question is formulated very badly. From the way that A is formulated, you would not deduplicate but rather the duplicates would have the same GUID. Then we have D, which is storing the information (assuming the hash is created without the timestamp). B is doing it right away. D only alludes to the actual deduplication. But it would be more efficient.
Why not D ? Generate a Hash for payload entry and maintain the value as metadata. Do the validation check on Dataflow..... A GUID will generate 2 different entries for same payload entry, it will not tackle duplication check
I though the answer was A 'cos it's more efficient. But I read the answer with more attention: GUID is given "at each data entry". It's not said that GUID was given from publisher. If GUID is given in data entry (subscriber), two equal messages can have different GUID. D is not complete 'cos it's not so precise about hash field that are used. I'm in doubt on this answer :-(
Data entry means record, it is not an action. that means that each record will have a unique id. so assuming our sink will not accept duplicates based on a key, the GUID will work.
Answer B: Option A: GUIDs can deduplicate the data but is expensive and good for multiple data processing. Option B: Using hash function to authenticate the unique rows, this function can be applied directly in bigquery. Option D, is complex and more expensive. `` `CREATE TEMP FUNCTION hashValue(input STRING) AS ( CAST(FARM_FINGERPRINT(input) AS STRING) ); ``
Just asked Chatgpt, it gave me option D
you cannot deduplicate data adding a random guid, with guid row is distinct than others
D. Maintain a database table to store the hash value and other metadata for each data entry. Storing a database table with hash values and metadata is an efficient way to deduplicate data. When new data is transmitted, you can calculate the hash of the payload and check whether it already exists in the database. This approach allows for efficient duplicate detection without the need to compare the new data with all historical data. It's a common and scalable technique used to ensure data consistency and avoid processing the same data multiple times. Options A (assigning GUIDs to each data entry) and C (storing each data entry as the primary key) can work, but they might be less efficient than using hash values when dealing with a large volume of data. Option B (computing the hash value of each data entry and comparing it with all historical data) can be computationally expensive and slow, especially if there's a significant amount of historical data to compare against. Storing hash values in a table allows for fast and efficient deduplication.
Answer : A "D" is not as efficient or error-proof due to two reasons 1. You need to calculate hash at sender as well as at receiver end to do the comparison. Waste of computing power. 2. Even if we discount the computing power, we should note that the system is sending inventory information. Two messages sent at different can denote same inventory level (and thus have same hash). Adding sender time stamp to hash will defeat the purpose of using hash as now retried messages will have different timestamp and a different hash. if timestamp is used as message creation timestamp than that can also be used as a UUID.
To deduplicate the data most efficiently, especially in a cloud environment where the data is sent periodically and re-transmissions can occur, the recommended approach would be: D. Maintain a database table to store the hash value and other metadata for each data entry. This approach allows you to quickly check if an incoming data entry is a duplicate by comparing hash values, which is much faster than comparing all fields of a data entry. The metadata, which includes the timestamp and possibly other relevant information, can help resolve any ambiguities that may arise if the hash function ever produces collisions.
1. My original vote was 'B'. I chose it over 'D' because option 'D' does not explicitly say anything about how that table will be used for deduplication. In hindsight, explicit usage of table should not be given much weightage so after review and seeing other comments, I thought of 'D' as the correct answer. 2. Now looking more clearly at option 'D' (and 'B' also), it's a little ambiguous of what keys will be used to create the hash. So, if you use the payload PLUS the timestamp, the hash is of no use. This is a little confusing 3. Finally, although I never thought this is the right option, 'A' seems to be the correct option. The GUID is created at Data entry, NOT at the transmission stage. So, the GUID should be representative of the payload only and NOT the timestamp which will make it unique per payload, not per transmission of the same payload. So, in the end, I feel like 'A' is the correct choice.