You are performing a join operation to combine values from a static userLookup table with a streaming DataFrame streamingDF.
Which code block attempts to perform an invalid stream-static join?
You are performing a join operation to combine values from a static userLookup table with a streaming DataFrame streamingDF.
Which code block attempts to perform an invalid stream-static join?
Performing an outer join between a streaming DataFrame and a static DataFrame is invalid. This is because outer joins involve matching rows from both DataFrames, and if there are unmatched rows from the streaming DataFrame, it will not have the necessary data at all times since streaming data continuously evolves. Hence, it cannot return a complete set of results for unmatched data.
Answer B is correct: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#support-matrix-for-joins-in-streaming-queries When we take a look in the supported join matrix between static and stream inputs, we can identify, that Stream-Static + outer is not supported. Answer E is wrong, because the Static-Stream + right join is supported.
in my exam today, BCD are removed. i chose E, because I recall that stream-static right join are less supported.
believe B is correct as provided below
B. We match all the records from a static DataFrame on the left with a stream DataFrame on the right. If records do not match from the static DF (Left) to stream DF (Right), then the system cannot return null since the data changes on stream DF (Right), and we cannot guarantee if we will get matching records. That is why full_outer join is not supported.
b is correct
Specifically, outer joins are not supported with a static DataFrame on the right and a streaming DataFrame on the left. This is because it’s not possible to guarantee all necessary rows will be available in the streaming DataFrame for every micro-batch.
I think the correct answer is D.
Sorry I missread the question.