HOTSPOT -
Which Azure Data Factory components should you recommend using together to import the customer data from Salesforce to Data Lake Storage? To answer, select the appropriate options in the answer area.
NOTE: Each correct selection is worth one point.
Hot Area:
Box 1: Self-hosted integration runtime
A self-hosted IR is capable of nunning copy activity between a cloud data stores and a data store in private network.
Box 2: Schedule trigger -
Schedule every 8 hours -
Box 3: Copy activity -
Scenario:
✑ Customer data, including name, contact information, and loyalty number, comes from Salesforce and can be imported into Azure once every eight hours. Row modified dates are not trusted in the source table.
✑ Product data, including product ID, name, and category, comes from Salesforce and can be imported into Azure once every eight hours. Row modified dates are not trusted in the source table.
Azure IR is perfectly capable of running Salesforce connection. No need for VM or SSIS runtime. Tumbling trigger will serve better here. Only the copy activity is the right answer.
why cant it be scheduled trigger ?
it is scheduled trigger since modified dates are not reliable
Agree with Azure IR as this is cloud to cloud. Scheduled trigger makes more sense as we need to get data every 8 hours
As modified dates are not reliable, tumbling windows should not be used, Scheduled trigger is the correct option
Modified dates do not play any role here. In a tumbling window you just set up a starting date/time and set up a frequency and that's it. It will work exactly the same as with a scheduled trigger with recurrence every 8 hours. But tumbling windows are a bit safer because you don't need to set up the first date/time in the future.
https://docs.microsoft.com/en-us/azure/data-factory/connector-salesforce
it is scheduled trigger since modified dates are not reliable
Modified dates do not play any role here. In a tumbling window you just set up a starting date/time and set up a frequency and that's it. It will work exactly the same as with a scheduled trigger with recurrence every 8 hours. But tumbling windows are a bit safer because you don't need to set up the first date/time in the future.
I am literally working a project right now where we ingest data from salesforce to data lake. Its the Azure IR.
See https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview#supported-data-stores-and-formats: "When you're copying data between two data stores that are publicly accessible through the internet from any IP, you can use the Azure integration runtime for the copy activity. This integration runtime is secure, reliable, scalable, and globally available. When you're copying data to and from data stores that are located on-premises or in a network with access control (for example, an Azure virtual network), you need to set up a self-hosted integration runtime." Further down on that page, Salesforce is listed as supported by Azure IR.
Important When you copy data into Salesforce, the default Azure Integration Runtime can't be used to execute copy. In other words, if your source linked service doesn't have a specified integration runtime, explicitly create an Azure Integration Runtime with a location near your Salesforce instance. Associate the Salesforce linked service as in the following example. So the given answer is correct.
In the question its mentioned to copy data from salesforce not into salesforce. I feel Azure IR should be correct here.
As it's mentioned every 8 hrs why not tumbling window trigger rather scheduled.. ?
"Row modified dates are not trusted in the source table."
Salesforce is a cloud data source even though there is no clear explanation from the question. Azure IR and Self-Hosted IR both will work via different approaches. Would prefer Azure IR as the answer due to simplicity. https://docs.microsoft.com/en-us/azure/data-factory/data-migration-guidance-s3-azure-storage
Azure IR is recommended for Azure services only
Any third party cloud or in fact Azure VM (or Azure IaaS) requires self-hosted IR
I agree on this. If you want to determine which IR to use you can read https://docs.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime#determining-which-ir-to-use. You need to look at the source and sink environment where you run the ADF pipeline. In case of salesForce, ADF has a SalesForce connector which does not have a source IR (https://docs.microsoft.com/en-gb/azure/data-factory/connector-salesforce). Instead it requires you to specify the IR of the sink. In this case the sink lies within Azure. Since the sink lies within Azure, the recommended IR is the Azure IR.
I just completed the exam, the question is along the lines of: "comes from Salesforce, a SaaS application" if that helps
If they say Salesforce SaaS, the answer should be: Azure IR (https://docs.microsoft.com/en-gb/azure/data-factory/copy-activity-overview - [Salesforce -> Supported by Azure IR -> ✓])
If they say Salesforce SaaS, the answer should be: Azure IR (https://docs.microsoft.com/en-gb/azure/data-factory/copy-activity-overview - [Salesforce -> Supported by Azure IR -> ✓])
Any third party cloud or in fact Azure VM (or Azure IaaS) requires self-hosted IR
For box 1 both Azure IR and self hosted IR are correct as the data is in Salesforce and Salesforce supports both: https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview#supported-data-stores-and-formats
the answer should be tumbling window trigger isn't it?
"Row modified dates are not trusted in the source table."
You keep saying this but its not clear why this makes a difference. Scheduled is time of day where as tumbling relates to every n hours. How does differing triggers for 800/1400/2000 hrs eg have any bearing on the row modified date? I'm not saying its wrong just you need to justify it better. I believe tumbling is a better approach because you have 1 not 3 triggers.
Tumbling Window in ADF depends on a time field of the source data to determine if it should process it or not. In the case of Scheduled trigger, it keeps track of the processed date and time for each row externally. In case of the scheduled trigger, you only need one trigger, not 3.
I'm pretty sure that's not true. I'm looking at it now and cannot see that dependency. Also there is not mention of that requirement here:https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger
You can Create Scheduled Type Triger with recurrence every n Hours. However MS says that the Scheduled is less Reliable. So will tend to agree that Tumbling is more appropriate. Here is difference Between Scheduled and Tumbling Triggers for ADF : https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#trigger-type-comparison
I'm pretty sure that's not true. I'm looking at it now and cannot see that dependency. Also there is not mention of that requirement here:https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger
You can Create Scheduled Type Triger with recurrence every n Hours. However MS says that the Scheduled is less Reliable. So will tend to agree that Tumbling is more appropriate. Here is difference Between Scheduled and Tumbling Triggers for ADF : https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#trigger-type-comparison
Tumbling Window in ADF depends on a time field of the source data to determine if it should process it or not. In the case of Scheduled trigger, it keeps track of the processed date and time for each row externally. In case of the scheduled trigger, you only need one trigger, not 3.
I'm pretty sure that's not true. I'm looking at it now and cannot see that dependency. Also there is not mention of that requirement here:https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger
You can Create Scheduled Type Triger with recurrence every n Hours. However MS says that the Scheduled is less Reliable. So will tend to agree that Tumbling is more appropriate. Here is difference Between Scheduled and Tumbling Triggers for ADF : https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#trigger-type-comparison
I'm pretty sure that's not true. I'm looking at it now and cannot see that dependency. Also there is not mention of that requirement here:https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger
You can Create Scheduled Type Triger with recurrence every n Hours. However MS says that the Scheduled is less Reliable. So will tend to agree that Tumbling is more appropriate. Here is difference Between Scheduled and Tumbling Triggers for ADF : https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#trigger-type-comparison
You keep saying this but its not clear why this makes a difference. Scheduled is time of day where as tumbling relates to every n hours. How does differing triggers for 800/1400/2000 hrs eg have any bearing on the row modified date? I'm not saying its wrong just you need to justify it better. I believe tumbling is a better approach because you have 1 not 3 triggers.
Tumbling Window in ADF depends on a time field of the source data to determine if it should process it or not. In the case of Scheduled trigger, it keeps track of the processed date and time for each row externally. In case of the scheduled trigger, you only need one trigger, not 3.
I'm pretty sure that's not true. I'm looking at it now and cannot see that dependency. Also there is not mention of that requirement here:https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger
You can Create Scheduled Type Triger with recurrence every n Hours. However MS says that the Scheduled is less Reliable. So will tend to agree that Tumbling is more appropriate. Here is difference Between Scheduled and Tumbling Triggers for ADF : https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#trigger-type-comparison
I'm pretty sure that's not true. I'm looking at it now and cannot see that dependency. Also there is not mention of that requirement here:https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger
You can Create Scheduled Type Triger with recurrence every n Hours. However MS says that the Scheduled is less Reliable. So will tend to agree that Tumbling is more appropriate. Here is difference Between Scheduled and Tumbling Triggers for ADF : https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#trigger-type-comparison
Tumbling Window in ADF depends on a time field of the source data to determine if it should process it or not. In the case of Scheduled trigger, it keeps track of the processed date and time for each row externally. In case of the scheduled trigger, you only need one trigger, not 3.
I'm pretty sure that's not true. I'm looking at it now and cannot see that dependency. Also there is not mention of that requirement here:https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger
You can Create Scheduled Type Triger with recurrence every n Hours. However MS says that the Scheduled is less Reliable. So will tend to agree that Tumbling is more appropriate. Here is difference Between Scheduled and Tumbling Triggers for ADF : https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#trigger-type-comparison
I'm pretty sure that's not true. I'm looking at it now and cannot see that dependency. Also there is not mention of that requirement here:https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger
You can Create Scheduled Type Triger with recurrence every n Hours. However MS says that the Scheduled is less Reliable. So will tend to agree that Tumbling is more appropriate. Here is difference Between Scheduled and Tumbling Triggers for ADF : https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#trigger-type-comparison
we will need Self-hosted since the salesforce is an on-premise Source. ref- https://docs.microsoft.com/en-us/azure/data-factory/connector-salesforce
I agree is azure IR because salesforce SaaS, but I don’t know what trigger is better. Schedule with frequency 8h fit perfectly but, at the same time Tumbling with 8h window do the same. I think the first one always trigger at these hours independently if the process was cancelled or not, and tumbling may introduce delays if process fails and has to be relaunched because next window will be 8h later than this second try. Am I right?
Schedule trigger: A trigger that invokes a pipeline on a wall-clock schedule. Tumbling window trigger: A trigger that operates on a periodic interval, while also retaining state. Event-based trigger: A trigger that responds to an event.
IR to be used is Azure IR as mentioned here --> https://docs.microsoft.com/en-us/azure/data-factory/connector-salesforce "The integration runtime to be used to connect to the data store. If not specified, it uses the default Azure Integration Runtime"
"When you copy data into Salesforce, the default Azure Integration Runtime can't be used to execute copy. In other words, if your source linked service doesn't have a specified integration runtime, explicitly create an Azure Integration Runtime with a location near your Salesforce instance. Associate the Salesforce linked service as in the following example." https://docs.microsoft.com/en-us/azure/data-factory/connector-salesforce
https://docs.microsoft.com/en-us/azure/data-factory/connector-salesforce-service-cloud: "When you copy data into Salesforce Service Cloud, the default Azure Integration Runtime can't be used to execute copy. In other words, if your source linked service doesn't have a specified integration runtime, explicitly create an Azure Integration Runtime with a location near your Salesforce Service Cloud instance." Integration runtime is Azure Trigger as schedule and activity as copy are correct
Based on reviewed information all the answers provided are correct.
As suggested it seems Azure IR is also possible and this is also the only solution, because it is mentioned to use PaaS and no VMs managed by Litware. Self-hosted IR requires managing VMs, therefore => Azure IR