What is generally the FASTEST way to bulk load data files from a stage?
What is generally the FASTEST way to bulk load data files from a stage?
Loading data by path (internal stages) or prefix is generally the fastest way to bulk load data files from a stage. This method allows you to load multiple files in a single operation without needing to specify each file individually. It is more scalable and efficient for handling large datasets, especially when dealing with frequent data updates. Specifying a list of specific files to load can be less efficient because it requires manually listing each file, which is not practical for a large number of files.
https://docs.snowflake.com/en/user-guide/data-load-considerations-load#:~:text=Of%20the%20three%20options%20for,load%20up%20to%201%2C000%20files.
Of the three options for identifying/specifying data files to load from a stage, providing a discrete list of files is generally the fastest; however, the FILES parameter supports a maximum of 1,000 files, meaning a COPY command executed with the FILES parameter can only load up to 1,000 files.
Option A, "Specifying a list of specific files to load," can also be fast but is generally less efficient for bulk data loading compared to option B, "Loading by path (internal stages) / prefix." The reason is that specifying a list of specific files means you need to provide each individual file name to be loaded. If you have a large number of files to load, it can be time-consuming and less efficient to list them all explicitly. Additionally, if new files are added frequently, you would need to update the list regularly. On the other hand, option B allows you to load all files with a common path or prefix, which means you can load multiple files in a single operation without needing to specify each file individually. This approach is more scalable and efficient for bulk data loading, especially when dealing with large datasets or frequent data updates.
Of the three options for identifying/specifying data files to load from a stage, providing a discrete list of files is generally the fastest; however, the FILES parameter supports a maximum of 1,000 files, meaning a COPY command executed with the FILES parameter can only load up to 1,000 files.
loading by path is recommended
vvvvvvv
B. most efficient for bulk loading, files can be separated by path/prefix
What is generally the FASTEST way to bulk load data files from a stage?