You are designing a data pipeline using Snowpipe to ingest data from multiple S3 buckets into a single Snowflake table. Each S3 bucket represents a different data source and contains files in JSON format. You want to use Snowpipe's auto-ingest feature and a single Snowpipe object for all buckets to simplify management and reduce overhead. However, each data source has a different JSON schem a. How can you best achieve this goal while ensuring data is loaded correctly and efficiently into the target table?
Correct Answer: D
The most efficient and manageable approach is to use a single Snowpipe with a UDF to handle schema variations. The UDF can inspect the S3 bucket path (available as metadata within the 'COPY INTO' statement) and apply the correct transformation logic for each data source. Creating separate Snowpipes (A) adds unnecessary overhead. Using a generic 'FILE FORMAT and a VIEW (B) might work for simple transformations, but it becomes complex with significant schema differences. Using VARIANT and external tables (C) defeats the purpose of Snowpipe. Pre-processing in S3 (E) adds complexity outside of Snowflake. UDF provides schema flexibility during ingest and leverages Snowpipe's capabilities directly.