Snowpipe is a feature that enables continuous, near-real-time data ingestion from external sources into Snowflake tables. Snowpipe can ingest files from Amazon S3, Google Cloud Storage, or Azure Blob Storage into Snowflake tables on any cloud platform. Snowpipe can be triggered in two ways: by using the Snowpipe REST API or by using cloud notifications2 To ingest files from the company's AWS storage accounts into the company's Snowflake GCP account, the Architect can use either of these methods:
* Configure the client application to call the Snowpipe REST endpoint when new files have arrived in Amazon S3 storage. This method requires the client application to monitor the S3 buckets for new files and send a request to the Snowpipe REST API with the list of files to ingest. The client application must also handle authentication, error handling, and retry logic3
* Create an AWS Lambda function to call the Snowpipe REST endpoint when new files have arrived in Amazon S3 storage. This method leverages the AWS Lambda service to execute a function that calls the Snowpipe REST API whenever an S3 event notification is received. The AWS Lambda function must be configured with the appropriate permissions, triggers, and code to invoke the Snowpipe REST API4 The other options are not valid methods for triggering Snowpipe:
* Configure the client application to call the Snowpipe REST endpoint when new files have arrived in Amazon S3 Glacier storage. This option is not feasible because Snowpipe does not support ingesting files from Amazon S3 Glacier storage, which is a long-term archival storage service. Snowpipe only supports ingesting files from Amazon S3 standard storage classes5
* Configure AWS Simple Notification Service (SNS) to notify Snowpipe when new files have arrived in Amazon S3 storage. This option is not applicable because Snowpipe does not support cloud notifications from AWS SNS. Snowpipe only supports cloud notifications from AWS SQS, Google Cloud Pub/Sub, or Azure Event Grid6
* Configure the client application to issue a COPY INTO <TABLE> command to Snowflake when new files have arrived in Amazon S3 Glacier storage. This option is not relevant because it does not use Snowpipe, but rather the standard COPY command, which is a batch loading method. Moreover, the COPY command also does not support ingesting files from Amazon S3 Glacier storage7 References:
* 1: SnowPro Advanced: Architect | Study Guide 8
* 2: Snowflake Documentation | Snowpipe Overview 9
* 3: Snowflake Documentation | Using the Snowpipe REST API 10
* 4: Snowflake Documentation | Loading Data Using Snowpipe and AWS Lambda 11
* 5: Snowflake Documentation | Supported File Formats and Compression for Staged Data Files 12
* 6: Snowflake Documentation | Using Cloud Notifications to Trigger Snowpipe 13
* 7: Snowflake Documentation | Loading Data Using COPY into a Table
* : SnowPro Advanced: Architect | Study Guide
* : Snowpipe Overview
* : Using the Snowpipe REST API
* : Loading Data Using Snowpipe and AWS Lambda
* : Supported File Formats and Compression for Staged Data Files
* : Using Cloud Notifications to Trigger Snowpipe
* : Loading Data Using COPY into a Table