Amazon S3 (Simple Storage Service) is a highly scalable and secure object storage solution that serves as a versatile component in data workflows. In a streaming ETL (Extract, Transform, Load) workflow, S3 often acts as a source, destination, or intermediate storage for raw and transformed data. Its flexibility and cost-effectiveness make it an ideal choice for handling large-scale data streams and archiving processed data.

Connecting S3 (IAM Role Setup - General Setup)

To integrate S3 into your ProxyHook you first need to create an IAM role for ProxyHook within your AWS account. If you've already done this, you can skip to the next section

1. Log into your AWS account and head to 'Identity and Access Management (IAM)'
2. Click 'Create Role'
3. Select 'AWS account' and select 'Another AWS Account' with the value 121269227306
4. Skip the permissions policy page by clicking 'Next'
5. Provide a name for the role (e.g. ProxyHook) and click Create Role
6. Note the ARN Role for your S3 Settings in the ProxyHook Settings page

Connecting S3 (IAM Role Setup - Bucket Access)

In order to grab files from your S3 bucket, you will need to ensure your ProxyHook IAM account has permissions to read from the desired bucket. To do so, follow these steps:

1. Log into your AWS account and head to 'Identity and Access Management (IAM)' > 'Roles'
2. Select your role made from above and under Permissions, Select Add permissions > Create Inline Police
3. Click JSON and paste the following, editing the code where it says bucket name

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": "s3:GetObject",

"Resource": "arn:aws:s3:::YOUR_S3_BUCKET_HERE/*"

}

]

}

Click next, create a policy name and hit save. If done correctly, the IAM role will now be able to read files from your S3 Bucket.

Enabling Event Notifications

The final step to start receiving the data from new S3 files is to add an Event Notification for your bucket.

1. Log into your AWS Account and head to your S3 Bucket
2. Click 'Properties', scroll down to 'Event Notifications' and select 'Create event notification'
3. In the event name, set it to 'proxyhook-WEBHOOKID'. Replace WEBHOOK id with the unique ID from your endpoint created in ProxyHook source. For example, if you endpoint is https://go.proxyhook.com/2me3mnbmljni, you'd use '2me3mnbmljni'. That means your event name in AWS should be 'proxyhook-2me3mnbmljni' (without quotes)
4. Select any event notification types
5. Under destination, select 'SNS Topic', and Manually enter the SNS topic of 'arn:aws:sns:us-east-1:121269227306:s3' (without quotes).
6. Save

At this point, any new files dropped into your S3 Bucket (based on optional criteria you set during notification setup) will feed into your ProxyHook event.