Automate FTP of files (csv) to Amazon S3 bucket

Amazon S3 is an object storage service. It cannot "pull" data from an external location.

Therefore, you will need a script or program that will:

• Retrieve the data from the FTP server, and
• Upload the data to Amazon S3

It would be best to run such a script from the FTP server itself, so that the data can be sent to S3 without having to download from the FTP server first. If this is not possible, then you could run the script on any computer on the Internet, such as your own computer or an Amazon EC2 instance.

The simplest way to upload to Amazon S3 is to use the AWS Command-Line Interface (CLI). It has a aws s3 cp command to copy files, or depending upon what needs to be copied it might be easier to use the aws s3 sync command that automatically copies new or modified files.

The script could be triggered via a schedule (cron on Linux or a Scheduled Task on Windows).

If you are using an Amazon EC2 instance, you could save money by turning off the instance when it is not required. The flow could be:

• Create an Amazon CloudWatch Event rule that triggers an AWS Lambda function
• The AWS Lambda function can call StartInstances() to start a stopped EC2 instance
• The Amazon EC2 instance can use a startup script (see details below) that will run your process
• At the end of the process, tell the operating system to shutdown (sudo shutdown now -h)

This might seem like a lot of steps, but the CloudWatch Event and Lambda function are trivial to configure.

To execute a script every time a Linux instance starts, put it in: /var/lib/cloud/scripts/per-boot/

See also: Auto-Stop EC2 instances when they finish a task - DEV Community