On AWS (EC2)
The instructions have been tested on Amazon Linux 2 AMI (HVM)

Create a new instance

  • Launch a new instance
  • Select instance AMI
  • Select instance type
    • For testing out Airbyte, a t2.medium instance is likely sufficient. Airbyte uses a lot of disk space with images and logs, so make sure to provision at least 30GBs of disk per node.
    • For long-running Airbyte installations, we recommend a t2.large instance.
  • Next: Configure Instance Details
    • You can tune parameters or keep the defaults
  • Next: Add Storage
    • You can tune parameters or keep the defaults
  • Next: Add Tags
    • You can tune parameters or keep the defaults
  • Next: Configure Security Groups
    • We are going to allow network for ssh
  • Review and Launch
  • Launch
  • Create a ssh key so you can connect to the instance
    • Download the key (and don't lose it or you won't be able to connect to the instance)
  • Launch Instances
  • Wait for the instance to become Running

Install environment

Note: The following commands will be entered either on your local terminal or in your ssh session on the instance terminal. The comments above each command block will indicate where to enter the commands.
  • Connect to your instance
1
# In your workstation terminal
2
SSH_KEY=~/Downloads/airbyte-key.pem # or wherever you've downloaded the key
3
INSTANCE_IP=REPLACE_WITH_YOUR_INSTANCE_IP
4
chmod 400 $SSH_KEY # or ssh will complain that the key has the wrong permissions
5
ssh -i $SSH_KEY [email protected]$INSTANCE_IP
Copied!
  • Install docker
1
# In your ssh session on the instance terminal
2
sudo yum update -y
3
sudo yum install -y docker
4
sudo service docker start
5
sudo usermod -a -G docker $USER
Copied!
  • Install docker-compose
1
# In your ssh session on the instance terminal
2
sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
3
sudo chmod +x /usr/local/bin/docker-compose
4
docker-compose --version
Copied!
  • Close the ssh connection to ensure the group modification is taken into account
1
# In your ssh session on the instance terminal
2
logout
Copied!

Install & start Airbyte

  • Connect to your instance
1
# In your workstation terminal
2
ssh -i $SSH_KEY [email protected]$INSTANCE_IP
Copied!
  • Install Airbyte
1
# In your ssh session on the instance terminal
2
mkdir airbyte && cd airbyte
3
wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
4
docker-compose up -d
Copied!

Connect to Airbyte

For security reasons, we strongly recommend to not expose Airbyte on Internet available ports. Future versions will add support for SSL & Authentication.
  • Create ssh tunnel for port 8000
If you want to use different ports you will need to modify API_URL in your .env file and restart Airbyte.
1
# In your workstation terminal
2
ssh -i $SSH_KEY -L 8000:localhost:8000 -N -f [email protected]$INSTANCE_IP
Copied!

Pushing Airbyte logs to CloudWatch

If you want to get your logs from your Airbyte Docker containers in CloudWatch, simply follow this guide to do so.

Troubleshooting

If you encounter any issues, just connect to our Slack. Our community will help! We also have a FAQ section in our docs for common problems.
Last modified 30d ago