On GCP (Compute Engine)

The instructions have been tested on Debian GNU/Linux 10 (buster)

Create a new instance

  • Launch a new instance

  • Configure new instance

    • For testing out Airbyte, an e2.medium instance is likely sufficient.

    • For long-running Airbyte installations, we recommend a n1-standard-2 instance.

  • Create

Install environment

This part assumes that you have access to a terminal on your workstation

  • Set variables in your terminal

# In your workstation terminal
PROJECT_ID=PROJECT_ID_WHERE_YOU_CREATED_YOUR_INSTANCE
INSTANCE_NAME=airbyte # or anyother name that you've used
  • Install gcloud

MacOS
Ubuntu
MacOS
# In your workstation terminal
brew cask install google-cloud-sdk
gcloud init # Follow instructions
Ubuntu
# FIXME
# Verify you can see your instance
gcloud --project $PROJECT_ID compute instances list
[...] # You should see the airbyte instance you just created
  • Connect to your instance

# In your workstation terminal
gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME
  • Install docker

# In your ssh session on the instance terminal
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add --
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian buster stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
sudo usermod -a -G docker $USER
  • Install docker-compose

# In your ssh session on the instance terminal
sudo apt-get -y install wget
sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version
  • Close the ssh connection to ensure the group modification is taken into account

# In your ssh session on the instance terminal
logout

Install & start Airbyte

  • Connect to your instance

# In your workstation terminal
gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME
  • Install Airbyte

# In your ssh session on the instance terminal
mkdir airbyte && cd airbyte
wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
docker-compose up -d

Connect to Airbyte

For security reasons, we strongly recommend to not expose Airbyte publicly. Future versions will add support for SSL & Authentication.

  • Create ssh tunnel.

# In your workstation terminal
gcloud --project=$PROJECT_ID beta compute ssh airbyte -- -L 8000:localhost:8000 -L 8001:localhost:8001 -N -f

Troubleshooting

If you encounter any issues, just connect to our Slack. Our community will help! We also have a FAQ section in our docs for common problems.