On GCP (Compute Engine)

The instructions have been tested on Debian GNU/Linux 10 (buster)
Create a new instance
Launch a new instance
Configure new instance
For testing out Airbyte, an e2.medium instance is likely sufficient.
For long-running Airbyte installations, we recommend a n1-standard-2 instance.
Create
Install environment
Note: The following commands will be entered either on your local terminal or in your ssh session on the instance terminal. The comments above each command block will indicate where to enter the commands.
Set variables in your terminal
# In your workstation terminal
PROJECT_ID=PROJECT_ID_WHERE_YOU_CREATED_YOUR_INSTANCE
INSTANCE_NAME=airbyte # or anyother name that you've used
Install gcloud
MacOS
Ubuntu
MacOS
# In your workstation terminal
brew cask install google-cloud-sdk
gcloud init # Follow instructions
Ubuntu
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get install apt-transport-https ca-certificates gnupg
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt-get update && sudo apt-get install google-cloud-sdk
# Verify you can see your instance
gcloud --project $PROJECT_ID compute instances list
[...] # You should see the airbyte instance you just created
Connect to your instance
# In your workstation terminal
gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME
Install docker
# In your ssh session on the instance terminal
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add --
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian buster stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
sudo usermod -a -G docker $USER
Install docker-compose
# In your ssh session on the instance terminal
sudo apt-get -y install wget
sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version
Close the ssh connection to ensure the group modification is taken into account
# In your ssh session on the instance terminal
logout
Install & start Airbyte
Connect to your instance
# In your workstation terminal
gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME
Install Airbyte
# In your ssh session on the instance terminal
mkdir airbyte && cd airbyte
wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
docker-compose up -d
Connect to Airbyte
For security reasons, we strongly recommend to not expose Airbyte publicly. Future versions will add support for SSL & Authentication.
Create ssh tunnel.
# In your workstation terminal
gcloud --project=$PROJECT_ID beta compute ssh airbyte -- -L 8000:localhost:8000 -N -f
Just visit http://localhost:8000 in your browser and start moving some data!
Troubleshooting
If you encounter any issues, just connect to our Slack. Our community will help! We also have a FAQ section in our docs for common problems.
On Azure(VM)
On Kubernetes (Beta)
Last updated 3 months ago