On GCP (Compute Engine)
The instructions have been tested on
Debian GNU/Linux 10 (buster)Create a new instance
- Launch a new instance
- Configure new instance
- For testing out Airbyte, an
e2.mediuminstance is likely sufficient. Airbyte uses a lot of disk space with images and logs, so make sure to provision at least 30GBs of disk per node. - For long-running Airbyte installations, we recommend a
n1-standard-2instance.
Create
Install environment
Note: The following commands will be entered either on your local terminal or in your ssh session on the instance terminal. The comments above each command block will indicate where to enter the commands.
- Set variables in your terminal
1
# In your workstation terminal
2
PROJECT_ID=PROJECT_ID_WHERE_YOU_CREATED_YOUR_INSTANCE
3
INSTANCE_NAME=airbyte # or anyother name that you've used
Copied!
- Install
gcloud
MacOS
Ubuntu
1
# In your workstation terminal
2
brew install --cask google-cloud-sdk
3
gcloud init # Follow instructions
Copied!
1
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
2
sudo apt-get install apt-transport-https ca-certificates gnupg
3
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
4
sudo apt-get update && sudo apt-get install google-cloud-sdk
Copied!
1
# Verify you can see your instance
2
gcloud --project $PROJECT_ID compute instances list
3
[...] # You should see the airbyte instance you just created
Copied!
- Connect to your instance
1
# In your workstation terminal
2
gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME
Copied!
- Install
docker
1
# In your ssh session on the instance terminal
2
sudo apt-get update
3
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common
4
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add --
5
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian buster stable"
6
sudo apt-get update
7
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
8
sudo usermod -a -G docker $USER
Copied!
- Install
docker-compose
1
# In your ssh session on the instance terminal
2
sudo apt-get -y install wget
3
sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
4
sudo chmod +x /usr/local/bin/docker-compose
5
docker-compose --version
Copied!
- Close the ssh connection to ensure the group modification is taken into account
1
# In your ssh session on the instance terminal
2
logout
Copied!
Install & start Airbyte
- Connect to your instance
1
# In your workstation terminal
2
gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME
Copied!
- Install Airbyte
1
# In your ssh session on the instance terminal
2
mkdir airbyte && cd airbyte
3
wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
4
docker-compose up -d
Copied!
Connect to Airbyte
For security reasons, we strongly recommend to not expose Airbyte publicly. Future versions will add support for SSL & Authentication.
- Create ssh tunnel.
1
# In your workstation terminal
2
gcloud --project=$PROJECT_ID beta compute ssh airbyte -- -L 8000:localhost:8000 -N -f
Copied!
Troubleshooting
Last modified 30d ago
Copy link
Edit on GitHub