This tutorial will describe how to determine if you need to run this upgrade process, and if you do, how to do so. This process does require temporarily turning off Airbyte.
All minor and major version releases requiring updating the data that Airbyte stores internally. We follow standard Semantic Versioning conventions. You can always find the latest stable version of Airbyte in our repository here. If you are upgrading to a new major or minor version follow the steps below to upgrade your configuration data.
Turn off Airbyte
Turn on the Airbyte web app, server, and db.
docker-compose up db server webapp
Navigate to the Admin page in the UI. Then go to the Configuration Tab. Click Export. This will download a gzipped tarball of all of your Airbyte configuration data and sync history. Note: Any secrets that you have entered into Airbyte will be in this archive, so you should treat it as secret.
Migrate the archive to the new version using the Migration App (packaged in a docker container).
docker run --rm -v <path to directory containing downloaded airbyte_archive.tar.gz>:/config airbyte/migration:<version you are upgrading to> --\--input /config/airbyte_archive.tar.gz\--output <path to where migrated archive will be written (should end in .tar.gz)>\--target-version <version you are migrating to>
Here's an example of what might look like with the values filled in. It assumes that the downloaded
airbyte_archive.tar.gz is in
docker run --rm -v /tmp:/config airbyte/migration:0.15.0-alpha --\--input /config/airbyte_archive.tar.gz\--output /config/airbyte_archive_migrated.tar.gz\--target-version 0.15.0-alpha
Turn off Airbyte fully.
Delete the existing Airbyte docker volumes. Note: Make sure you have already exported your data (step 3). This command is going to delete your data in Docker!
docker volume rm $(docker volume ls -q | grep airbyte)
Upgrade the docker instance to new version.
i. If you are running Airbyte from a cloned version of the Airbyte repo and want to use the current most recent stable version, just
ii. If you are running Airbyte from a
.env, edit the
VERSION field in that file to be the desired version.
Bring Airbyte back online.
Complete Preferences section. In the subsequent setup page click "Skip Onboarding". Navigate to the Admin page in the UI. Then go to the Configuration Tab. Click Import. This will prompt you to upload the migrated archive to Airbyte. After this completes, your upgraded Airbyte instance will now be running with all of your original configuration.
This step will throw an exception if the data you are trying to upload does not match the version of Airbyte that is running.
If you prefer to import and export your data via API instead the UI, follow these instructions:
Instead of Step 3 above use the following curl command to export the archive:
curl -H "Content-Type: application/json" -X POST localhost:8001/api/v1/deployment/export --output /tmp/airbyte_archive.tar.gz
Instead of Step X above user the following curl command to import the migrated archive:
curl -H "Content-Type: application/x-gzip" -X POST localhost:8001/api/v1/deployment/import --data-binary @<path to arhive>
Here is an example of what this request might look like assuming that the migrated archive is called
airbyte_archive_migrated.tar.gz and is in the
curl -H "Content-Type: application/x-gzip" -X POST localhost:8001/api/v1/deployment/import --data-binary @/tmp/airbyte_archive_migrated.tar.gz