Upgrading Airbyte

Overview

This tutorial will describe how to determine if you need to run this upgrade process, and if you do, how to do so. This process does require temporarily turning off Airbyte.

Determining if you need to Upgrade

All minor and major version releases requiring updating the data that Airbyte stores internally. We follow standard Semantic Versioning conventions. You can always find the latest stable version of Airbyte in our repository here. If you are upgrading to a new major or minor version follow the steps below to upgrade your configuration data.

If you inadvertently upgrade to a version of Airbyte that is not compatible with your data, the docker containers will not start up and will log an error stating the incompatibility. In these cases, you should downgrade to the previous version that worked and follow the steps below. On the other hand, if you don't mind losing your current Airbyte configuration or have never setup any proper connections yet, you can skip the migrating operations and jump directly to step 5 below.

Upgrading (Docker)

  1. In a terminal, on the host where Airbyte is running, turn off Airbyte.

    docker-compose down
  2. Turn back on the Airbyte web app, server, and db (using the "old" working version from which you are upgrading from).

    docker-compose up -d db server webapp

    Note: the -d flag in the previous command is optional and is meant to run the docker services in the background. This would let you free to keep interacting with the terminal and type the next commands in the tutorial.

  3. Switching over to your browser, navigate to the Admin page in the UI. Then go to the Configuration Tab. Click Export. This will download a compressed back-up archive (gzipped tarball) of all of your Airbyte configuration data and sync history locally.

    Note: Any secrets that you have entered into Airbyte will be in this archive, so you should treat it as secret.

  4. Back to the terminal, migrate the local archive to the new version using the Migration App (packaged in a docker container).

    docker run --rm -v <path to directory containing downloaded airbyte_archive.tar.gz>:/config airbyte/migration:<version you are upgrading to> --\
    --input /config/airbyte_archive.tar.gz\
    --output <path to where migrated archive will be written (should end in .tar.gz)>\
    [ --target-version <version you are migrating to or empty for latest> ]

    Here's an example of what it might look like with the values filled in. It assumes that the downloaded airbyte_archive.tar.gz is in /tmp.

    docker run --rm -v /tmp:/config airbyte/migration:0.26.2-alpha --\
    --input /config/airbyte_archive.tar.gz\
    --output /config/airbyte_archive_migrated.tar.gz
  5. Turn off Airbyte fully and (see warning) delete the existing Airbyte docker volumes.

    WARNING: Make sure you have already exported your data (step 3). This command is going to delete your data in Docker, you may lose your airbyte configurations!

    This is where all airbyte configurations are saved. Those configuration files need to be upgraded and restored with the proper version in the following steps.

    # Careful, this is deleting data!
    docker-compose down -v

    The -v flag is equivalent to running the following two steps in a single command:

    # Turn off Airbyte fully
    docker-compose down
    # Delete the existing Airbyte docker volumes where all airbyte configurations are saved.
    docker volume rm $(docker volume ls -q | grep airbyte)

    5.b. (Skip if you are using default database airbyte-db) If you deployed an external database to be used by Airbyte, you might need to drop all tables from the database and re-initialize them from scratch using the latest updated schema script (replace "airbyte" and "docker" with the appropriate values). If no changes were made to the SQL tables since the last Airbyte version you were using, you can safely skip this step.

  6. Upgrade the docker instance to new version.

    i. If you are running Airbyte from a cloned version of the Airbyte GitHub repo and want to use the current most recent stable version, just git pull.

    ii. If you are running Airbyte from downloaded docker-compose.yaml and .env files without a GitHub repo, run wget -N https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml} to pull the latest versions and overwrite both files.

  7. Bring Airbyte back online.

    docker-compose up
  8. Complete Preferences section. In the subsequent setup page click "Skip Onboarding". Navigate to the Admin page in the UI. Then go to the Configuration Tab. Click Import. This will prompt you to upload the migrated archive to Airbyte. After this completes, your upgraded Airbyte instance will now be running with all of your original configuration.

This step will throw an exception if the data you are trying to upload does not match the version of Airbyte that is running.

Upgrading (K8s)

This process is similar to the Docker upgrade process with several changes to account for the Kubernetes resources.

  1. Follow Step 3 in the Docker upgrade process to export your current configurations as an archive.

  2. Follow Step 4. in the Docker upgrade process process to run the migration on your archive.

  3. Turn off Airbyte fully and (see warning) delete the existing Airbyte Kubernetes volumes.

    WARNING: Make sure you have already exported your data (step 1). This command is going to delete your data in Kubernetes, you may lose your airbyte configurations!

    This is where all airbyte configurations are saved. Those configuration files need to be upgraded and restored with the proper version in the following steps.

    # Careful, this is deleting data!
    kubectl delete -k kube/overlays/stable
  4. Follow Step 6 in the Docker upgrade process to check out the most recent version of Airbyte. Although it is possible to migrate by changing the .env file in the kube overlay directory, this is not recommended as it does not capture any changes to the Kubernetes manifests.

  5. Bring Airbyte back up.

    kubectl apply -k kube/overlays/stable
  6. Follow Step 8 in the Docker upgrade process to upload your migrated Archive and restore your configuration and data.

API Instruction

If you prefer to import and export your data via API instead the UI, follow these instructions:

  1. Instead of Step 3 above use the following curl command to export the archive:

    curl -H "Content-Type: application/json" -X POST localhost:8000/api/v1/deployment/export --output /tmp/airbyte_archive.tar.gz
  2. Instead of Step X above user the following curl command to import the migrated archive:

    curl -H "Content-Type: application/x-gzip" -X POST localhost:8000/api/v1/deployment/import --data-binary @<path to arhive>

Here is an example of what this request might look like assuming that the migrated archive is called airbyte_archive_migrated.tar.gz and is in the /tmp directory.

curl -H "Content-Type: application/x-gzip" -X POST localhost:8000/api/v1/deployment/import --data-binary @/tmp/airbyte_archive_migrated.tar.gz