The following technologies are required to build Airbyte locally.
Java 14
Node 14
Python 3.7
Docker
Postgresql
Jq
CMake
Manually switching between different language versions can get hairy. We recommend using a version manager such as pyenv
or jenv
.
To start contributing:
Start by forking the repository
Clone the fork on your workstation:
git clone [email protected]:{YOUR_USERNAME}/airbyte.gitcd airbyte
You're ready to start!
To compile the code and run unit tests:
./gradlew clean build
This will build all the code and run all the unit tests.
./gradle build
creates all the necessary artifacts (Webapp, Jars and Docker images) so that you can run Airbyte locally.
On Mac, if you run into an error while compiling openssl (this happens when running pip install), you may need to explicitly add these flags to your bash profile so that the C compiler can find the appropriate libraries.
export LDFLAGS="-L/usr/local/opt/openssl/lib"export CPPFLAGS="-I/usr/local/opt/openssl/include"
./gradlew builddocker-compose --env-file .env.dev -f docker-compose.yaml -f docker-compose.dev.yaml up
The build will take a few minutes. Once it completes, Airbyte compiled at current git revision will be running in dev
mode in your environment.
In dev
mode, all data will be persisted in /tmp/dev_root
.
To run acceptance (end-to-end) tests, you must have the Airbyte running locally.
./gradlew buildVERSION=dev docker-compose up./gradlew :airbyte-tests:acceptanceTests
The easiest way to develop on one of Airbyte's modules is to spin up the whole Airbyte system on your workstation, and shutdown the module you want to work on.
Spin up Airbyte locally so the UI can make requests against the local API.
Stop the webapp
.
docker-compose stop webapp
Start up the react app.
cd airbyte-webappnpm installnpm start
Happy Hacking!
Spin up Airbyte locally.
Stop the server
.
docker-compose stop server
Run the server
with the command line. It will build and start a server
with the current state of the code. You can also start the server
from your IDE if you need to use a debugger.
./gradlew :airbyte-server:run
Make sure everything is working by testing out a call to the API.
curl -H "Content-Type: application/json"\-X POST localhost:8001/api/v1/workspaces/get\-d '{ "workspaceId": "5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6" }'
Happy Hacking!
Note: We namespace most API calls with a workspace id. For now there is only ever one workspace that is hardcoded to the id used in this example. If you ever need a workspace id, just use this one.
Spin up Airbyte locally.
Stop the scheduler
.
docker-compose stop scheduler
Run the scheduler
with the command line. It will build and start a scheduler
with the current state of the code. You can also start the scheduler
from your IDE if you need to use a debugger.
./gradlew :airbyte-scheduler:run
Happy Hacking!
The Configuration API caches connector specifications. This is done to avoid needing to run docker everytime one is needed in the UI. Without this caching, the UI crawls. If you update the specification of a connector and you need to clear this cache so the API / UI pick up the change. You have two options: 1. Go to the Admin page in the UI and update the version of the connector. Updating to the same version will for the cache to clear for that connector. 1. Restart the server
docker-compose --env-file .env.dev -f docker-compose.yaml -f docker-compose.dev.yaml down -vdocker-compose --env-file .env.dev -f docker-compose.yaml -f docker-compose.dev.yaml up
Sometimes you'll want to reset the data in your local environment. One common case for this is if you are updating an connector's entry in the database (airbyte-config/init/src/main/resources/config
), often the easiest thing to do, is wipe the local database and start it from scratch. To reset your data back to a clean install of Airbyte, follow these steps:
Delete the datastore volumes in docker
docker-compose --env-file .env.dev -f docker-compose.yaml -f docker-compose.dev.yaml down -v
Remove the data on disk
rm -rf /tmp/dev_rootrm -rf /tmp/airbyte_local
Rebuild the project
./gradlew builddocker-compose --env-file .env.dev -f docker-compose.yaml -f docker-compose.dev.yaml up -V