This is the changelog for Airbyte Platform. For our connector changelog, please visit our Connector Changelog page.
You can now properly cancel deployments when deploying on K8s.
You can now send notifications via webhook for successes and failures on Airbyte syncs.
Scheduling jobs and worker jobs are now separated, allowing for workers to be scaled horizontally.
Fixed a bug that made it possible for connector definitions to be duplicated, violating uniqueness.
Nothing of note.
The scheduler now waits for the server before it creates any databases.
You can now apply tolerations for Airbyte Pods on K8s deployments.
Syncs now have a
max_sync_timeout that times them out after 3 days.
Fixed Kube deploys when logging with Minio.
Nothing of note.
Migration of Python connector template images to Alpine Docker images to reduce size.
Nothing of note.
Nothing of note.
Re-release: Fixed errant ENV variable in
Connector pods no longer fail with edge case names for the associated Docker images.
Nothing of note.
Introduced implementation for date-time support in normalization.
Importing configuration no longer removes available but unused connectors.
Fixed nil pointer exception in version migrations.
When migrating, types represented in the config archive need to be a subset of the types declared in the schema.
DEFAULT_WORKSPACE_ID; default workspace no longer exists by default.
Backend now handles workspaceId for WebBackend operations.
K8s: Overly-sensitive logs are now silenced.
Acceptance test dependencies fixed.
Fixed unreliable logging on Kubernetes deployments.
Introduced pre-commit to auto-format files on commits.
Config persistence is now migrated to the internal Airbyte database.
Source connector ports now properly close when deployed on Kubernetes.
Missing dependencies added that allow acceptance tests to run.
Fixed some minor API spec errors.
GCP environment variable is now stubbed out to prevent noisy and harmless errors.
New API endpoint: List workspaces
K8s: Server doesn't start up before Temporal is ready to operate now.
Silent source failures caused by last patch fixed to throw exceptions.
Airbyte now automatically upgrades on server startup!
Airbyte will check whether your
.env Airbyte version is compatible with the Airbyte version in the database and upgrade accordingly.
When running Airbyte on K8s logs will automatically be stored in a Minio bucket unless configured otherwise.
CDC for MySQL now handles decimal types correctly.
First-Class Kubernetes support!
Custom dbt transformations!
You can now configure your destination namespace at the table level when setting up a connection!
Migrate basic normalization settings to the sync operations.
Bugfix: Handle TINYINT(1) and BOOLEAN correctly and fix target file comparison for MySQL CDC.
Bugfix: Updating the source/destination name in the UI now works as intended.
Bugfix: Ensure that logs from threads created by replication workers are added to the log file.
Remove hash from table names when it's not necessary for normalization outputs.
PythonCDK: change minimum Python version to 3.7.0
Minor fixes to documentation
Reliability updates in preparation for custom transformations
Limit Docker log size to 500 MB (#3702)
Fix for file names being too long in Windows deployments (#3625)
Allow users to access the API and WebApp from the same port (#3603)
Checkpointing for incremental syncs that will now continue where they left off even if they fail! (#3290)
Avoid dbt runtime exception "maximum recursion depth exceeded" in ephemeral materialization (#3470)
Documentation to deploy locally on Windows is now available (#3425)
Connector icons are now displayed in the UI
Restart core containers if they fail automatically (#3423)
Progress on supporting custom transformation using dbt. More updates on this soon!
Bump K8s deployment version to latest stable version, thanks to Coetzee van Staden
Progress on checkpointing to support rate limits better
Upgrade normalization to use dbt from docker images (#3186)
Split replication and normalization into separate temporal activities (#3136)
Fix normalization Nesting bug (#3110)
Replace timeout for sources (#3031)
Fix UI issue where tables with the same name are selected together (#3032)
Fix feed handling when feeds are unavailable (#2964)
Export whitelisted tables (#3055)
Namespace support: supported source-destination pairs will now sync data into the same namespace as the source (#2862)
Add “Refresh Schema” button (#2943)
In the Settings, you can now add a webhook to get notified when a sync fails
Add destinationSyncModes to connection form
Add tooltips for connection status icons
When displaying the schema for a source, you can now search for table names, thanks to @jamakase
Better feedback UX when manually triggering a sync with “Sync now”
New Connections page where you can see the list of all your connections and their statuses.
New Settings page to update your preferences.
Bugfix where very large schemas caused schema discovery to fail.
Surface the health of each connection so that a user can spot any problems at a glance.
Added support for deduplicating records in the destination using a primary key using incremental dedupe -
A source’s extraction mode (incremental, full refresh) is now decoupled from the destination’s write mode -- so you can repeatedly append full refreshes to get repeated snapshots of data in your source.
New Upgrade all button in Admin to upgrade all your connectors at once
New Cancel job button in Connections Status page when a sync job is running, so you can stop never-ending processes.
Improved the speed of get spec, check connection, and discover schema by migrating to the Temporal workflow engine.
Exposed cancellation for sync jobs in the API (will be exposed in the UI in the next week!).
Bug fix: Fix issue where migration app was OOMing.
Creating and deleting multiple workspaces is now supported via the API. Thanks to @Samuel Gordalina for contributing this feature!
Normalization now supports numeric types with precision greater than 32 bits
Normalization now supports union data types
Support longform text inputs in the UI for cases where you need to preserve formatting on connector inputs like .pem keys
Expose the latest available connector versions in the API
Connector Contributions: All connectors now describe how to contribute to them without having to touch Airbyte’s monorepo build system -- just work on the connector in your favorite dev setup!
Integration with Airflow is here. Thanks to @Marcos Marx, you can now run Airbyte jobs from Airflow directly. A tutorial is on the way and should be coming this week!
Add a prefix for tables, so that tables with the same name don't clobber each other in the destination
We made some progress to address nested tables in our normalization.
Previously, basic normalization would output nested tables as-is and append a number for duplicate tables. For example, Stripe’s nested address fields go from:
After the change, the parent tables are combined with the name of the nested table to show where the nested table originated. This is a breaking change for the consumers of nested tables. Consumers will need to update to point at the new tables.
We now handle nested tables with the normalization steps. Check out the video below to see how it works.
Display Airbyte's version number
Describe schemas using JsonSchema
Better feedback on buttons
Add connector build status dashboard
Support Schema Changes in Sources
Support Import / Export of Airbyte Data in the Admin section of the UI
If Airbyte is closed during a sync the running job is not marked as failed
Airbyte should fail when deployment version doesn't match data version
Upgrade Airbyte Version without losing existing configuration / data
Ability to skip onboarding
Miscellaneous bug fixes:
A long discovery request causes a timeout in the UI type/bug
Out of Memory when replicating large table from MySQL
Increase timeout for long running catalog discovery operations from 3 minutes to 30 minutes to avoid prematurely failing long-running operations
Writing boolean columns to Redshift destination now works correctly
Allow skipping the onboarding flow in the UI
Add the ability to reset a connection's schema when the underlying data source schema changes
Fix UI race condition which showed config for the wrong connector when rapidly choosing between different connector
Fix a bug in MSSQL and Redshift source connectors where custom SQL types weren't being handled correctly. Pull request
Support incremental sync for Salesforce, Slack, and Braintree sources
Gracefully handle invalid nuemric values (e.g NaN or Infinity) in MySQL, MSSQL, and Postgtres DB sources
Fix flashing red sources/destinations fields after success submit
Fix a bug which caused getting a connector's specification to hang indefinitely if the connector docker image failed to download
You can now deploy Airbyte on Kuberbetes **(alpha version)
Support incremental sync for Mixpanel and Hubspot sources
Fixes on connectors:
Fixed a bug in the GitHub connector where the connector didn’t verify the provided API token was granted the correct permissions
Fixed a bug in the Google Sheets connector where rate limits were not always respected
Alpha version of Facebook marketing API v9. This connector is a native Airbyte connector (current is Singer based).
New source: Plaid (contributed by @tgiardina - thanks Thomas!)
New chat app from the web app so you can directly chat with the team for any issues you run into
Debugging has been made easier in the UI, with checks, discover logs, and sync download logs
Support of Kubernetes in local. GKE will come at the next release.
New source: Looker **
Incremental - Append"
We now allow sources to replicate only new or modified data. This enables to avoid re-fetching data that you have already replicated from a source.
The delta from a sync will be appended to the existing data in the data warehouse.
Here are all the details of this feature.
It has been released for 15 connectors, including Postgres, MySQL, Intercom, Zendesk, Stripe, Twilio, Marketo, Shopify, GitHub, and all the destination connectors. We will expand it to all the connectors in the next couple of weeks.
Improve interface for writing python sources (should make writing new python sources easier and clearer).
Add support for running Standard Source Tests with files (making them easy to run for any language a source is written in)
Add ability to reset data for a connection.
Update version of test containers we use to avoid pull issues while running tests.
Fix issue where jobs were not sorted by created at in connection detail view.
New sources: Intercom, Mixpanel, Jira Cloud, Zoom, Drift, Microsoft Teams
New destination: our own Redshift warehouse connector. You can also use this connector for Panoply.
New sources: 8 additional source connectors including Recurly, Twilio, Freshdesk. Greenhouse, Redshift (source), Braintree, Slack, Zendesk Support
Support multiple destinations
New source: Sendgrid
Support basic normalization
New sources: 10 additional source connectors, including Files (CSV, HTML, JSON...), Shopify, MSSQL, Mailchimp
Here is what we are working on right now:
New destination: our own Snowflake warehouse connector
New sources: Facebook Ads, Google Ads.
New sources: Salesforce, GitHub, Google Sheets, Google Analytics, Hubspot, Rest APIs, and MySQL
Integration test suite for sources
Improve build speed
a new Admin section to enable users to add their own connectors, in addition to upgrading the ones they currently use
improve the developer experience (DX) for contributing new connectors with additional documentation and a connector protocol
our own BigQuery warehouse connector
our own Postgres warehouse connector
simplify the process of supporting new Singer taps, ideally make it a 1-day process
This is our very first release after 2 months of work.
New sources: Stripe, Postgres
New destinations: BigQuery, Postgres
Only one destination: we only support one destination in that 1st release, but you will soon be able to add as many as you need.
Logs & monitoring: you can now see your detailed logs
Scheduler: you now have 10 different frequency options for your recurring syncs
Deployment: you can now deploy Airbyte via a simple Docker image, or directly on AWS and GCP
New website: this is the day we launch our website - airbyte.io. Let us know what you think
New documentation: this is the 1st day for our documentation too
New blog: we published a few articles on our startup journey, but also about our vision to making data integrations a commodity.
Stay tuned, we will have new sources and destinations very soon! Don't hesitate to subscribe to our newsletter to receive our product updates and community news.