Changelog

11/17/2021 Summary

Hey Airbyte Community! Let's go over all the changes from v.32.1 and prior! But first, there's an important announcement I need to make about upgrading Airbyte to v.32.1.
⚠️ WARNING ⚠️ Upgrading to v.32.0 is equivalent to a major version bump. If your current version is v.32.0, you must upgrade to v.32.0 first before upgrading to any later version
Keep in mind that this upgrade requires your all of your connector Specs to be retrievable, or Airbyte will fail on startup. You can force delete your connector Specs by setting the VERSION_0_32_0_FORCE_UPGRADE environment variable to true. Steps to specifically check out v.32.0 and details around this breaking change can be found here.
Now back to our regularly scheduled programming.
🎃 Hacktoberfest Submissions 🎃
  • New Destination: ScyllaDB (contributed by Ivica Taseski)
  • New Source: Azure Table Storage (contributed by geekwhocodes)
  • New Source: Linnworks (contributed by Juozas)
  • ✨ Source MySQL: Now has basic performance tests.
  • ✨ Source Salesforce: We now automatically transform and handle incorrect data for the anyType and calculated types.
  • 🐛 IBM Db2 Source: Now handles conversion from DECFLOAT to BigDecimal correctly.
  • 🐛 MSSQL Source: Now handles VARBINARY correctly.
  • 🐛 CockroachDB Source: Improved parsing of various data types.
As usual, thank you so much to our wonderful contributors this week that have made Airbyte into what it is today: Achmad Syarif Hidayatullah, Tuan Nguyen, Ivica Taseski, Hai To, Juozas, gunu, Shadab, Per-Victor Persson, and Harsha Teja Kanna!

11/11/2021 Summary

Time to go over changes from v.30.39! And... let's get another update on Hacktoberfest.
🎃 Hacktoberfest Submissions 🎃
  • New Destination: Cassandra (contributed by Ivica Taseski)
  • New Destination: Pulsar (contributed by Mario Molina)
  • New Source: Confluence (contributed by Tuan Nguyen)
  • New Source: Monday (contributed by Tuan Nguyen)
  • New Source: Commerce Tools (contributed by James Wilson)
  • New Source: Pinterest Marketing (contributed by us!)
  • ✨ Shopify Source: Now supports the FulfillmentOrders and Fulfillments streams.
  • ✨ Greenhouse Source: Now supports the Demographics stream.
  • ✨ Recharge Source: Broken requests should now be re-requested with improved backoff.
  • ✨ Stripe Source: Now supports the checkout_sessions, checkout_sessions_line_item, and promotion_codes streams.
  • ✨ Db2 Source: Now supports SSL.
  • 🐛 We've made some updates to incremental normalization to fix some outstanding issues. Details
  • 🐛 Airbyte Server no longer crashes due to too many open files.
  • 🐛 MSSQL Source: Data type conversion with smalldatetime and smallmoney works correctly now.
  • 🐛 Salesforce Source: anyType fields can now be retrieved properly with the BULK API
  • 🐛 BigQuery-Denormalized Destination: Fixed JSON parsing with $ref fields.
As usual, thank you to our awesome contributors that have done awesome work during the last week: Tuan Nguyen, Harsha Teja Kanna, Aaditya S, James Wilson, Vladimir Remar, Yuhui Shi, Mario Molina, Ivica Taseski, Collin Scangarella, and haoranyu!

11/03/2021 Summary

It's patch notes time. Let's go over the changes from 0.30.24 and before. But before we do, let's get a quick update on how Hacktober is going!
🎃 Hacktoberfest Submissions 🎃
  • New Destination: Elasticsearch (contributed by Jeremy Branham)
  • New Source: Salesloft (contributed by Pras)
  • New Source: OneSignal (contributed by Bo)
  • New Source: Strava (contributed by terencecho)
  • New Source: Lemlist (contributed by Igli Koxha)
  • New Source: Amazon SQS (contributed by Alasdair Brown)
  • New Source: Freshservices (contributed by Tuan Nguyen)
  • New Source: Freshsales (contributed by Tuan Nguyen)
  • New Source: Appsflyer (contributed by Achmad Syarif Hidayatullah)
  • New Source: Paystack (contributed by Foluso Ogunlana)
  • New Source: Sentry (contributed by koji matsumoto)
  • New Source: Retently (contributed by Subhash Gopalakrishnan)
  • New Source: Delighted! (contributed by Rodrigo Parra)
with 18 more currently in review...
🎉 Incremental Normalization is here! 🎉
💎 Basic normalization no longer runs on already normalized data, making it way faster and cheaper.
💎
🎉 Airbyte Compiles on M1 Macs!
Airbyte developers with M1 chips in their MacBooks can now compile the project and run the server. This is a major step towards being able to fully run Airbyte on M1. (contributed by Harsha Teja Kanna)
  • ✨ BigQuery Destination: You can now run transformations in batches, preventing queries from hitting BigQuery limits. (contributed by Andrés Bravo)
  • ✨ S3 Source: Memory and Performance optimizations, also some fancy new PyArrow CSV configuration options.
  • ✨ Zuora Source: Now supports Unlimited as an option for the Data Query Live API.
  • ✨ Clickhouse Source: Now supports SSL and connection via SSH tunneling.
  • 🐛 Oracle Source: Now handles the LONG RAW data type correctly.
  • 🐛 Snowflake Source: Fixed parsing of extreme values for FLOAT and NUMBER data types.
  • 🐛 Hubspot Source: No longer fails due to lengthy URI/URLs.
  • 🐛 Zendesk Source: The chats stream now pulls data past the first page.
  • 🐛 Jira Source: Normalization now works as expected.
As usual, thank you to our awesome contributors that have done awesome work during this productive spooky season: Tuan Nguyen, Achmad Syarif Hidayatullah, Christopher Wu, Andrés Bravo, Harsha Teja Kanna, Collin Scangarella, haoranyu, koji matsumoto, Subhash Gopalakrishnan, Jeremy Branham, Rodrigo Parra, Foluso Ogunlana, EdBizarro, Gergely Lendvai, Rodeoclash, terencecho, Igli Koxha, Alasdair Brown, bbugh, Pras, Bo, Xiangxuan Liu, Hai To, s-mawjee, Mario Molina, SamyPesse, Yuhui Shi, Maciej Nędza, Matt Hoag, and denis-sokolov!

10/20/2021 Summary

It's patch notes time! Let's go over changes from 0.30.16! But before we do... I want to remind everyone that Airbyte Hacktoberfest is currently taking place! For every connector that is merged into our codebase, you'll get $500, so make sure to submit before the hackathon ends on November 19th.
  • 🎉 New Source: WooCommerce (contributed by James Wilson)
  • 🎉 K8s deployments: Worker image pull policy is now configurable (contributed by Mario Molina)
  • ✨ MSSQL destination: Now supports basic normalization
  • 🐛 LinkedIn Ads source: Analytics streams now work as expected.
We've had a lot of contributors over the last few weeks, so I'd like to thank all of them for their efforts: James Wilson, Mario Molina, Maciej Nędza, Pras, Tuan Nguyen, Andrés Bravo, Christopher Wu, gunu, Harsha Teja Kanna, Jonathan Stacks, darian, Christian Gagnon, Nicolas Moreau, Matt Hoag, Achmad Syarif Hidayatullah, s-mawjee, SamyPesse, heade, zurferr, denis-solokov, and aristidednd!

09/29/2021 Summary

It's patch notes time, let's go over the changes from our new minor version, v0.30.0. As usual, bug fixes are in the thread.
  • New source: LinkedIn Ads
  • New source: Kafka
  • New source: Lever Hiring
  • 🎉 New License: Nothing changes for users of Airbyte/contributors. You just can't sell your own Airbyte Cloud!
  • 💎 New API endpoint: You can now call connections/search in the web backend API to search sources and destinations. (contributed by Mario Molina)
  • 💎 K8s: Added support for ImagePullSecrets for connector images.
  • 💎 MSSQL, Oracle, MySQL sources & destinations: Now support connection via SSH (Bastion server)
  • ✨ MySQL destination: Now supports connection via TLS/SSL
  • ✨ BigQuery (denormalized) destination: Supports reading BigQuery types such as date by reading the format field (contributed by Nicolas Moreau)
  • ✨ Hubspot source: Added contacts associations to the deals stream.
  • ✨ GitHub source: Now supports pulling commits from user-specified branches.
  • ✨ Google Search Console source: Now accepts admin email as input when using a service account key.
  • ✨ Greenhouse source: Now identifies API streams it has access to if permissions are limited.
  • ✨ Marketo source: Now Airbyte native.
  • ✨ S3 source: Now supports any source that conforms to the S3 protocol (Non-AWS S3).
  • ✨ Shopify source: Now reports pre_tax_price on the line_items stream if you have Shopify Plus.
  • ✨ Stripe source: Now actually uses the mandatory start_date config field for incremental syncs.
  • 🏗 Python CDK: Now supports passing custom headers to the requests in OAuth2, enabling token refresh calls.
  • 🏗 Python CDK: Parent streams can now be configured to cache data for their child streams.
  • 🏗 Python CDK: Now has a Transformer class that can cast record fields to the data type expected by the schema.
  • 🐛 Amplitude source: Fixed schema for date-time objects.
  • 🐛 Asana source: Schema fixed for the sections, stories, tasks, and users streams.
  • 🐛 GitHub source: Added error handling for streams not applicable to a repo. (contributed by Christopher Wu)
  • 🐛 Google Search Console source: Verifies access to sites when performing the connection check.
  • 🐛 Hubspot source: Now conforms to the V3 API, with streams such as owners reflecting the new fields.
  • 🐛 Intercom source: Fixed data type for the updated_at field. (contributed by Christian Gagnon)
  • 🐛 Iterable source: Normalization now works as expected.
  • 🐛 Pipedrive source: Schema now reflects the correct types for date/time fields.
  • 🐛 Stripe source: Incorrect timestamp formats removed for coupons and subscriptions streams.
  • 🐛 Salesforce source: You can now sync more than 10,000 records with the Bulk API.
  • 🐛 Snowflake destination: Now accepts any date-time format with normalization.
  • 🐛 Snowflake destination: Inserts are now split into batches to accommodate for large data loads.
Thank you to our awesome contributors. Y'all are amazing: Mario Molina, Pras, Vladimir Remar, Christopher Wu, gunu, Juliano Benvenuto Piovezan, Brian M, Justinas Lukasevicius, Jonathan Stacks, Christian Gagnon, Nicolas Moreau, aristidednd, camro, minimax75, peter-mcconnell, and sashkalife!

09/16/2021 Summary

Now let's get to the 0.29.19 changelog. As with last time, bug fixes are in the thread!
  • New Destination: Databricks 🎉
  • New Source: Google Search Console
  • New Source: Close.com
  • 🏗 Python CDK: Now supports auth workflows involving query params.
  • 🏗 Java CDK: You can now run the connector gradle build script on Macs with M1 chips! (contributed by @Harsha Teja Kanna)
  • 💎 Google Ads source: You can now specify user-specified queries in GAQL.
  • ✨ GitHub source: All streams with a parent stream use cached parent stream data when possible.
  • ✨ Shopify source: Substantial performance improvements to the incremental sync mode.
  • ✨ Stripe source: Now supports the PaymentIntents stream.
  • ✨ Pipedrive source: Now supports the Organizations stream.
  • ✨ Sendgrid source: Now supports the SingleSendStats stream.
  • ✨ Bing Ads source: Now supports the Report stream.
  • ✨ GitHub source: Now supports the Reactions stream.
  • ✨ MongoDB source: Now Airbyte native!
  • 🐛 Facebook Marketing source: Numeric values are no longer wrapped into strings.
  • 🐛 Facebook Marketing source: Fetching conversion data now works as expected. (contributed by @Manav)
  • 🐛 Keen destination: Timestamps are now parsed correctly.
  • 🐛 S3 destination: Parquet schema parsing errors are fixed.
  • 🐛 Snowflake destination: No longer syncs unnecessary tables with S3.
  • 🐛 SurveyMonkey source: Cached responses are now decoded correctly.
  • 🐛 Okta source: Incremental sync now works as expected.
Also, a quick shout out to Jinni Gu and their team who made the DynamoDB destination that we announced last week!
As usual, thank you to all of our contributors: Harsha Teja Kanna, Manav, Maciej Nędza, mauro, Brian M, Iakov Salikov, Eliziario (Marcos Santos), coeurdestenebres, and mohammadbolt.

09/09/2021 Summary

We're going over the changes from 0.29.17 and before... and there's a lot of big improvements here, so don't miss them!
New Source: Facebook Pages New Destination: MongoDB New Destination: DynamoDB
  • 🎉 You can now send notifications via webhook for successes and failures on Airbyte syncs. (This is a massive contribution by @Pras, thank you)
    🎉
  • 🎉 Scheduling jobs and worker jobs are now separated, allowing for workers to be scaled horizontally.
  • 🎉 When developing a connector, you can now preview what your spec looks like in real time with this process.
  • 🎉 Oracle destination: Now has basic normalization.
  • 🎉 Add XLSB (binary excel) support to the Files source (contributed by Muutech).
  • 🎉 You can now properly cancel K8s deployments.
  • ✨ S3 source: Support for Parquet format.
  • ✨ Github source: Branches, repositories, organization users, tags, and pull request stats streams added (contributed by @Christopher Wu).
  • ✨ BigQuery destination: Added GCS upload option.
  • ✨ Salesforce source: Now Airbyte native.
  • ✨ Redshift destination: Optimized for performance.
  • 🏗 CDK:
    🎉
    We’ve released a tool to generate JSON Schemas from OpenAPI specs. This should make specifying schemas for API connectors a breeze!
    🎉
  • 🏗 CDK: Source Acceptance Tests now verify that connectors correctly format strings which are declared as using date-time and date formats.
  • 🏗 CDK: Add private options to help in testing: _limit and _page_size are now accepted by any CDK connector to minimze your output size for quick iteration while testing.
  • 🐛 Fixed a bug that made it possible for connector definitions to be duplicated, violating uniqueness.
  • 🐛 Pipedrive source: Output schemas no longer remove timestamp from fields.
  • 🐛 Github source: Empty repos and negative backoff values are now handled correctly.
  • 🐛 Harvest source: Normalization now works as expected.
  • 🐛 All CDC sources: Removed sleep logic which caused exceptions when loading data from high-volume sources.
  • 🐛 Slack source: Increased number of retries to tolerate flaky retry wait times on the API side.
  • 🐛 Slack source: Sync operations no longer hang indefinitely.
  • 🐛 Jira source: Now uses updated time as the cursor field for incremental sync instead of the created time.
  • 🐛 Intercom source: Fixed inconsistency between schema and output data.
  • 🐛 HubSpot source: Streams with the items property now have their schemas fixed.
  • 🐛 HubSpot source: Empty strings are no longer handled as dates, fixing the deals, companies, and contacts streams.
  • 🐛 Typeform source: Allows for multiple choices in responses now.
  • 🐛 Shopify source: The type for the amount field is now fixed in the schema.
  • 🐛 Postgres destination: \u0000(NULL) value processing is now fixed.
As usual... thank you to our wonderful contributors this week: Pras, Christopher Wu, Brian M, yahu98, Michele Zuccala, jinnig, and luizgribeiro!

09/01/2021 Summary

Got the changes from 0.29.13... with some other surprises!
  • 🔥 There's a new way to create Airbyte sources! The team at Faros AI has created a Javascript/Typescript CDK which can be found here and in our docs here. This is absolutely awesome and give a huge thanks to Chalenge Masekera, Christopher Wu, eskrm, and Matthew Tovbin!
  • ✨ New Destination: Azure Blob Storage ✨
New Source: Bamboo HR (contributed by @Oren Haliva) New Source: BigCommerce (contributed by @James Wilson) New Source: Trello New Source: Google Analytics V4 New Source: Amazon Ads
  • 💎 Alpine Docker images are the new standard for Python connectors, so image sizes have dropped by around 100 MB!
  • ✨ You can now apply tolerations for Airbyte Pods on K8s deployments (contributed by @Pras).
  • 🐛 Shopify source: Rate limit throttling fixed.
  • 📚 We now have a doc on how to deploy Airbyte at scale. Check it out here!
  • 🏗 Airbyte CDK: You can now ignore HTTP status errors and override retry parameters.
As usual, thank you to our awesome contributors: Oren Haliva, Pras, James Wilson, and Muutech.

08/26/2021 Summary

New Source: Short.io (contributed by @Apostol Tegko)
  • 💎 GitHub source: Added support for rotating through multiple API tokens!
  • ✨ Syncs are now scheduled with a 3 day timeout (contributed by @Vladimir Remar).
  • ✨ Google Ads source: Added UserLocationReport stream (contributed by @Max Krog).
  • ✨ Cart.com source: Added the order_items stream.
  • 🐛 Postgres source: Fixed out-of-memory issue with CDC interacting with large JSON blobs.
  • 🐛 Intercom source: Pagination now works as expected.
As always, thank you to our awesome community contributors this week: Apostol Tegko, Vladimir Remar, Max Krog, Pras, Marco Fontana, Troy Harvey, and damianlegawiec!

08/20/2021 Summary

Hey Airbyte community, we got some patch notes for y'all. Here's all the changes we've pushed since the last update.
  • New Source: S3/Abstract Files
  • New Source: Zuora
  • New Source: Kustomer
  • New Source: Apify
  • New Source: Chargebee
  • New Source: Bing Ads
New Destination: Keen
  • ✨ Shopify source: The status property is now in the Products stream.
  • ✨ Amazon Seller Partner source: Added support for GET_MERCHANT_LISTINGS_ALL_DATA and GET_FBA_INVENTORY_AGED_DATA stream endpoints.
  • ✨ GitHub source: Existing streams now don't minify the user property.
  • ✨ HubSpot source: Updated user-defined custom field schema generation.
  • ✨ Zendesk source: Migrated from Singer to the Airbyte CDK.
  • ✨ Amazon Seller Partner source: Migrated to the Airbyte CDK.
  • 🐛 Shopify source: Fixed the products schema to be in accordance with the API.
  • 🐛 S3 source: Fixed bug where syncs could hang indefinitely.
And as always... we'd love to shout out the awesome contributors that have helped push Airbyte forward. As a reminder, you can now see your contributions publicly reflected on our contributors page.
Thank you to Rodrigo Parra, Brian Krausz, Max Krog, Apostol Tegko, Matej Hamas, Vladimir Remar, Marco Fontana, Nicholas Bull, @mildbyte, @subhaklp, and Maciej Nędza!

07/30/2021 Summary

For this week's update, we got... a few new connectors this week in 0.29.0. We found that a lot of sources can pull data directly from the underlying db instance, which we naturally already supported.
  • New Source: PrestaShop ✨
  • New Source: Snapchat Marketing ✨
  • New Source: Drupal
  • New Source: Magento
  • New Source: Microsoft Dynamics AX
  • New Source: Microsoft Dynamics Customer Engagement
  • New Source: Microsoft Dynamics GP
  • New Source: Microsoft Dynamics NAV
  • New Source: Oracle PeopleSoft
  • New Source: Oracle Siebel CRM
  • New Source: SAP Business One
  • New Source: Spree Commerce
  • New Source: Sugar CRM
  • New Source: Wordpress
  • New Source: Zencart
  • 🐛 Shopify source: Fixed the products schema to be in accordance with the API
  • 🐛 BigQuery source: No longer fails with nested array data types.
View the full release highlights here: Platform, Connectors
And as always, thank you to our wonderful contributors: Madison Swain-Bowden, Brian Krausz, Apostol Tegko, Matej Hamas, Vladimir Remar, Oren Haliva, satishblotout, jacqueskpoty, wallies

07/23/2021 Summary

What's going on? We just released 0.28.0 and here's the main highlights.
  • New Destination: Google Cloud Storage ✨
  • New Destination: Kafka ✨ (contributed by @Mario Molina)
  • New Source: Pipedrive
  • New Source: US Census (contributed by @Daniel Mateus Pires (Earnest Research))
  • ✨ Google Ads source: Now supports Campaigns, Ads, AdGroups, and Accounts streams.
  • ✨ Stripe source: All subscription types (including expired and canceled ones) are now returned.
  • 🐛 Facebook source: Improved rate limit management
  • 🐛 Square source: The send_request method is no longer broken due to CDK changes
  • 🐛 MySQL destination: Does not fail on columns with JSON data now.
View the full release highlights here: Platform, Connectors
And as always, thank you to our wonderful contributors: Mario Molina, Daniel Mateus Pires (Earnest Research), gunu, Ankur Adhikari, Vladimir Remar, Madison Swain-Bowden, Maksym Pavlenok, Sam Crowder, mildbyte, avida, and gaart

07/16/2021 Summary

As for our changes this week...
  • New Source: Zendesk Sunshine
  • New Source: Dixa
  • New Source: Typeform
  • 💎 MySQL destination: Now supports normalization!
  • 💎 MSSQL source: Now supports CDC (Change Data Capture)
  • ✨ Snowflake destination: Data coming from Airbyte is now identifiable
  • 🐛 GitHub source: Now uses the correct cursor field for the IssueEvents stream
  • 🐛 Square source: The send_request method is no longer broken due to CDK changes
View the full release highlights here: Platform, Connectors
As usual, thank you to our awesome community contributors this week: Oliver Meyer, Varun, Brian Krausz, shadabshaukat, Serhii Lazebnyi, Juliano Benvenuto Piovezan, mildbyte, and Sam Crowder!

07/09/2021 Summary

  • New Source: PayPal Transaction
  • New Source: Square
  • New Source: SurveyMonkey
  • New Source: CockroachDB
  • New Source: Airbyte-Native GitHub
  • New Source: Airbyte-Native GitLab
  • New Source: Airbyte-Native Twilio
  • ✨ S3 destination: Now supports anyOf, oneOf and allOf schema fields.
  • ✨ Instagram source: Migrated to the CDK and has improved error handling.
  • ✨ Shopify source: Add support for draft orders.
  • ✨ K8s Deployments: Now support logging to GCS.
  • 🐛 GitHub source: Fixed issue with locked breaking normalization of the pull_request stream.
  • 🐛 Okta source: Fix endless loop when syncing data from logs stream.
  • 🐛 PostgreSQL source: Fixed decimal handling with CDC.
  • 🐛 Fixed random silent source failures.
  • 📚 New document on how the CDK handles schemas.
  • 🏗️ Python CDK: Now allows setting of network adapter args on outgoing HTTP requests.
View the full release highlights here: Platform, Connectors
As usual, thank you to our awesome community contributors this week: gunu, P.VAD, Rodrigo Parra, Mario Molina, Antonio Grass, sabifranjo, Jaime Farres, shadabshaukat, Rodrigo Menezes, dkelwa, Jonathan Duval, and Augustin Lafanechère.

07/01/2021 Summary

  • New Destination: Google PubSub
  • New Source: AWS CloudTrail
The risks and issues with upgrading Airbyte are now gone...
  • 🎉 Airbyte automatically upgrades versions safely at server startup 🎉
  • 💎 Logs on K8s are now stored in Minio by default, no S3 bucket required
  • ✨ Looker Source: Supports the Run Look output stream
  • ✨ Slack Source: is now Airbyte native!
  • 🐛 Freshdesk Source: No longer fails after 300 pages
  • 📚 New tutorial on building Java destinations
Starting from next week, our weekly office hours will now become demo days! Drop by to get sneak peeks and new feature demos.
  • We added the #careers channel, so if you're hiring, post your job reqs there!
  • We added a #understanding-airbyte channel to mirror this section on our docs site. Ask any questions about our architecture or protocol there.
  • We added a #contributing-to-airbyte channel. A lot of people ask us about how to contribute to the project, so ask away there!
View the full release highlights here: Platform, Connectors
As usual, thank you to our awesome community contributors this week: Harshith Mullapudi, Michael Irvine, and sabifranjo.

06/24/2021 Summary

  • New Source: IBM Db2
  • 💎 We now support Avro and JSONL output for our S3 destination! 💎
  • 💎 Brand new BigQuery destination flavor that now supports denormalized STRUCT types.
  • ✨ Looker source now supports self-hosted instances.
  • ✨ Facebook Marketing source is now migrated to the CDK, massively improving async job performance and error handling.
View the full connector release notes here.
As usual, thank you to some of our awesome community contributors this week: Harshith Mullapudi, Tyler DeLange, Daniel Mateus Pires, EdBizarro, Tyler Schroeder, and Konrad Schlatte!

06/18/2021 Summary

  • New Source: Snowflake
  • 💎 We now support custom dbt transformations! 💎
  • ✨ We now support configuring your destination namespace at the table level when setting up a connection!
  • ✨ The S3 destination now supports Minio S3 and Parquet output!
View the full release notes here: Platform, Connectors
As usual, thank you to some of our awesome community contributors this week: Tyler DeLange, Mario Molina, Rodrigo Parra, Prashanth Patali, Christopher Wu, Itai Admi, Fred Reimer, and Konrad Schlatte!

06/10/2021 Summary

  • New Destination: S3!!
  • New Sources: Harvest, Amplitude, Posthog
  • 🐛 Ensure that logs from threads created by replication workers are added to the log file.
  • 🐛 Handle TINYINT(1) and BOOLEAN correctly and fix target file comparison for MySQL CDC.
  • Jira source: now supports all available entities in Jira Cloud.
  • 📚 Added a troubleshooting section, a gradle cheatsheet, a reminder on what the reset button does, and a refresh on our docs best practices.

Connector Development:

  • Containerized connector code generator
  • Added JDBC source connector bootstrap template.
  • Added Java destination generator.
View the full release notes highlights here: Platform, Connectors
As usual, thank you to some of our awesome community contributors this week (I've noticed that we've had more contributors to our docs, which we really appreciate). Ping, Harshith Mullapudi, Michael Irvine, Matheus di Paula, jacqueskpoty and P.VAD.

Overview

Airbyte is comprised of 2 parts:

Airbyte Platform Releases

Production v. Dev Releases

The "production" version of Airbyte is the version of the app specified in .env. With each production release, we update the version in the .env file. This version will always be available for download on DockerHub. It is the version of the app that runs when a user runs docker-compose up.
The "development" version of Airbyte is the head of master branch. It is the version of the app that runs when a user runs ./gradlew build && VERSION=dev docker compose up.

Production Release Schedule

Scheduled Releases

Airbyte currently releases a new minor version of the application on a weekly basis. Generally this weekly release happens on Monday or Tuesday.

Hotfixes

Airbyte releases a new version whenever it discovers and fixes a bug that blocks any mission critical functionality.
Mission Critical
e.g. Non-ASCII characters break the Salesforce source.
Non-Mission Critical
e.g. Buttons in the UI are offset.

Unscheduled Releases

We will often release more frequently than the weekly cadence if we complete a feature that we know that a user is waiting on.

Development Release Schedule

As soon as a feature is on master, it is part of the development version of Airbyte. We merge features as soon as they are ready to go (have been code reviewed and tested). We attempt to keep the development version of the app working all the time. We are iterating quickly, however, and there may be intermittent periods where the development version is broken.
If there is ever a feature that is only on the development version, and you need it on the production version, please let us know. We are very happy to do ad-hoc production releases if it unblocks a specific need for one of our users.

Airbyte Connector Releases

Each connector is tracked with its own version. These versions are separate from the versions of Airbyte Platform. We generally will bump the version of a connector anytime we make a change to it. We rely on a large suite of tests to make sure that these changes do not cause regressions in our connectors.
When we updated the version of a connector, we usually update the connector's version in Airbyte Platform as well. Keep in mind that you might not see the updated version of that connector in the production version of Airbyte Platform until after a production release of Airbyte Platform.
Last modified 15d ago