The GitHub source supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run.
This connector outputs the following full refresh streams:
This connector outputs the following incremental streams:
Only 3 streams from above 12 incremental streams (
issues) are pure incremental meaning that they:
read only new records;
output only new records.
Other 8 incremental streams are also incremental but with one difference, they:
read all records;
output only new records.
Please, consider this behaviour when using those 8 incremental streams because it may affect you API call limits.
We are passing few parameters (
direction) to GitHub in order to filter records and sometimes for large streams specifying very distant
start_date in the past may result in keep on getting error from GitHub instead of records (respective
WARN log message will be outputted). In this case Specifying more recent
start_date may help.
Full Refresh Sync
Incremental - Append Sync
Replicate Incremental Deletes
The Github connector should not run into Github API limitations under normal usage. Please create an issue if you see any rate limit issues that are not automatically retried successfully.
access_token - Github Personal Access Token wih the necessary permissions (described below);
start_date - start date for 3 incremental streams:
repository - Space-delimited list of GitHub repositories/organizations which looks like
<owner>/<repo> <organization>/* <organization_new>/* <owner_new>/<repo_new>.
Note: if you want to specify the organization to receive data from all its repositories, then you should specify it according to the following pattern:
Log into Github and then generate a personal access token.
Your token should have at least the
repo scope. Depending on which streams you want to sync, the user generating the token needs more permissions:
For syncing Collaborators, the user which generates the personal access token must be a collaborator. To become a collaborator, they must be invited by an owner. If there are no collaborators, no records will be synced. Read more about access permissions here.
To sync the Projects stream, the repository must have the Projects feature enabled.
Add option to pull commits from user-specified branches
Don't minimize any output fields & add better error handling
Add caching for all streams
Add reaction streams
Handling empty repository, check method using RepositoryStats stream
Add more streams
Handle negative backoff values
Fix set up validation
Support syncing multiple repositories/organizations
Extended existing schemas with
Fix bug with IssueEvents stream and add handling for rate limiting
Fix schema in the
New Source: GitHub