Configure sync sources¶

Sync sources pull documents from external services (Google Drive, S3/MinIO, a public website, a Git repository, a SharePoint site or a OneDrive) into knowledge collections on their own. Each source stores a connector type, a target collection, connector-specific settings, a sync mode, an optional schedule, and the id of the vault secret that authenticates it - a website needs none.

When a sync runs, the connector lists remote files, downloads them to a temporary directory, and feeds them through the standard ingestion pipeline (parse, chunk, embed, store). When the listing is complete, documents the source brought in earlier and no longer lists are removed. A SyncLog entry records the outcome of every sync operation.

Architecture at a glance¶

Component	Location	Role
`BaseSyncConnector`	`app/services/rag/connectors/__init__.py`	Abstract base for all connectors
`RemoteFile`	`app/services/rag/connectors/__init__.py`	Pydantic model describing a remote file
`CONNECTOR_REGISTRY`	`app/services/rag/connectors/__init__.py`	Maps connector type strings to classes
`SyncSource` (DB model)	`app/db/models/sync_source.py`	Persists source configurations
`SyncLog` (DB model)	`app/db/models/sync_log.py`	Tracks individual sync operations
`SyncSourceService`	`app/services/sync_source.py`	Business logic for CRUD + trigger
RAG CLI commands	`app/commands/rag.py`	CLI interface for managing sources
RAG API routes	`app/api/routes/v1/rag.py`	REST API for managing sources

Quick start -- CLI¶

List available connector types¶

# Shows all registered connectors (e.g. gdrive, s3, git, sharepoint)
uv run agenticos cmd rag-sources

Add a Google Drive source -- sync every 2 hours¶

uv run agenticos cmd rag-source-add \
  --name "Legal docs" \
  --type gdrive \
  --org 0c8f2b1e-... \
  --collection legal \
  --config '{"folder_id": "1abc123def", "include_subfolders": true}' \
  --sync-mode new_only \
  --schedule 120

Add an S3 source -- manual sync only¶

uv run agenticos cmd rag-source-add \
  --name "Marketing" \
  --type s3 \
  --org 0c8f2b1e-... \
  --collection marketing \
  --config '{"bucket": "my-docs", "prefix": "marketing/"}' \
  --sync-mode full \
  --schedule 0

Add a Git source -- a repository's docs, nightly¶

uv run agenticos cmd rag-source-add \
  --name "Handbook" \
  --type git \
  --org 0c8f2b1e-... \
  --collection handbook \
  --config '{"repository_url": "https://github.com/acme/handbook.git", "branch": "main", "path_prefix": "docs"}' \
  --sync-mode new_only \
  --schedule 1440

Then choose its access token as the source's credential in the UI, or send secret_id with a PATCH — see Git repository setup.

Add a SharePoint source -- one library folder, every 6 hours¶

uv run agenticos cmd rag-source-add \
  --name "HR policies" \
  --type sharepoint \
  --org 0c8f2b1e-... \
  --collection hr \
  --secret-id <vault-secret-id> \
  --config '{"site_url": "https://contoso.sharepoint.com/sites/HR", "folder_path": "Policies"}' \
  --sync-mode new_only \
  --schedule 360

--secret-id is the Microsoft Entra app in the organization's Vault — see SharePoint and OneDrive setup.

Trigger sync manually¶

# Sync a single source by ID
uv run agenticos cmd rag-source-sync <source-id>

# Sync all active sources
uv run agenticos cmd rag-source-sync --all

Remove a source¶

uv run agenticos cmd rag-source-remove <source-id>

The <source-id> is a UUID printed when you create the source and shown in the rag-sources listing.

Quick start -- UI¶

Navigate to Knowledge Base and open the Sync tab.
Click "+ Add Source".
Select a connector type (Google Drive, S3, Website, Git repository, SharePoint & OneDrive). The form fields are generated from the JSON Schema of the connector's CONFIG_MODEL. A website has no credential step.
Fill in the connector-specific config fields (e.g. folder ID, bucket name).
Choose a target collection, sync mode, and schedule interval.
Click "Create Source".
Use the "Sync Now" button to trigger an immediate sync, or wait for the schedule to fire automatically.

To change a source later, click the pencil on its row. The same wizard opens with the source's values filled in. The connector cannot be changed, and "Save Changes" sends only the fields you changed. The source stays filed under the knowledge base it is on.

The UI calls the same REST API documented below, so anything you can do in the UI you can also do with curl or any HTTP client.

Sync modes¶

Mode	Behavior
`full`	Re-sync everything. All files are (re-)ingested, existing documents replaced.
`new_only`	Add new files + update changed files. Uses SHA-256 hash to detect changes — unchanged files are skipped.
`update_only`	Only update files already in the collection. New files are skipped. Uses SHA-256 hash to skip unchanged files.

new_only for most workflows

It adds new files and updates modified ones while skipping unchanged files, which is the fastest incremental sync. update_only refreshes existing documents without adding new ones; full is a clean re-import every time.

What a sync removes¶

In every mode, a sync removes the documents its source brought in earlier and no longer lists: a page taken off the site, a file deleted from the Drive folder, an object removed from the bucket. The sync log counts them under removed.

It removes nothing unless the listing was complete. A crawl that stopped at its page limit, or could not read one of the pages, has not seen what it does not list. That run keeps every document and says so in the sync log's message. The next sync with a complete listing removes what is gone. A document that could not be removed counts as a failed file, and the next sync tries again.

One sync of a source runs at a time. A sync started while another sync of the same source is still running does not start, and its log says so.

Only the source's own documents are removed. An upload, or a document another source brought into the same collection, is never touched. When two sources on one collection list the same document, it stays until both stop listing it. A document ingested before its source recorded this (September 2026) is kept until a sync of the source lists it again.

What a second sync does¶

A sync after the first one does as little as the source lets it:

An unchanged file costs a download, not an embedding. Its SHA-256 matches the stored document's, so it is counted as skipped and never parsed or embedded again.
An unchanged source costs one request. A connector that can say what its whole content is at — a Git branch's head commit, a SharePoint library's change feed — records that after every run that finished with nothing failed. The next new_only or update_only run that finds the same value, under the same configuration, stops before it lists anything: its log shows no files processed, and it removes nothing. Changing the configuration, the collection or the mode makes the next run read everything again, and full never stops early.
A file a stopped sync left half-done is put right. A worker that stopped after storing a file's vectors, and before recording them, leaves them untracked. The next sync of that source deletes what nothing tracks and ingests the file again, and it does not stop early while such a file is waiting.

A run with a failed file records no state, so the next run reads the source in full and retries it.

Schedule¶

The schedule_minutes field controls how often the source syncs automatically:

Value	Meaning
`0` (or `null`)	Manual only -- trigger via CLI or UI
`30`	Every 30 minutes
`120`	Every 2 hours
`1440`	Once per day

A schedule needs the worker

The dispatch-due-syncs schedule wakes every 60 seconds and queues a sync for each source that has come due, so schedule_minutes does nothing without the worker container make dev starts. Without it, a manual trigger (CLI, API or the UI) queues its sync too, and the sync waits for a worker as well. A source never has two syncs running at once.

Google Drive setup¶

1. Create a service account¶

Go to the Google Cloud Console.
Create a new project (or select an existing one).
Enable the Google Drive API.
Go to IAM & Admin > Service Accounts and create a new service account.
Create a JSON key for the service account and download it.

Open Google Drive and navigate to the folder you want to sync.
Click Share and add the service account email address (it looks like name@project.iam.gserviceaccount.com).
Grant at least Viewer access.

3. Give the source the key¶

Paste the contents of the JSON key file into the source's Service Account JSON field. A gdrive source runs on the credential its own configuration carries and on nothing else — there is no deployment-wide fallback, because one would let a source's folder_id decide what is listed under the operator's service account.

GOOGLE_DRIVE_CREDENTIALS_FILE in .env is for the rag-sync-gdrive CLI command only.

4. Get the folder ID¶

The folder ID is the last segment of the Google Drive folder URL:

https://drive.google.com/drive/folders/1abc123def456ghi
                                        ^^^^^^^^^^^^^^^
                                        This is the folder ID

5. Google Drive connector config fields¶

Field	Type	Required	Default	Description
`folder_id`	string	Yes	--	Google Drive folder ID from the URL
`include_subfolders`	boolean	No	`true`	Recursively include files from subfolders

The service account itself is not a config field. Add it to the Vault as a gcp_service_account credential and point the source at it with secret_id: it is stored once and referenced by every source that needs it, rather than pasted into each one (#937). Posting it under config is refused.

A folder_id may hold only what Google issues — letters, digits, - and _. Anything else is refused when the source is created, because the id is interpolated into the Drive query and a single quote in it widens what the query lists.

Google Docs, Sheets, and Slides are automatically exported to portable formats (PDF, XLSX, PPTX) during download. A file whose Drive name contains path separators is written as one file inside the sync directory, never at the path its name spells.

S3 / MinIO setup¶

1. Configure the environment¶

Add the following variables to your .env:

S3_RAG_ENDPOINT=https://s3.amazonaws.com   # or your MinIO URL, e.g. http://localhost:9000
S3_RAG_ACCESS_KEY=your-access-key
S3_RAG_SECRET_KEY=your-secret-key
S3_RAG_REGION=us-east-1                    # required for AWS, optional for MinIO

For MinIO, the endpoint is typically http://minio:9000 (Docker) or http://localhost:9000 (local).

A source's own endpoint_url is checked like a web source's start URL, when you save the source and again when a sync starts: a loopback, link-local or cloud-metadata address is always refused, and an endpoint on the private network, such as a MinIO container, needs its host listed in OUTBOUND_PRIVATE_ALLOWLIST. The operator's S3_RAG_ENDPOINT is not checked.

2. S3 connector config fields¶

Field	Type	Required	Default	Description
`bucket`	string	Yes	--	S3 bucket name
`prefix`	string	No	`""`	Key prefix to limit sync scope (e.g. `documents/legal/`). Leave empty for the entire bucket.

Website setup¶

A web source reads a public website, usually a product's documentation site. It needs no credential and no vault entry. Give it a start URL, and it either follows links from that page or reads the pages a sitemap lists.

uv run agenticos cmd rag-source-add \
  --name "Product docs" \
  --type web \
  --org 0c8f2b1e-... \
  --collection product-docs \
  --config '{"root_url": "https://docs.example.com/guide/", "max_depth": 3}' \
  --sync-mode new_only \
  --schedule 1440

Website connector config fields¶

Field	Type	Required	Default	Description
`root_url`	string	Yes	--	The page the crawl starts from. Its host is the only host the source reads.
`max_depth`	integer	No	`2`	How many links away from the start URL to follow, `0` to `10`. `0` reads the start page only.
`path_prefix`	string	No	the start URL's folder	Only pages whose path starts with this are read. `https://docs.example.com/guide/intro` reads `/guide/` by default; set `/` for the whole host.
`sitemap_url`	string	No	--	Read the pages this sitemap lists instead of following links. It must be on the start URL's host, and use `https://` when the start URL does. A sitemap index is followed to its sitemaps.
`max_pages`	integer	No	`500`	The crawl stops after reading this many pages, `1` to `5000`.

What bounds a crawl¶

One host and one path. Links to other hosts, and to paths outside path_prefix, are not followed. A redirect that leaves them is not followed either. A start URL on https:// is never left for http://: a link or a redirect to a cleartext page is not followed.
The deployment's network is out of reach. Every request - robots.txt, the sitemap, each page and each redirect - is checked against the same SSRF policy as webhooks and MCP servers. It is sent to the address that passed the check. A start URL that resolves to a loopback, link-local or cloud-metadata address is always refused when you save the source. One that resolves to a private address is refused too, unless the operator lists its host in OUTBOUND_PRIVATE_ALLOWLIST.
robots.txt is obeyed for sitemaps and pages, including Crawl-delay up to ten seconds. The crawler identifies itself as AgenticOS-Crawler. It waits at least half a second between requests, and a page that says noindex or nofollow is honoured. A page that a sitemap still lists after it says noindex, or after it is gone, is removed from the collection.
Size and time. A page larger than 5 MB is not read. The crawl stops at max_pages. A sync stops reading the site after six hours, and a sync that stopped removes nothing.

Each page is stored as a Markdown document holding its text and the URL it came from, without its query string. Navigation, headers, footers and scripts are left out. A page is re-embedded only when its text changes. A new build stamp or tracking script in the markup does not count as a change.

Who can read what it imports¶

A website source has no credential, so its reach is what the site shows to anyone on the internet. It never gets past a login. Everything it imports is searchable by everyone who can search the collection it feeds, as with any other source. See who ends up able to read what a source ingested.

Only HTML pages are imported. A PDF or other file linked from a page is not downloaded.

Git repository setup¶

A git source reads a repository's documentation over HTTPS — GitHub, GitLab or any other host that serves git over HTTPS. It needs the clone URL and an access token, not either platform's API.

1. Issue a token for the one repository¶

A token's reach is the source's reach. Everything the source ingests becomes searchable by whoever can read the collection, so a token that can read every private repository its owner can is a token that can publish all of them to that audience. See who ends up able to read what a source ingested.

GitHub: a fine-grained personal access token, Only select repositories, with the one repository, and Contents: Read-only as its only permission.
GitLab: a project access token on the one project, role Reporter, scope read_repository only.

Give it an expiry date. When it expires, the source's next sync fails with the repository refused the source's token, and the fix is a new token in the same vault secret.

2. Add it to the Vault¶

Add the token to the Vault as a Git access token, with the host it belongs to: github.com, gitlab.com, or your own server such as git.example.com:8443. A host with non-ASCII letters is entered in its encoded form, such as xn--bcher-kva.example for bücher.example. Then choose the token on the source's credential step. It is sent as an HTTP Authorization header, never in the URL, and never in a command line another process can read.

The host is the token's, not the source's. Whoever edits a source chooses its repository URL, and a token is sent only to the host it was added with. So editing a source cannot aim the organization's token at another server, and no other kind of key, such as a model provider's API key, can be chosen for a Git source at all.

3. Git connector config fields¶

Field	Type	Required	Default	Description
`repository_url`	string	Yes	--	The HTTPS clone URL, e.g. `https://github.com/acme/handbook.git`. No user name or token in it.
`branch`	string	No	`main`	The branch to read.
`path_prefix`	string	No	--	A directory inside the repository, e.g. `docs`. Leave empty for the whole repository.
`include`	list of strings	No	`*/.md`, `*/.txt`	Which files to ingest, as `.gitignore`-style patterns relative to `path_prefix`.

The default is documentation, not the whole tree: a repository's source code is not a corpus, and ingesting it fills a knowledge base with code nobody asked to search. Add a pattern such as **/*.pdf for another format the collection's parser reads. A pattern cannot start with !.

Each file is a document whose address is git://<host>/<owner>/<repo>@<branch>/<path>, with :<port> after the host when the port is not 443. The branch is part of the address, so two sources reading two branches of one repository into one collection keep separate documents.

4. What a sync transfers¶

The first request of every sync is git ls-remote for the branch — about a kilobyte. When the head commit has not moved since the last clean run, the sync stops there. When it has moved, the connector makes a shallow, partial, sparse clone: one commit, and only the files the include patterns match. A monorepo's documentation therefore costs its documentation, not its source tree.

Before the clone writes anything to the worker's disk, the connector measures every file it would write. A file over the knowledge base's document cap (MAX_UPLOAD_SIZE_MB, 50 MB by default), or more than 512 MB of files in all, is refused, and nothing is written.

Symbolic links and submodules are not followed, and a link is not ingested as a document.

Network rules¶

The URL must be https://. Its host is resolved once and checked like any other address a tenant chooses: a host that resolves to a private, loopback or link-local address is refused when the source is saved and again when it syncs, and git connects only to the addresses that check approved. The operator can admit a host on the deployment's network with OUTBOUND_PRIVATE_ALLOWLIST. Redirects are not followed. A deployment behind an egress proxy (HTTPS_PROXY) keeps using it; the proxy then resolves the host itself.

The worker image ships git. A worker built from another image needs git 2.37 or newer on its PATH.

SharePoint and OneDrive setup¶

A sharepoint source reads one document library through Microsoft Graph: a SharePoint site's library, or a person's OneDrive, which Microsoft 365 keeps as a site of its own. It can read the whole library or one folder in it. It signs in as a Microsoft Entra app registration that an administrator of your tenant creates once.

1. Register an app in Microsoft Entra¶

In the Microsoft Entra admin center, open App registrations → New registration. Give it a name such as AgenticOS sync, keep Accounts in this organizational directory only, and leave the redirect URI empty. The source signs in as the app itself, not as a person.

From the app's Overview, note the Application (client) ID and the Directory (tenant) ID. Under Certificates & secrets, add a client secret and copy its Value. The value is shown once. The secret's ID is not what the source needs.

2. Grant it one site, not the tenant¶

The app's reach is the source's reach. Everything the source ingests becomes searchable by everyone who can read the collection it feeds. See who ends up able to read what a source ingested.

Under API permissions, add the Microsoft Graph application permission Sites.Selected and grant admin consent for it. On its own, Sites.Selected reads nothing. An administrator then grants the app read access to the one site the source reads:

POST https://graph.microsoft.com/v1.0/sites/{site-id}/permissions
Content-Type: application/json

{
  "roles": ["read"],
  "grantedToIdentities": [
    {"application": {"id": "<client-id>", "displayName": "AgenticOS sync"}}
  ]
}

That call needs Sites.FullControl.All, so an administrator makes it from Graph Explorer or with PnP PowerShell's Grant-PnPAzureADAppSitePermission, and not with the app itself. GET https://graph.microsoft.com/v1.0/sites/contoso.sharepoint.com:/sites/HR?$select=id answers the site id. A OneDrive is granted the same way, through its site: contoso-my.sharepoint.com:/personal/jane_contoso_com.

Files.Read.All or Sites.Read.All makes every library in the tenant readable

Those permissions are consented tenant-wide. A source with them still reads only the library it names. But whoever can edit the source can point its site URL at any site or OneDrive in the organization, and the next sync makes that library searchable by everyone who can read the collection. Nothing in this product can tell which permission the app was given, because a token does not say. Use Sites.Selected, and grant one app per audience.

3. Add it to the Vault¶

Add the app to the Vault as a Microsoft Entra app: the tenant id (or the tenant's domain, such as contoso.onmicrosoft.com), the client id and the client secret's value. Then choose it on the source's credential step. The secret is sent only to login.microsoftonline.com, to sign in.

A client secret expires, after two years at most. When it has expired, the next sync fails with Microsoft Entra refused the app registration's credentials (invalid_client). Add a new secret to the app and replace the value in the same vault secret. Every source that uses it picks up the new value on its next sync.

4. SharePoint connector config fields¶

Field	Type	Required	Default	Description
`site_url`	string	Yes	--	The site, e.g. `https://contoso.sharepoint.com/sites/HR`, or a OneDrive, e.g. `https://contoso-my.sharepoint.com/personal/jane_contoso_com`. Only the site: a library or page URL copied from the browser is refused.
`library`	string	No	the site's default library	The library's name as SharePoint shows it, e.g. `Documents`. A OneDrive has only its default library.
`folder_path`	string	No	--	A folder inside the library, e.g. `Policies/HR`. Leave empty for the whole library.
`include_subfolders`	boolean	No	`true`	Whether to read the folders below `folder_path`.
`extensions`	list of strings	No	`.pdf`, `.docx`, `.md`, `.txt`	Which file types to ingest. Add `.pptx`, `.xlsx` or another type only if the collection's parser reads it: a file the parser cannot read counts as a failed file on every sync.

Each file is a document whose address is sharepoint://<drive-id>/<item-id>. The address follows the file, not its path, so a file that is renamed or moved inside the library keeps its document. The document keeps the file name it was ingested with until the file's content changes.

OneNote notebooks and files of other types are not ingested.

5. What a sync transfers¶

Every sync starts with Microsoft Graph's change feed for the library. When nothing in the library has changed since the last clean run, the sync stops there: one request and no listing. The feed covers the whole library, because Graph offers it only for a library's root. A change in another folder of the same library therefore makes the next sync list the source's folder again.

When something has changed, the sync lists the folder and downloads every file of the configured types. A file whose content has not changed is then skipped before it is embedded, so only new and changed files cost an embedding. A file is downloaded from the address Graph gives for it, without the app's token. A file over the knowledge base's document cap (MAX_UPLOAD_SIZE_MB, 50 MB by default) is not downloaded and counts as a failed file.

Graph slows down a client that reads quickly. A request that Graph throttles (HTTP 429) or that fails with a 5xx or a network error is tried four times in all, and the sync waits as long as Graph's Retry-After asks, up to a minute. A folder that still cannot be listed is named in the sync log, and that sync removes nothing, because the files in that folder were not seen. The files in the other folders are still ingested.

A file deleted after the listing and before its download is removed like a file the listing did not name.

Network¶

The worker needs outbound HTTPS to login.microsoftonline.com, graph.microsoft.com and your tenant's *.sharepoint.com hosts. Nothing a source's editor types becomes an address the worker connects to: the site URL is only a name Graph looks up. Microsoft's national clouds, such as US Government and China, use other hosts and are not supported.

API reference¶

All sync source endpoints live under /api/v1/rag/sync/. Listing takes collections:view and everything that changes a source takes collections:edit. There is no admin role in it. The paragraphs below say which sources each one reaches. See who may reach a collection.

GET /api/v1/rag/sync/sources lists every source of your organization, unassigned ones included, only with connections:manage, as /org/integrations does. Without it, the list has only the sources that feed a base you can read, and collection_name must name a collection you can read, or the answer is 404. If you create or detach an unassigned source without the permission, keep its id from the response, because no list shows it to you. The list of existing integrations that the knowledge base wizard offers to reuse also needs connections:manage. See Permissions.

To update, delete or trigger a source through /api/v1/rag/sync/sources/{id}, you must also be able to write the base the source feeds. This includes a PATCH that sets collection_name to null. An unassigned source needs only your organization. /org/integrations manages sources at the organization level and needs connections:manage instead.

Sync history and a source's run log show the runs of your organization's sources, also through an app-scoped base that every organization reads. A platform admin also sees the other organizations' sources of the bases they read. A run with no source, such as a local sync or a run whose source was deleted, is listed by its collection name to everyone who reads that collection.

Sync sources CRUD¶

Method	Endpoint	Description
`GET`	`/api/v1/rag/sync/sources`	List sync sources (all of them with `connections:manage`)
`POST`	`/api/v1/rag/sync/sources`	Create a new sync source
`PATCH`	`/api/v1/rag/sync/sources/{id}`	Update an existing sync source
`DELETE`	`/api/v1/rag/sync/sources/{id}`	Delete a sync source
`POST`	`/api/v1/rag/sync/sources/{id}/trigger`	Manually trigger a sync

Connectors & logs¶

Method	Endpoint	Description
`GET`	`/api/v1/rag/sync/connectors`	List available connector types with config schemas
`GET`	`/api/v1/rag/sync/logs`	List sync history (filterable by `collection_name`)

Example: create a source via API¶

curl -X POST http://localhost:8000/api/v1/rag/sync/sources \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Legal Drive",
    "connector_type": "gdrive",
    "collection_name": "legal",
    "config": {
      "folder_id": "1abc123def",
      "include_subfolders": true
    },
    "sync_mode": "new_only",
    "schedule_minutes": 120
  }'

Example: trigger a sync via API¶

curl -X POST http://localhost:8000/api/v1/rag/sync/sources/{source_id}/trigger \
  -H "Authorization: Bearer $TOKEN"

Example: check sync history¶

curl http://localhost:8000/api/v1/rag/sync/logs?limit=10 \
  -H "Authorization: Bearer $TOKEN"

Example: discover available connectors¶

curl http://localhost:8000/api/v1/rag/sync/connectors \
  -H "Authorization: Bearer $TOKEN"

The response includes each connector's config_schema, which the frontend uses to render dynamic forms. It is also useful for building integrations programmatically.

Updating a source¶

You can update any subset of fields on an existing source with PATCH:

curl -X PATCH http://localhost:8000/api/v1/rag/sync/sources/{source_id} \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "sync_mode": "full",
    "schedule_minutes": 60,
    "is_active": false
  }'

Updatable fields: name, config, sync_mode, schedule_minutes, is_active, collection_name.

Set is_active to false to pause a source without deleting it.

Monitoring sync operations¶

Every sync creates a SyncLog entry with the following fields:

Field	Description
`source`	Connector type or `"local"` for CLI ingestion
`collection_name`	Target collection
`status`	`running`, `done`, or `error`
`mode`	`full`, `new_only`, or `update_only`
`total_files`	Number of files discovered
`ingested`	Successfully ingested (new)
`updated`	Successfully re-ingested (replaced)
`skipped`	Skipped (already present or unchanged)
`failed`	Failed to ingest, including pages or files the listing could not read, and documents that could not be removed
`removed`	Removed because the source no longer lists them (see what a sync removes)
`error_message`	What went wrong, or why nothing was removed. A run can be `done` and still have a message, for example when a crawl stopped at its page limit
`started_at`	When the sync started
`completed_at`	When the sync finished

View logs via CLI output or the API:

curl http://localhost:8000/api/v1/rag/sync/logs?collection_name=legal&limit=5 \
  -H "Authorization: Bearer $TOKEN"

Adding custom connectors¶

To add a new connector type (e.g. Notion, Confluence, Dropbox), see Add a sync connector.

The short version:

Create a class inheriting BaseSyncConnector in app/services/rag/connectors/.
Implement list_files(), _fetch(), and optionally validate_config().
Declare SECRET_KIND — what kind of vault secret authenticates it — and a CONFIG_MODEL, a Pydantic model saying how to find the documents. The credential is never one of its fields.
Register it in CONNECTOR_REGISTRY in app/services/rag/connectors/__init__.py.

Once registered, the connector appears automatically in the CLI, API, and UI.

Troubleshooting¶

"No sync sources configured"¶

You have not created any sources yet. Use rag-source-add (CLI) or POST /api/v1/rag/sync/sources (API) to create one.

"Unknown connector type"¶

The connector type you specified is not in CONNECTOR_REGISTRY. Check available types with rag-sources or GET /api/v1/rag/sync/connectors. Google Drive (gdrive) is available. S3 (s3) is available. Website (web) is available. Git (git) is available.

Google Drive: "this source has no credential"¶

The source's secret_id is empty, or the vault secret it named has been deleted. Add the service account JSON to the Vault and choose it on the source's credential step — GOOGLE_DRIVE_CREDENTIALS_FILE does not stand in for it, and only the rag-sync-gdrive CLI command reads that setting.

"A Google Drive source needs a service account credential"¶

The secret_id names a credential of the wrong kind — an AWS key pair, say. A Drive source takes a gcp_service_account and an S3 source an aws_credentials pair; the wizard offers only the matching ones, so this is reachable through the API.

Google Drive: "folder ID may contain only letters, digits, '-' and '_'"¶

The value is not a Drive folder id. Take it from the folder URL: it is the last segment, and nothing else in that URL belongs in the field.

Google Drive: "Cannot access folder"¶

Make sure you shared the folder with the service account email. The service account needs at least Viewer access.

S3: "Cannot access bucket"¶

Verify that S3_RAG_ACCESS_KEY, S3_RAG_SECRET_KEY, and S3_RAG_ENDPOINT are set correctly in .env. For MinIO, ensure the endpoint includes the port (e.g. http://localhost:9000).

Website: "resolves to private/internal address"¶

The start URL, or the sitemap, points inside the deployment's network, or its name resolves there. A website source reads only public addresses, unless the operator lists the site's host in OUTBOUND_PRIVATE_ALLOWLIST. Otherwise, publish its pages somewhere public, or upload the files directly.

Website: "The site's robots.txt could not be read, so it was not crawled"¶

/robots.txt on the start URL's host timed out or answered with a server error (5xx) three times in a row. The crawler does not guess what an unreachable robots.txt would allow, so the run stops. A missing robots.txt (404) or a forbidden one (403) means no rules, and the crawl goes ahead.

Website: "The start URL … did not lead to an HTML page"¶

The start URL answered 404, redirected to another host or outside path_prefix, or served something other than HTML. Open it in a browser, then use the address it ends at as root_url.

Website: "robots.txt does not allow the start URL"¶

The site asks crawlers to stay out of that path. Choose a start URL the site allows, or ask the site's owner to allow AgenticOS-Crawler.

Website: "… answered HTTP 403" or "… could not be reached"¶

The page needs a login, or the site refused the crawler. It failed after three attempts if the answer was a timeout, 429 or 5xx. Each page like this counts as a failed file. Nothing is removed on that run, because the pages behind it were not seen.

"The source could not be listed completely, so documents it may no longer hold were kept"¶

The listing stopped short: a crawl reached max_pages, some pages could not be read, or a SharePoint folder could not be listed. What was found was ingested, and nothing was removed. For a website, raise max_pages, or narrow the crawl with path_prefix, until a run finishes without this message. For SharePoint, see the entry for that folder below.

Git: "The repository refused the source's token"¶

The token has expired, was revoked, or cannot read this repository. Issue a new one as described in Git repository setup and replace the value of the vault secret the source uses; every source using that secret picks it up on its next sync.

Git: "The repository was not found, or the source's token cannot see it"¶

Check the clone URL first. A private repository answers not found rather than forbidden to a token that cannot read it, so a fine-grained token issued for a different repository reads as this.

Git: "The repository has no branch named …"¶

The branch field names a branch the repository does not have. Its default is main; an older repository's default branch may be master.

Git: "… is … MB, and a synced file may be at most … MB"¶

A file the include patterns match is larger than the knowledge base's document cap. Narrow include or path_prefix so that the file is left out. Nothing was written for this sync.

Git: "… over the … MB one sync may check out"¶

All the files the include patterns match are more than 512 MB together. Narrow include or path_prefix, or split the repository into more than one source, each with its own prefix.

Git: "This token was added for …, and the repository is on …"¶

The source's repository is on a different host from the one its token was added with. Either the URL is wrong, or the source needs a token added for that host. Nothing was sent to the repository's host.

Git: "A Git source needs a Git access token"¶

The source names a secret of another kind, such as an API key. Add the token as a Git access token, with its host, and choose that one.

"Another sync of this source is still running"¶

A run was triggered while another run of the same source was in progress, so it did not start. The run in progress finishes normally; trigger again after it if the source changed meanwhile.

Git: "… resolves to a private address"¶

The repository's host resolves inside the deployment's network, so the source is refused. A self-hosted Git server on an internal address cannot be reached by a sync source unless the operator lists its host in OUTBOUND_PRIVATE_ALLOWLIST.

Git: "git is not installed on this worker"¶

The worker runs from an image without git. The shipped backend/Dockerfile installs it; a custom image needs it added.

Git: a sync finished with no files processed¶

The branch's head commit is the one the last clean run read, under the same configuration, so there was nothing to do. Switch the source to full for one run to read everything again regardless.

SharePoint: "Microsoft Entra refused the app registration's credentials (…)"¶

Microsoft Entra did not sign the app in. invalid_client usually means the client secret has expired or the vault holds the secret's ID instead of its value. unauthorized_client or invalid_request usually means the tenant id or the client id is wrong. Check the three values against the app's Overview and Certificates & secrets, and replace them in the vault secret.

SharePoint: "Microsoft Graph did not accept the app's token for …"¶

The app signed in, but its token carries no Microsoft Graph permission. Add the application permission Sites.Selected, not a delegated one, and grant admin consent, as described in step 2.

SharePoint: "Microsoft Graph denied the app access to … (accessDenied)"¶

The app has Sites.Selected but no grant on this site. Grant it read on the site, as described in step 2. If it names a folder below the source's folder, that folder has permissions of its own that exclude the app. The rest of the library was ingested and nothing was removed.

SharePoint: "There is no SharePoint site at …, or the app cannot see it"¶

Check the site URL first: it is the site's own address, such as https://contoso.sharepoint.com/sites/HR, not a library or page inside it. With Sites.Selected, a site the app has no grant on can also answer this way.

SharePoint: "The site has no document library named …"¶

The message lists the libraries the app can see. Copy one of those names into library, or leave library empty for the site's default library. The default library is named in the site's language, such as Documents or Dokumente.

SharePoint: "The library has no folder …" or "… is a file, not a folder"¶

folder_path does not name a folder in the library. It is relative to the library's top level, e.g. Policies/HR, and does not repeat the library's name.

SharePoint: "The folder … could not be listed: …"¶

A folder below the source's folder could not be read. The reason follows the colon. The other folders were ingested, and nothing was removed on that run.

SharePoint: "Microsoft 365 stayed unavailable or kept throttling … after 4 attempts"¶

Graph was still throttling or failing after four tries. The next sync starts again. If this happens on every sync, schedule the source less often, or split a very large library into more than one source, each with its own folder_path.

SharePoint: "A SharePoint source needs a Microsoft Entra app credential"¶

The source names a secret of another kind, such as an API key. Add the app to the Vault as a Microsoft Entra app and choose that one.

SharePoint: a sync finished with no files processed¶

Nothing in the library changed since the last clean run under the same configuration, so there was nothing to do. Switch the source to full for one run to read everything again regardless.

Scheduled syncs are not running¶

A background task system must be running. Check that your worker process is active:

Without a worker, only manual triggers via CLI or API will work.