Secrets and the vault¶

One module, and deliberately no second mechanism

Every provider key, channel bot token, MCP credential and third-party API key in this platform passes through app/core/vault.py. Adding a second way to hold a credential at rest is the defect two migrations removed.

Envelope encryption¶

Each secret is sealed with its own random data key. That data key is sealed with a key derived from the master key and the scope that owns the secret — an organization, or the member a personal connection belongs to.

flowchart LR
    M["VAULT_MASTER_KEY<br/><i>version n</i>"] --> K
    S["the owning scope<br/><i>org id, or member id</i>"] --> K
    K["derived key"] -->|wraps| D["a random data key<br/><i>one per secret</i>"]
    D -->|seals| C["the ciphertext<br/><i>+ key_version</i>"]

Two properties follow, and both are the reason for the shape:

A ciphertext cannot be moved between owners

Even with full database access, a row copied from organization A into organization B fails to unwrap. Tenant isolation here is cryptographic, not a WHERE clause somebody might forget.

The master key is rotatable. It never encrypts a payload directly, only data keys, so rotating it re-wraps one small blob per secret instead of re-encrypting every value. Each envelope records the key_version that sealed it, which is what makes a staged rotation possible at all.

The vault decides nothing about who may read a secret — that is the permission layer. It guarantees only that a secret at rest is unreadable without the master key and unusable outside the scope it was sealed for.

How it became one mechanism¶

That sentence at the top took two rounds to become true, and the story is worth a minute because it is the shape of the mistake.

Three mechanisms used to hold secrets at rest, and only one bound a ciphertext to its owner. Provider keys went through the vault, channel bot tokens through a single deployment-wide Fernet key, MCP tokens through another. A Slack token could be copied out of one organization's row into another's and it decrypted. One migration removed those two, before the chain was squashed into 0001_baseline.

A fourth survived it, and outlived the sentence above it by some months. app/core/crypto.py held one deployment-wide Fernet key over the credential fields of sync_sources.config — the Google service account JSON and the AWS key pair a RAG sync connector authenticates with.

It was honest about itself in its own docstring, and it was still a second mechanism, so a reader who believed "there is no second mechanism" was wrong about one table.

What kept it alive was an ordering problem rather than a disagreement: an envelope is derived from its owner's id, and sync_sources.organization_id was nullable because the CLI created rows without one. #707 gave rag-source-add an organization, 0042_sync_source_secret_id made the column say so, and #937 deleted the module.

A sync source now references a vault secret by id, the way ModelProfile.secret_id and CapabilityBindingSpec.secret_id do, and its config holds only what a connector needs to find the documents.

Two consequences beyond the crypto, and they are the ones an operator notices: a credential is added once and reused by every source that needs it, rather than pasted per source and rotated in as many places; and it appears on the Vault page like everything else, so "does this organization hold a Google credential" has an answer.

Kinds¶

A secret is not always a string, and forcing every credential into one "API key" field produces a form somebody fills in correctly and still ends up with a credential that fails at the first run. So a secret has a kind, and the kind decides which fields exist.

Kind	Fields
`api_key`	One opaque token
`azure_openai`	Key, endpoint, pinned API version
`aws_credentials`	Access key id, secret access key, region, optional session token
`gcp_service_account`	The service account JSON, validated on the way in
`github_oauth_app`	A GitHub OAuth App's public client id and its secret
`git_token`	An access token for git over HTTPS, and the one host it may be sent to
`entra_app`	A Microsoft Entra app registration's tenant id, client id and client secret
`http_credential`	A token a workflow's HTTP step sends, an optional username, and the origins it may be sent to
`smtp_account`	SMTP host, port, TLS mode, sender and login credentials, sealed together for workflow email
`microsoft_oauth_app`	The same three fields, for an app registration people sign in through
`openai_codex`	A ChatGPT sign-in: an access token, a refresh token and the account id
`none`	Not a secret — the marker for an endpoint needing no credential

github_oauth_app is spent by the platform rather than picked by a person — the GitHub connect flow reads it server-side to run the token exchange — so it must be org-visible, and there must be exactly one: a member's private credential is never silently used for the whole organization's connection, and with two org-visible apps stored the connect is refused (naming both) rather than keyed to whichever name sorts first.

The same rule - org-visible only - covers the API key the organization's speech-to-text default and a channel bot's own transcription pair are sent with: the platform spends it for every member's dictation and every voice note a bot receives, so a member's private key is refused when either is saved, and a key made private afterwards stops being used. Unlike github_oauth_app, these keys are named by id, so several may be stored side by side.

The speech key picker names visible keys that cannot be selected because they are private or shared only with selected people, and explains that they must be shared with the organization first. A ChatGPT sign-in is identified separately: it provides model access through a ChatGPT subscription and cannot replace an OpenAI API key for voice features. The picker never changes a credential's access automatically.

git_token carries its host because a Git sync source sends the token to a URL whoever edits the source chooses. Bound to the host the token was added with, the token cannot be aimed at another server by editing the source, and no other key can stand in for it.

http_credential is git_token's rule for a workflow's http.request step: the step's URL is typed by whoever edits the workflow, and may be bound from a run's input, so the origins the token may go to are sealed with it, such as https://api.example.com. The step checks the origin of the URL it is about to call, and of every redirect it follows, and sends the token nowhere else. No other kind of secret can stand in for it.

A workflow step's secret field offers New secret: the vault's own form, fixed to the kind the field takes and stored as custom, the one purpose that takes any shape. The new secret is chosen on save. Its value goes to the vault and nowhere else, and the graph holds only its id.

entra_app is what a SharePoint or OneDrive sync source signs in as. Its reach is set in Microsoft Entra, not here: the Graph permissions an administrator consented to decide which sites it can read, and a token does not say which those are. Grant it Sites.Selected on the one site the source reads — see SharePoint and OneDrive setup. Its hint is the last four characters of the client id, which is public, rather than of the secret.

microsoft_oauth_app has the same fields and is spent differently: the Outlook trigger's connect flow signs a person in through it, with the delegated Mail.Read of whoever consents - their own mailbox and the shared ones they can already open - where an entra_app acts with the application permissions granted for the whole tenant. Like github_oauth_app it must be org-visible, and there must be exactly one. See Outlook.

openai_codex is the one kind nobody types. It is stored by signing in with ChatGPT, and it changes while it is used: each refresh spends the refresh token, so the new pair is sealed back under the same id. Its hint is the last four characters of the account id. When a sign-in stops working, Reconnect signs in again in its place.

aws_credentials is the clearest case for kinds existing at all: the access key id is not secret and the secret access key is, and a single field cannot express that. gcp_service_account is validated at paste time because the failure mode of a malformed one is an authentication error hours later with nothing pointing back at the paste that caused it.

none is what you store for Ollama on localhost. It is a kind rather than an empty string so the resolver can switch on a total set — and because the vault refuses to seal an empty value. Only the runtime can hold none; nobody can save one, which is what keeps "a secret with no value" out of the API schema.

Every field that authenticates — an API key, a secret access key, a client secret — must be at least eight characters. The listing shows the last four characters of a credential as its hint, so a shorter value would be published whole by its own hint; the floor also catches a truncated paste while the form is still open.

Where they are used¶

Model providers. Named by a model profile. Spend is attributed to the secret the run resolved to, which is how "which key is costing the most" gets an answer. Keying a profile lends the key to everyone who runs on it, so creating or editing one - its key or its endpoint - needs secrets:view on that key. A deleted key leaves its profiles marked missing_key until they get another, including one in front of a gateway. Only a profile saved with no key on purpose keeps running without one (A profile without a key).

Speech to text. An API key whose purpose is the provider, named by id next to the provider and model wherever speech to text is set up: the organization's default, an agent's override and a channel bot's own pair. Nothing looks a key up at run time. The organization's and a bot's key must be org-visible; an agent's follows the lending rule - the publisher must be able to view it. Each transcription's usage record names the key it was sent with. See which engine a recording reaches. A key's list of the agents using it does not show this use.

Voice calls. The key a voice call is held with is named on the agent's Voice card, beside the realtime provider and model. Its purpose must be the provider and its kind the provider's: an api_key, or an azure_openai secret for Azure, which also carries the endpoint and API version. It follows the lending rule - the publisher must be able to view it - and nothing looks a key up at run time. The key is unsealed to open a call and never reaches the browser: the server answers the WebRTC offer and no client secret is minted. To hang up a call whose process died, the voice sweep reads it through OrganizationSecretService.voice_call_key, a reader whose plaintext goes to the provider's hang-up endpoint and nowhere else. A call row keeps only the key's id, and a deleted key leaves the agent unable to take calls until another is chosen.

Capabilities. A capability declares that it needs a credential of a given kind — never an instance. Code says "I need an API key"; a binding's secret_id says which one. Where the binding names the service the key is for - the provider it chose, or the one service the capability uses, such as LlamaParse - publishing refuses a key stored for another service, since it would be sent to the wrong provider at every call. A custom key, which every key stored before purposes existed is, is accepted. See the capability catalog.

MCP connections. Bearer tokens and OAuth payloads, sealed to the organization or to the member. See MCP.

Connected accounts. A Google account a virtual terminal folder reads as, connected from the vault page. A person connects their own accounts, and an agent mounting "each person's own accounts" finds them when that person talks to it. The organization's accounts take mcp:manage to connect or disconnect and connections:view to list, and a folder names one by id. The consent uses the organization's google_oauth_app secret and asks for Drive, Gmail, Calendar, Docs, Sheets and Slides at once. Google lets a person withhold any of them, and a folder that needs a withheld one says so. The grant is an OAuth payload sealed like an MCP connection's, and a response names the account's address and its granted scopes, never a token.

Channel bots. Every credential on the row, sealed to the bot's organization at one shared key_version: the bot token, a Slack app's signing secret and app token, and the shared secret an inbound webhook is authenticated against — Telegram's X-Telegram-Bot-Api-Secret-Token, a Mattermost outgoing webhook's token. See Channels.

Event triggers. The secret an event trigger's inbound webhook is verified against - GitHub's HMAC key, or the signing secret a mail or API relay sends - sealed to the organization and stored inline on the trigger row with the key_version that sealed it, the same shape as a channel bot's signing secret. It is never returned or logged in the clear; the verification unseals it, compares in constant time, and a delivery that fails is a 403. See Concepts.

Workflow webhooks. The HMAC key a workflow webhook's deliveries are signed with, minted by the platform, sealed to the organization and stored on the webhook's row with its key_version. It is shown once, when it is made or replaced, and never again; a master-key rotation rewraps it with the rest. See Workflows.

Embeds. A jwt widget verifies visitor tokens against an HS256 signing secret the customer's backend holds. It is sealed to the agent's organization and records its key_version like every other sealed row, so a master-key rotation can rewrap it and the widget keeps verifying — where an embed that did not record its version could never be opened after a rotation.

A row with several ciphertext columns — a channel bot's four, an embed's one — seals them through vault.seal_fields, which seals every field at one version and hands that version back to store: the one way to write such a row, so "no version column" and "reset one field to v1" cannot be spelled by hand.

Third-party services. A small catalog of services an organization may bring its own key for:

Service	Used by
Tavily	`web_research`
Brave Search	`web_research`
Exa	`web_research`
Logfire	Per-agent observability — traces to a project of its own
LlamaParse	PDF parsing, billed to the organization's own key
ElevenLabs	`text_to_speech` — speech in the voices on the organization's ElevenLabs account
BytePlus ModelArk	`video_generation` — ByteDance's Seedance video models

What never happens¶

Four guarantees, pinned by tests rather than by convention

No plaintext in a response, in a log line, in an audit entry, or in an exported spec - and a capability never learns where its credential came from.

No API response returns a plaintext. There is no endpoint for it. The service that owns organization secrets has two readers that yield one, and neither hands it to a caller: the runner's, while it builds an agent, and the model catalog's, which spends a bearer token on one outbound request to a provider and returns the model names that came back. Nothing outside that service opens a secret — the model listing route used to, and that was the layering defect.
No log line or audit entry contains one. Every secret-bearing field is a Pydantic SecretStr, so the dataclasses carrying credentials mask themselves in a repr — which is the way a plaintext key usually escapes.
No spec carries one. An exported agent spec references secrets by id. That is what makes it safe to commit to a client's git repository.
A capability never learns where its credential came from, and the model never sees it at all.

Those four are pinned by tests, not by convention.

Access¶

Permission	Grants
`secrets:view`	See that a secret exists, its kind, its label
`secrets:edit`	Create, rotate, delete
`mcp:manage`	Organization MCP connections and their credentials
`connections:manage`	Org-wide credentials: model provider connections and sync-source integrations

Scopes differ by role — an Owner edits any secret in the organization, a Member edits only their own. A secret can also be shared to a specific member or agent with a resource grant, which widens access to that one row without promoting anybody. See Permissions.

Operations¶

The master key is VAULT_MASTER_KEY. It falls back to SECRET_KEY so a fresh checkout runs with no extra setup, and the config refuses an unset key anywhere except local/development — staging is a first-class deployment and routinely holds real provider keys, so it gets the same refusal production does.

Losing every configured key means every stored credential is gone

There is no recovery path and no escrow copy: every secret has to be re-entered by hand. Back the key up somewhere the database backup is not.

Rotating is a staged operation, and VAULT_MASTER_KEYS is the staged form: a JSON map of every version still in use. The highest version seals new secrets; the older ones keep existing rows readable until they are re-wrapped. key_version on each sealed row records which version wrapped it, and asking for a version with no configured key fails naming the missing entry rather than as a generic decrypt error.

# 1. Configure both keys — the old one as the version that sealed today's rows,
#    the new one above it — and unset the single VAULT_MASTER_KEY.
#    VAULT_MASTER_KEYS={"1": "<old>", "2": "<new>"}
# 2. Prove every stored envelope opens before anything moves:
uv run agenticos cmd vault-rotate --dry-run
# 3. Re-wrap every sealed row to the new version:
uv run agenticos cmd vault-rotate
# 4. Once it reports zero failures, drop version 1 from VAULT_MASTER_KEYS.

Do not drop the old key until vault-rotate reports zero failures

A row that fails is named and left as it was, and the command exits non-zero - dropping version 1 on a partial rotation makes those rows unreadable.

vault-rotate walks every table holding envelopes and moves each row's ciphertexts together with its version column, or not at all. A row holding no envelope but naming a version — a connection whose credentials were cleared — has that claim moved to the current version too, so the next secret sealed into it lands on a key that still exists. Only the wrapped data key is re-sealed — payloads are untouched, which is what makes rotation cheap.

uv run agenticos cmd doctor    # reports whether a vault key is configured at all

make platform-bootstrap BOOTSTRAP_API_KEY=sk-... stores the first provider key for you. See Configuration for the environment, and the production checklist before going live with a generated default.

Recap¶

One module, app/core/vault.py. There is no second mechanism, and adding one is the defect two migrations removed.
A secret is sealed with its own data key, wrapped by a key derived from the master key and the owning scope — so a ciphertext cannot move between owners.
A secret has a kind, because aws_credentials is four fields and one of them is not secret.
Four guarantees, pinned by tests: no plaintext in a response, a log, an audit entry or an exported spec.
Rotation is staged — configure both versions, vault-rotate --dry-run, rotate, then drop the old key once it reports zero failures.

External A2A source consent also binds the referenced vault credential’s authorization generation. Replacing its value invalidates a standing review, even when the secret ID stays the same. Renaming it and vault encryption-key maintenance preserve that generation. Existing A2A reviews require re-review after this upgrade.