Confidant

Confidant is a secure, personal data synchronization daemon written in Go. It incrementally syncs data from your external accounts on a background interval, stores and indexes it locally in a SQLite database with full-text search, and exposes an HTTP API and an MCP server with tag-based access controls — designed to let agentic AI tools safely query your personal data without blanket access.

Today the only data domain is mail (IMAP: Apple Mail, Gmail, vanilla IMAP servers). The architecture is built so that additional domains — calendar, finance, health — can be added as a bounded, repeatable task.

Core concepts

Confidant is organized around four concepts:

• Domain — a category of data the system models and serves (mail, and later calendar, finance, …). A domain defines its own resource types, storage, query surface, and agent commands. Domains are compiled in.
• Provider — integration code that talks to an external system over some protocol (imap, and later caldav, google, …). A provider declares which domains it yields: imap → [mail]. Providers are compiled in.
• Source — a configured account: an instance of a provider with credentials and an enabled subset of that provider's domains (e.g. gmail-personal, provider imap, domains [mail]). Sources are runtime data, created and edited through the API.
• Resource — one synced item (an email, and later an event or transaction). Every resource carries source_id, domain, and type, and is tagged for authorization.

A source belongs to a provider, not a domain — a single account can contribute to multiple domains (e.g. a future Google source → mail + calendar).

How It Works

Confidant runs as a local daemon. You initialize config and the database, start the server, then create your accounts (sources) through the authenticated HTTP API (or the confidant sources CLI, which calls that API). Each API key can be scoped to specific tags, so different agents only see the data you've explicitly granted them.

The database is the single source of truth for sources; there is no YAML source config. Source identity and per-provider connection config live in SQLite, and credentials live in a filesystem secret store keyed by source id (so database backups never contain plaintext passwords).

The IMAP sync engine tracks progress per-mailbox using UID ranges and UIDValidity, so it only fetches new messages on each run. Emails are parsed from raw MIME — including nested multipart structures with base64 and quoted-printable encodings — and stored with both plain text and HTML bodies. A SQLite FTS5 virtual table indexes subjects and body text for fast full-text search.

The daemon syncs all sources automatically on a configurable interval (default every 15 minutes, starting immediately at boot). Manual syncs via api ingest still work and share the same per-source job tracking, so a scheduled tick never doubles up with a running job.

Project Structure

confidant/
├── cmd/confidant/        # CLI entry point and command definitions
│   ├── main.go           # Root command tree (config, init, serve, sources, api, agent)
│   ├── config.go         # config init/show/check (server settings only)
│   ├── sources.go        # sources CRUD commands (call the daemon API)
│   ├── init.go           # Database migration and bootstrap token setup
│   ├── serve.go          # Start the HTTP daemon
│   ├── api.go            # API client commands (ingest, emails, admin)
│   └── agent.go          # Agent-facing command surface
├── internal/
│   ├── core/             # Domain-agnostic spine: Source identity, Resource, SecurityContext,
│   │                     #   Provider/Syncer/Registry, SecretStore, sync orchestration, permissions
│   ├── domains/
│   │   └── mail/         # Mail domain: Email payload, query surface, mail.Store (folder/mailbox lives here)
│   ├── providers/
│   │   └── imap/         # IMAP provider: Provider impl, imap_sources config, mail syncer
│   ├── database/         # SQLite adapter implementing the core/mail/imap stores, migrations, FTS5
│   ├── mcp/              # MCP server: tool definitions over core/mail, streamable HTTP handler
│   ├── api/              # HTTP handlers, DTOs, sources CRUD, auth middleware, MCP mounting
│   ├── config/           # Server config loading, path resolution, bootstrap key
│   └── server/           # Composition root: registers providers, wires services
├── pkg/imap/             # Standalone IMAP protocol client (reusable outside this project)
├── Makefile              # Build, test, init, run, clean, reset targets
├── go.mod
└── go.sum

Dependency direction is one-way: core imports nothing internal; domains depend on core; providers depend on core and the domains they yield; database and server sit at the bottom/top and wire everything together. The pkg/imap package is a standalone IMAP client with its own Store interface, usable in other projects.

Prerequisites

• Go 1.22 or later
• Make

No CGO required — the SQLite driver (modernc.org/sqlite) is pure Go.

Quick Start

# 1. Build the binary
make build

# 2. Initialize config, database, and the local CLI environment
make init

# 3. Start the daemon
make run

# 4. Add an email account (source) through the API
./bin/confidant sources add gmail-personal \
  --name "Gmail Personal" \
  --host imap.gmail.com \
  --username you@gmail.com \
  --password "your-app-password" \
  --mailboxes INBOX --mailboxes Sent

# 5. Trigger a first sync (the daemon also syncs automatically every 15 minutes)
./bin/confidant api ingest --source gmail-personal

# 6. Create a scoped API key for an AI agent (see "MCP Server" below)
./bin/confidant api admin keys create
./bin/confidant api admin keys set-tags <key-id> --tags "source:gmail-personal"

make init creates a ./config/ directory with a starter config.yaml (server settings only), writes a one-time bootstrap key to ./config/secrets/bootstrap_key, initializes the SQLite database at ./data/confidant.db, registers a "local" CLI environment in ./config/environments.json, and removes the temporary bootstrap key file after the environment is configured.

Because sources are runtime data, the daemon must be running before you add them (step 3 before step 4). The sources CLI commands talk to the API using the active environment, just like api and agent commands.

Configuration

config.yaml contains server settings only:

server:
  host: 127.0.0.1
  port: 8080
sync:
  interval: 15m

sync.interval is the background sync cadence as a Go duration. It must be at least 1m, or "0" to disable the background loop entirely (manual api ingest still works). It can be overridden with the CONFIDANT_SYNC_INTERVAL environment variable or the --sync-interval flag on serve.

Sources are not authored in YAML — they are created, edited, and deleted through the API. Inspect or validate the server configuration with:

./bin/confidant config show --config-dir ./config --data-dir ./data
./bin/confidant config show --resolved --config-dir ./config --data-dir ./data
./bin/confidant config check --config-dir ./config --data-dir ./data

Managing sources

Sources are managed with the confidant sources commands (which call the daemon API) or directly against the HTTP API:

# Create an imap source (writes identity spine, imap connection config, and secret)
./bin/confidant sources add gmail-personal \
  --name "Gmail Personal" \
  --host imap.gmail.com \
  --username you@gmail.com \
  --password "your-app-password" \
  --mailboxes INBOX --mailboxes Sent

# List / inspect
./bin/confidant sources list
./bin/confidant sources get gmail-personal

# Update (connection fields, name, mailboxes, password, enabled domains)
./bin/confidant sources update gmail-personal --name "Personal Gmail"
./bin/confidant sources update gmail-personal --mailboxes INBOX --mailboxes Archive
./bin/confidant sources update gmail-personal --password "new-app-password"

# Delete (cascades the connection config and all synced resources, and removes the secret)
./bin/confidant sources delete gmail-personal

--provider defaults to imap and --domains defaults to mail. --port defaults to 993 and --mailboxes defaults to INBOX. The equivalent HTTP API is POST /sources, GET /sources, GET /sources/{id}, PATCH /sources/{id}, and DELETE /sources/{id} (all admin-gated).

Secrets (Passwords)

Passwords are never stored in the database. When you create or update a source with a password, Confidant writes it to a filesystem secret store with 0600 permissions, independent of the database. At sync time the password is resolved, checked in order:

1. Environment variable: CONFIDANT_SOURCE_<ID>_PASSWORD (ID is uppercased)
2. Secret file: <config-dir>/secrets/source_<id>_password

For example, for a source with id: gmail-personal:

# Option A: environment variable
export CONFIDANT_SOURCE_GMAIL-PERSONAL_PASSWORD="your-app-password"

# Option B: secret file (or just pass --password to `sources add`/`sources update`)
echo "your-app-password" > ./config/secrets/source_gmail-personal_password

Note for Gmail: You'll need an App Password, not your regular password. For iCloud, use an app-specific password.

The filesystem secret store is a known weak spot to be revisited later; it is intentionally kept outside the database so backups don't contain plaintext credentials.

Bootstrap Key

The bootstrap key grants full administrator (superuser) access. config init creates it as a secret, and init reads it from either:

1. Environment variable: CONFIDANT_BOOTSTRAP_KEY
2. Secret file: <config-dir>/secrets/bootstrap_key

make init uses this key to initialize the database and create the local CLI environment, then removes ./config/secrets/bootstrap_key so the bootstrap key is not left behind in the config tree. If you run ./bin/confidant init manually, keep the key secure and remove the file after you have stored the token elsewhere.

Environment Overrides

The server host and port can be overridden via environment variables:

CONFIDANT_SERVER_HOST=0.0.0.0
CONFIDANT_SERVER_PORT=9090

CLI flags (--host, --port on the serve command) take highest precedence.

CLI Reference

The CLI has these root commands: config, init, serve, sources, api, agent, version, and env.

config init

Create the initial configuration files and directory structure.

./bin/confidant config init --config-dir ./config --data-dir ./data
./bin/confidant config init --force                  # overwrite existing config.yaml
./bin/confidant config init --bootstrap-key <token>  # use a specific bootstrap key

config show / config check

Display or validate the server configuration.

./bin/confidant config show --config-dir ./config --data-dir ./data
./bin/confidant config show --resolved   # includes resolved paths
./bin/confidant config check --config-dir ./config --data-dir ./data

config check validates server host and port. (Source validation now happens in the API at create/update time.)

sources

Manage configured sources through the daemon API. Like api and agent commands, these use the active environment's base URL and API key (or --base-url/--api-key).

./bin/confidant sources list
./bin/confidant sources get gmail-personal
./bin/confidant sources add gmail-personal --name "Gmail Personal" \
  --host imap.gmail.com --username you@gmail.com --password "app-password" \
  --mailboxes INBOX --mailboxes Sent
./bin/confidant sources update gmail-personal --name "Personal Gmail"
./bin/confidant sources delete gmail-personal

init

Run database migrations and register the bootstrap admin token.

./bin/confidant init --config-dir ./config --data-dir ./data

The bootstrap key must already be available from CONFIDANT_BOOTSTRAP_KEY or <config-dir>/secrets/bootstrap_key. make init handles this automatically.

serve

Start the daemon server. Handles graceful shutdown on SIGINT/SIGTERM (including cancelling in-flight syncs).

./bin/confidant serve --config-dir ./config --data-dir ./data
./bin/confidant serve --port 9090            # override port
./bin/confidant serve --sync-interval 30m    # override background sync cadence
./bin/confidant serve --sync-interval 0      # disable background sync

api ingest

Start in-memory sync jobs. Jobs return immediately and continue in the daemon. The daemon also starts these jobs itself on the configured sync.interval; manual ingest is for backfills and immediate refreshes.

./bin/confidant api ingest                              # start jobs for all sources
./bin/confidant api ingest --source gmail-personal      # one source
./bin/confidant api ingest --source gmail-personal --source icloud
./bin/confidant api ingest --source gmail-personal --since 2024-01-01 --reset-sync-state
./bin/confidant api ingest --source gmail-personal --full
./bin/confidant api ingest status
./bin/confidant api ingest status --source gmail-personal

There is one active ingest job per source. The unit of sync is (source × domain): for each source, each enabled domain's provider syncer runs. Starting a job for a source already running returns the existing job. Completed job status is kept in memory until the daemon restarts or a new job starts.

On a first sync, Confidant defaults to fetching messages since 7 days ago. Use --since (YYYY-MM-DD or RFC3339) to choose a different first-sync/backfill start. Once a source has sync state, Confidant continues from the last synced IMAP UID; to backfill, pass --reset-sync-state with --since, or use --full (reset + epoch + no count limit). --limit caps the number of matching messages fetched in one run (0/omitted means no cap; oldest matching UIDs are fetched first).

api emails

Email API commands can be filtered by source and mailbox. These filters are scoped to the emails command, so place them before list, get, or search.

./bin/confidant api emails --source gmail-personal list
./bin/confidant api emails --source gmail-personal --mailbox INBOX list
./bin/confidant api emails --source gmail-personal --mailbox Sent search "invoice"
./bin/confidant api emails --source gmail-personal --mailbox Archive get <email-resource-id>
./bin/confidant api emails list --limit 20 --offset 0

agent

Agent commands are the curated, workflow-oriented surface for autonomous clients. They use the active environment's API base URL and key, return JSON, and expose user-world concepts instead of daemon or IMAP internals.

./bin/confidant agent status
./bin/confidant agent whoami
./bin/confidant agent mailboxes list
./bin/confidant agent mail list --mailbox INBOX --limit 25
./bin/confidant agent mail search "invoice" --limit 10
./bin/confidant agent mail read <email-resource-id>

agent status verifies the daemon can authenticate the current key and reports the data domains available across configured sources. agent whoami shows the current key ID, permissions, allowed tags, and the domains visible to that key (derived from its permissions and the configured sources). agent mailboxes list summarizes visible mailboxes using the same access controls as mail list/search/read.

api admin keys

Manage API keys. Requires admin privileges.

./bin/confidant api admin keys create          # generate a new API key
./bin/confidant api admin keys delete <id>     # revoke a key

api admin keys set-tags / get-tags

Control which data an API key can access. Resources are tagged with source:<id> at sync time. An API key with tag source:gmail-personal can only see resources from that source.

# Grant a key access to gmail data only
./bin/confidant api admin keys set-tags <key-id> --tags "source:gmail-personal"

# Grant access to multiple sources
./bin/confidant api admin keys set-tags <key-id> --tags "source:gmail-personal,source:icloud"

# Check current tags
./bin/confidant api admin keys get-tags <key-id>

A key with no tags sees nothing. Any key holding the admin permission is a data superuser and bypasses all tag restrictions (the bootstrap key has admin, and it keeps it across rotation).

api admin meter

Inspect request metering and usage statistics.

env

Manage CLI API environments. make init creates a local environment automatically.

./bin/confidant env list --config-dir ./config
./bin/confidant env create local --base-url http://127.0.0.1:8080 --api-key <token>
./bin/confidant env use local

version

Print build metadata generated from git.

./bin/confidant version
./bin/confidant version --verbose

Global Options

Option	Description
--config-dir	Path to config directory (default: ~/.config/confidant)
--data-dir	Path to data directory (default: ~/.local/share/confidant)
--env	Environment name override
--base-url	API base URL override
--api-key	API key override
-v, --verbose	Enable verbose output for commands that support it

When using api, agent, and sources commands, the CLI loads the active environment from <config-dir>/environments.json, which stores the base URL and API key configured during init.

Access Control Model

Confidant uses a two-layer security model:

1. API Key Authentication: Every request must include a valid API key. Keys carry a set of permissions. Permissions follow a <domain>:<action> convention — mail:list, mail:read, mail:search — alongside cross-cutting capabilities ingest and admin. The convention extends cleanly as new domains land (e.g. calendar:list).

2. Tag-Based Row Filtering: Each resource is tagged at sync time (default tag: source:<source-id>). Each API key has an allowed tag set. When listing or searching, the database adapter dynamically joins the tags table to filter results — a key only ever sees resources whose tags overlap its own. A key holding the admin permission is a data superuser and sees everything — superuser status is permission-based, not tied to a specific key id, so it survives key rotation. This model is domain-agnostic and applies unchanged to every future domain.

So you can give an AI agent an API key scoped to source:work and it will never see your personal Gmail, even though both live in the same database.

MCP Server

The daemon exposes its agent surface over the Model Context Protocol at /api/v1/mcp (streamable HTTP, stateless). MCP is a second transport over the same services as the HTTP API: the endpoint requires a valid API key as a bearer token, every tool call re-authenticates and enforces the same <domain>:<action> permissions as the HTTP routes, and tag-based row filtering applies unchanged.

Tool	Permission	Description
confidant_whoami	any valid key	The calling agent's permissions, allowed tags, and visible domains
mail_list_mailboxes	mail:list	Visible mailboxes by source, with counts and latest message dates
mail_list	mail:list	Email summaries, filterable by source/mailbox/date
mail_search	mail:search	Full-text search with context snippets
mail_read	mail:read	One email's complete plaintext content

To connect an agent, create a scoped key and point your MCP client at the endpoint. For Claude Code:

# Create a key and scope it to one source
./bin/confidant api admin keys create
./bin/confidant api admin keys set-tags <key-id> --tags "source:gmail-personal"

# Register the server with the key as a bearer token
claude mcp add --transport http confidant http://127.0.0.1:8080/api/v1/mcp \
  --header "Authorization: Bearer <token>"

Agents should call confidant_whoami first to discover what data they can see.

Makefile Targets

Target	Description
make build	Compile the CLI binary to ./bin/confidant
make generate	Generate CLI build metadata from git
make test	Run the full test suite
make lint	Format and vet the codebase
make init	Build, create config, migrate database, and register the local environment
make run	Build, init, and start the daemon on port 8080
make clean	Remove build artifacts
make reset	Delete all local config and data (start fresh)

Testing

make test

The test suite covers:

• Server configuration save/load and validation, including sync interval parsing and precedence
• Background sync loop lifecycle: immediate and ticked syncs, disabled interval, shutdown cancelling in-flight syncs
• MCP transport: session auth, tool listing, per-tool permission enforcement, and tag-boundary filtering
• IMAP ingest options, backfill date parsing, and sync-state reset helpers
• SQLite WAL mode and migration verification
• Source identity CRUD and per-provider imap_sources config (including FK cascade on delete)
• Resource insertion with domain tagging and source:<id> access tags
• Row-level security filtering (superuser vs. restricted tag contexts)
• FTS5 full-text search with security boundary enforcement
• Key tag get/set persistence
• IMAP sync state tracking
• Multi-part MIME email parsing (plain text, base64 HTML, quoted-printable)
• CLI command-tree wiring

Version Metadata

The CLI uses git.sr.ht/~jakintosh/command-go/pkg/version for build metadata. cmd/confidant/generate_version.go marks the main package for go generate; make build and make test run generation before compiling. Generated version_generated.go files are ignored by git. Before the first commit exists, generation may print a git HEAD warning and fall back to version dev.