From 445a4cbbf94fae3a2d28abebfc0cb37a5c717f2a Mon Sep 17 00:00:00 2001 From: Roberto Date: Fri, 15 May 2026 23:32:40 +0200 Subject: [PATCH] docs: add scouts refactor + gmail scout implementation plan Covers Phases 1-3 (rename, connector skeleton, Gmail end-to-end) as 28 TDD tasks. Phase 4 (Stage 2 categorization + brief HITL) deferred to separate spec. Co-Authored-By: Claude Opus 4.7 --- ...uts-refactor-and-gmail-integration-plan.md | 3080 +++++++++++++++++ 1 file changed, 3080 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-15-scouts-refactor-and-gmail-integration-plan.md diff --git a/docs/superpowers/specs/2026-05-15-scouts-refactor-and-gmail-integration-plan.md b/docs/superpowers/specs/2026-05-15-scouts-refactor-and-gmail-integration-plan.md new file mode 100644 index 0000000..3aac8ae --- /dev/null +++ b/docs/superpowers/specs/2026-05-15-scouts-refactor-and-gmail-integration-plan.md @@ -0,0 +1,3080 @@ +# Scouts Refactor + Gmail Integration Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Rename the entire "Agents" subsystem to "Scouts" across UI/code/Postgres/SQLite/Langfuse, then add a Gmail cloud scout with two-stage pipeline (BE triage + on-connect delivery) and a reusable `SourceConnector` abstraction. Phase 4 (Stage 2 categorization + HITL surface) is deferred to a future spec. + +**Architecture:** Phase 1 is a single atomic rename PR (no behavior change). Phase 2 lays connector skeleton, queue table, WS frame contract, and SQLite suggestion table — no user-visible feature yet. Phase 3 ships Gmail end-to-end: OAuth setup UI, push (`users.watch`) + cron-fallback polling, BE-side spam triage with opt-in auto-trash, deliver-on-reconnect drain into local `scoutSuggestions` table with `category='unprocessed'` stub. + +**Tech Stack:** Python 3.12 / FastAPI / SQLAlchemy 2.0 (async) / Alembic / Pydantic v2 / Langfuse for prompts / Google API Client for Gmail / Pub/Sub for push notifications. Electron / TypeScript / React 19 / tRPC v11 / Drizzle ORM / better-sqlite3 / i18next. + +**Spec:** [docs/superpowers/specs/2026-05-15-scouts-refactor-and-gmail-integration-design.md](2026-05-15-scouts-refactor-and-gmail-integration-design.md) + +**Repository layout:** monorepo with submodules `api/` (FastAPI) and `adiuvAI/` (Electron). All tasks operate on absolute paths under `c:\Users\PC-Roby\Documents\_adiuvai_workspace`. + +--- + +## File Structure + +### New files (BE — `api/`) + +| Path | Responsibility | +|------|----------------| +| `api/alembic/versions/007_rename_agents_to_scouts.py` | Rename Postgres tables/columns from `*_agent_*` to `*_scout_*`. | +| `api/alembic/versions/008_scout_triage_queue.py` | Create `scout_triage_queue` + alter `cloud_scout_configs`. | +| `api/app/scouts/__init__.py` | Package marker. | +| `api/app/scouts/engine.py` | `ScoutEngine` orchestrator (triage, queue, deliver). | +| `api/app/scouts/connectors/__init__.py` | Package marker. | +| `api/app/scouts/connectors/base.py` | `SourceConnector` Protocol + `ItemRef`/`ItemMetadata`/`ItemContent`/`TriageVerdict` Pydantic models. | +| `api/app/scouts/connectors/registry.py` | Connector registration + lookup. | +| `api/app/scouts/connectors/gmail.py` | `GmailConnector` impl (wraps existing `app/integrations/gmail.py`). | +| `api/app/api/routes/scout_webhooks.py` | `POST /api/v1/scouts/webhooks/gmail` — Pub/Sub receiver. | +| `api/tests/test_scout_engine.py` | Unit tests for `ScoutEngine` w/ mocked connector. | +| `api/tests/test_scout_connectors_gmail.py` | Tests for `GmailConnector` (mocked Gmail API). | +| `api/tests/test_scout_webhook.py` | Webhook + Pub/Sub JWT verification. | + +### Renamed files (BE) + +| Before | After | +|--------|-------| +| `api/app/api/routes/agents.py` | `api/app/api/routes/scouts.py` | +| `api/app/api/routes/agent_setup.py` | `api/app/api/routes/scout_setup.py` | +| `api/app/core/agent_runner.py` | `api/app/core/scout_runner.py` | +| `api/app/core/agent_session_buffer.py` | `api/app/core/scout_session_buffer.py` | +| `api/app/core/agent_registry.py` | `api/app/core/scout_registry.py` | + +### New files (Electron — `adiuvAI/`) + +| Path | Responsibility | +|------|----------------| +| `adiuvAI/src/main/db/migrations/0007_scouts_rename.sql` | Drizzle SQL: rename `agent_runs` → `scout_runs`, `agent_run_actions` → `scout_run_actions`. | +| `adiuvAI/src/main/db/migrations/0008_scout_suggestions.sql` | Drizzle SQL: create `scout_suggestions` table. | +| `adiuvAI/src/main/scouts/scout-suggestion-handler.ts` | Handles incoming `scout_proposal` WS frame, inserts into `scoutSuggestions`, sends `scout_proposal_ack`. | + +### Renamed files (Electron) + +| Before | After | +|--------|-------| +| `adiuvAI/src/main/agents/agent-scheduler.ts` | `adiuvAI/src/main/scouts/scout-scheduler.ts` | +| `adiuvAI/src/renderer/components/settings/AgentsSection.tsx` | `adiuvAI/src/renderer/components/settings/ScoutsSection.tsx` | +| `adiuvAI/src/renderer/components/settings/AgentRow.tsx` | `adiuvAI/src/renderer/components/settings/ScoutRow.tsx` | +| `adiuvAI/src/renderer/components/settings/LocalAgentConfigPanel.tsx` | `adiuvAI/src/renderer/components/settings/LocalScoutConfigPanel.tsx` | +| `adiuvAI/src/renderer/components/settings/CloudAgentConfigPanel.tsx` | `adiuvAI/src/renderer/components/settings/CloudScoutConfigPanel.tsx` | +| `adiuvAI/src/renderer/components/settings/InlineAgentCreationStepper.tsx` | `adiuvAI/src/renderer/components/settings/InlineScoutCreationStepper.tsx` | + +### Modified files (high-impact, full list per task) + +- `api/app/models.py` — class renames + table/column renames + relationship updates. +- `api/app/main.py` — register new router, lifespan crons. +- `api/app/schemas.py` — `WsFrameType` enum + new frame Pydantic models. +- `api/app/config/settings.py` — add Gmail Pub/Sub topic env vars. +- `adiuvAI/src/main/db/schema.ts` — rename tables + add `scoutSuggestions`. +- `adiuvAI/src/main/router/index.ts` — rename `agent.*` sub-routers → `scout.*`. +- `adiuvAI/src/main/api/backend-client.ts` — handle new WS frames. +- `adiuvAI/src/renderer/locales/{en,it,es,fr,de}/translation.json` — rename i18n keys. +- `adiuvAI/src/renderer/routes/settings.tsx` — update SECTIONS array. + +--- + +# PHASE 1 — Rename Only + +Goal: pure rename, no behavior change, all tests green at end of phase. + +**Important caveat:** the project uses git submodules. Each task touches files in `api/` or `adiuvAI/` submodule — commits happen inside the submodule, then a separate commit in the monorepo updates the submodule pointer. Each task's commit step shows both. + +--- + +## Task 1: Postgres rename — Alembic migration + +**Files:** +- Create: `api/alembic/versions/007_rename_agents_to_scouts.py` + +**Context:** Latest existing migration is `d6e3f4a5b6c7_folder_index_tables.py` with `down_revision = "006"`. New migration uses revision `"007"` and depends on `d6e3f4a5b6c7`. Tables to rename: `local_agent_configs`, `cloud_agent_configs`, `agent_run_logs`. Columns to rename: `agent_config` (in `local_agent_configs`), `agent_id` and `agent_type` (in `agent_run_logs`). + +- [ ] **Step 1: Create the migration file** + +```python +"""Rename agents to scouts. + +Revision ID: 007 +Revises: d6e3f4a5b6c7 +Create Date: 2026-05-15 + +Renames the entire agents subsystem identifiers to scouts. +Pre-1.0 — no data preservation concerns beyond ALTER TABLE rename. +""" + +from typing import Sequence, Union + +from alembic import op + + +revision: str = "007" +down_revision: Union[str, None] = "d6e3f4a5b6c7" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # Tables + op.rename_table("local_agent_configs", "local_scout_configs") + op.rename_table("cloud_agent_configs", "cloud_scout_configs") + op.rename_table("agent_run_logs", "scout_run_logs") + + # Columns + op.alter_column("local_scout_configs", "agent_config", new_column_name="scout_config") + op.alter_column("scout_run_logs", "agent_id", new_column_name="scout_id") + op.alter_column("scout_run_logs", "agent_type", new_column_name="scout_type") + + +def downgrade() -> None: + op.alter_column("scout_run_logs", "scout_type", new_column_name="agent_type") + op.alter_column("scout_run_logs", "scout_id", new_column_name="agent_id") + op.alter_column("local_scout_configs", "scout_config", new_column_name="agent_config") + + op.rename_table("scout_run_logs", "agent_run_logs") + op.rename_table("cloud_scout_configs", "cloud_agent_configs") + op.rename_table("local_scout_configs", "local_agent_configs") +``` + +- [ ] **Step 2: Run migration against a fresh test DB to verify it applies cleanly** + +Run from `api/` directory: +```bash +alembic upgrade head +``` +Expected: no errors. `\d local_scout_configs` shows the renamed table. `\d scout_run_logs` shows the `scout_id` column (not `agent_id`). + +- [ ] **Step 3: Verify downgrade works** + +```bash +alembic downgrade -1 +``` +Expected: tables go back to `local_agent_configs` etc. Then re-run `alembic upgrade head`. + +- [ ] **Step 4: Commit (inside submodule)** + +```bash +cd api +git add alembic/versions/007_rename_agents_to_scouts.py +git commit -m "feat(db): rename agents to scouts (alembic 007)" +``` + +--- + +## Task 2: SQLAlchemy model rename — `LocalScoutConfig`, `CloudScoutConfig`, `ScoutRunLog` + +**Files:** +- Modify: `api/app/models.py:161-227` (and the `AgentRunLog` class — find via grep) + +**Context:** Renames Python class names to match the new tables. Must update `__tablename__` and `mapped_column` for renamed columns. Relationships and `back_populates` strings must reference the new class names. + +- [ ] **Step 1: Read current models around `LocalAgentConfig`, `CloudAgentConfig`, `AgentRunLog`** + +Use Read tool on `api/app/models.py` to confirm the exact line ranges (the `AgentRunLog` class + relationships use string-based `primaryjoin` + `back_populates="local_agent"` / `back_populates="cloud_agent"` patterns). + +- [ ] **Step 2: Apply the renames in `models.py`** + +Replace class names and tablenames: + +```python +class LocalScoutConfig(Base): + __tablename__ = "local_scout_configs" + + # ... id/user_id/etc unchanged ... + scout_config: Mapped[dict | None] = mapped_column(JSON, nullable=True) + # ... rest unchanged ... + + run_logs: Mapped[list["ScoutRunLog"]] = relationship( + back_populates="local_scout", + primaryjoin="and_(ScoutRunLog.scout_id == LocalScoutConfig.id, ScoutRunLog.scout_type == 'local')", + foreign_keys="ScoutRunLog.scout_id", + cascade="all, delete-orphan", + overlaps="run_logs,cloud_scout", + ) + + +class CloudScoutConfig(Base): + __tablename__ = "cloud_scout_configs" + + # ... unchanged columns ... + + run_logs: Mapped[list["ScoutRunLog"]] = relationship( + back_populates="cloud_scout", + primaryjoin="and_(ScoutRunLog.scout_id == CloudScoutConfig.id, ScoutRunLog.scout_type == 'cloud')", + foreign_keys="ScoutRunLog.scout_id", + cascade="all, delete-orphan", + overlaps="run_logs,local_scout", + ) + + +class ScoutRunLog(Base): + __tablename__ = "scout_run_logs" + + # ... id, user_id, started_at, finished_at, status, error etc. unchanged ... + scout_id: Mapped[str] = mapped_column(...) + scout_type: Mapped[str] = mapped_column(...) + + local_scout: Mapped["LocalScoutConfig | None"] = relationship( + back_populates="run_logs", + primaryjoin="and_(ScoutRunLog.scout_id == LocalScoutConfig.id, ScoutRunLog.scout_type == 'local')", + foreign_keys="ScoutRunLog.scout_id", + overlaps="run_logs,cloud_scout", + ) + + cloud_scout: Mapped["CloudScoutConfig | None"] = relationship( + back_populates="run_logs", + primaryjoin="and_(ScoutRunLog.scout_id == CloudScoutConfig.id, ScoutRunLog.scout_type == 'cloud')", + foreign_keys="ScoutRunLog.scout_id", + overlaps="run_logs,local_scout", + ) +``` + +- [ ] **Step 3: Sweep all imports of the old class names** + +Run from `api/`: +```bash +grep -rn "AgentRunLog\|LocalAgentConfig\|CloudAgentConfig" app/ tests/ --include="*.py" +``` + +For each hit, replace with the corresponding `Scout*` name. Do not skip test files. + +- [ ] **Step 4: Run the test suite — expect failures and fix incrementally** + +Run from `api/`: +```bash +pytest -x +``` +Fix import errors and reference errors as they surface. The test fixture at `api/tests/conftest.py` does `Base.metadata.create_all` so renamed tables will be created automatically. Don't worry about migrations in tests — they use `create_all`. + +- [ ] **Step 5: Verify the suite is green** + +```bash +pytest +``` +Expected: all tests pass. + +- [ ] **Step 6: Commit (inside submodule)** + +```bash +cd api +git add app/models.py +git add -u app/ tests/ # any files changed by the import sweep +git commit -m "refactor(models): rename Agent classes to Scout" +``` + +--- + +## Task 3: Rename BE routes — `routes/agents.py` → `routes/scouts.py`, `agent_setup.py` → `scout_setup.py` + +**Files:** +- Rename: `api/app/api/routes/agents.py` → `api/app/api/routes/scouts.py` +- Rename: `api/app/api/routes/agent_setup.py` → `api/app/api/routes/scout_setup.py` +- Modify: `api/app/main.py` — update router registration +- Modify: `api/app/api/routes/device_ws.py:30-31` — update imports + +- [ ] **Step 1: Rename the files via git** + +```bash +cd api +git mv app/api/routes/agents.py app/api/routes/scouts.py +git mv app/api/routes/agent_setup.py app/api/routes/scout_setup.py +``` + +- [ ] **Step 2: Inside both renamed files, change `APIRouter(prefix="/agents", ...)` to `prefix="/scouts"` and `prefix="/agent-setup"` to `prefix="/scout-setup"`** + +In `app/api/routes/scouts.py` find and replace: +```python +router = APIRouter(prefix="/agents", tags=["agents"]) +``` +With: +```python +router = APIRouter(prefix="/scouts", tags=["scouts"]) +``` + +In `app/api/routes/scout_setup.py` find and replace: +```python +router = APIRouter(prefix="/agent-setup", tags=["agent-setup"]) +``` +With: +```python +router = APIRouter(prefix="/scout-setup", tags=["scout-setup"]) +``` + +- [ ] **Step 3: Update `app/main.py` to import the new module names** + +Find the imports of `agents` and `agent_setup` routers and replace with `scouts` and `scout_setup`. + +- [ ] **Step 4: Update `app/api/routes/device_ws.py:30-31` import** + +Change: +```python +from app.api.routes.agent_setup import handle_journey_message, handle_journey_start +``` +To: +```python +from app.api.routes.scout_setup import handle_journey_message, handle_journey_start +``` + +- [ ] **Step 5: Sweep remaining import references** + +```bash +grep -rn "from app.api.routes.agents\|from app.api.routes.agent_setup\|app\.api\.routes\.agents\|app\.api\.routes\.agent_setup" app/ tests/ +``` +Replace each. + +- [ ] **Step 6: Run tests** + +```bash +pytest +``` +Expected: pass. URL-based tests must use new paths (`/api/v1/scouts/...`, `/api/v1/scout-setup/...`). + +- [ ] **Step 7: Commit (inside submodule)** + +```bash +cd api +git add -A app/ tests/ +git commit -m "refactor(routes): rename /agents and /agent-setup to /scouts and /scout-setup" +``` + +--- + +## Task 4: Rename `core/agent_runner.py`, `agent_session_buffer.py`, `agent_registry.py` + +**Files:** +- Rename: `api/app/core/agent_runner.py` → `api/app/core/scout_runner.py` +- Rename: `api/app/core/agent_session_buffer.py` → `api/app/core/scout_session_buffer.py` +- Rename: `api/app/core/agent_registry.py` → `api/app/core/scout_registry.py` + +- [ ] **Step 1: Rename via git** + +```bash +cd api +git mv app/core/agent_runner.py app/core/scout_runner.py +git mv app/core/agent_session_buffer.py app/core/scout_session_buffer.py +git mv app/core/agent_registry.py app/core/scout_registry.py +``` + +- [ ] **Step 2: Update top-of-file references in each renamed file** + +Inside each file, rename all references to `agent_*` symbols that are now defined here. For example, if `agent_runner.py` had `def trigger_agent_run(...)`, change to `def trigger_scout_run(...)`. Do NOT rename names that belong to `app/agents/*` (those LLM helpers stay). + +- [ ] **Step 3: Sweep external imports** + +```bash +grep -rn "from app.core.agent_runner\|from app.core.agent_session_buffer\|from app.core.agent_registry" app/ tests/ +``` +Replace each `agent_runner` → `scout_runner`, etc. + +Also sweep symbol references — for example if you renamed `trigger_agent_run` → `trigger_scout_run`: +```bash +grep -rn "trigger_agent_run\|agent_session_buffer\|agent_registry" app/ tests/ +``` +Replace. + +- [ ] **Step 4: Run tests** + +```bash +pytest +``` +Expected: green. + +- [ ] **Step 5: Commit (inside submodule)** + +```bash +cd api +git add -A app/ tests/ +git commit -m "refactor(core): rename agent_runner/session_buffer/registry to scout_*" +``` + +--- + +## Task 5: Rename Pydantic schemas + WS frame names referring to "agent" + +**Files:** +- Modify: `api/app/schemas.py` (or wherever `WsFrameType` enum + Agent-related schemas live) + +**Context:** Pydantic schemas like `AgentRunStatus`, `AgentTriggerRequest`, etc. need rename. The `WsFrameType` enum may have values like `"agent_run_started"` — rename to `"scout_run_started"`. Frontend will be updated to match in later tasks. + +- [ ] **Step 1: Find all Pydantic models + enum values containing "agent"** + +```bash +cd api +grep -n "class .*Agent\|\"agent_" app/schemas.py +``` + +- [ ] **Step 2: Rename each — class names and enum values** + +Examples (apply each found in step 1): +- `class AgentRunStatus(str, Enum)` → `class ScoutRunStatus(str, Enum)` +- `class AgentTriggerRequest(BaseModel)` → `class ScoutTriggerRequest(BaseModel)` +- `WsFrameType.AGENT_RUN_STARTED = "agent_run_started"` → `SCOUT_RUN_STARTED = "scout_run_started"` + +- [ ] **Step 3: Sweep all references** + +```bash +grep -rn "AgentRunStatus\|AgentTriggerRequest\|agent_run_started\|agent_run_completed" app/ tests/ +``` +Replace. + +- [ ] **Step 4: Run tests** + +```bash +pytest +``` + +- [ ] **Step 5: Commit** + +```bash +cd api +git add -A +git commit -m "refactor(schemas): rename Agent* schemas and WS frame types to Scout*" +``` + +--- + +## Task 6: Rename Langfuse prompts (user-facing scout prompts only) + +**Files:** +- Manual operation against Langfuse via MCP tools (no source files unless prompt names are referenced in code). + +**Context:** Langfuse prompts whose names start with `agent-` and refer to the user-facing scout subsystem (e.g. `agent-runner-system`, `agent-setup-journey-system`, `cloud-agent-trigger`, `local-agent-extraction`). Prompts for `app/agents/*` LLM helpers (e.g. `task-agent-system`, `note-agent-summarize`) STAY. + +- [ ] **Step 1: List all Langfuse prompts** + +Use the Langfuse MCP tool: +``` +mcp__langfuse__listPrompts (no arguments) +``` + +- [ ] **Step 2: Identify scout-subsystem prompts to rename** + +From the listing, pick prompts whose names begin with `agent-` AND whose content/usage maps to the user-facing scout subsystem. If unsure for a given prompt, grep its name in `api/app/`: +```bash +grep -rn "get_prompt_or_fallback(\"agent-" api/app/ api/tests/ +``` +Names that appear in the renamed `scout_*.py` files are scout-subsystem prompts. + +- [ ] **Step 3: For each scout prompt, create a new prompt with the renamed name** + +Use `mcp__langfuse__getPromptUnresolved` to fetch the current prompt body, then `mcp__langfuse__createTextPrompt` (or `createChatPrompt` if it's a chat prompt) with the new name (e.g. `scout-runner-system`) and label `production`. + +- [ ] **Step 4: Update code references to use the new prompt names** + +Find each `get_prompt_or_fallback("agent-..."` call and rename the literal: +```bash +grep -rn 'get_prompt_or_fallback("agent-' api/app/ +``` +Replace each `"agent-X"` with `"scout-X"`. + +- [ ] **Step 5: Run tests + manual smoke** + +```bash +cd api +pytest +``` +Then start dev server (`uvicorn app.main:app --reload`), trigger a known scout-subsystem agent invocation (use the agent setup journey or a scout trigger endpoint via curl) and confirm the prompt loads without falling back. + +- [ ] **Step 6: Delete old prompts from Langfuse** + +For each old `agent-*` prompt confirmed superseded, delete via Langfuse UI or API (Langfuse MCP doesn't expose delete; use the Langfuse web console). + +- [ ] **Step 7: Commit code changes** + +```bash +cd api +git add -A +git commit -m "refactor(prompts): rename scout-subsystem Langfuse prompt references to scout-*" +``` + +--- + +## Task 7: Drizzle migration — rename `agent_runs` → `scout_runs`, `agent_run_actions` → `scout_run_actions` + +**Files:** +- Create: `adiuvAI/src/main/db/migrations/0007_scouts_rename.sql` +- Modify: `adiuvAI/src/main/db/schema.ts` + +**Context:** Drizzle migrations are SQL files in `adiuvAI/src/main/db/migrations/`. Latest is `0006_misty_cammi.sql`. The auto-generated migration name suffix isn't strictly required but follow the pattern (drizzle-kit generates one from CHANGELOG entries). For a manual rename, a hand-written name is fine. + +- [ ] **Step 1: Update `adiuvAI/src/main/db/schema.ts` first — change the table name in the Drizzle definition** + +Find: +```typescript +export const agentRuns = sqliteTable('agent_runs', { ... }); +export const agentRunActions = sqliteTable('agent_run_actions', { ... }); +``` + +Replace with: +```typescript +export const scoutRuns = sqliteTable('scout_runs', { + id: text().primaryKey(), + scoutId: text('scout_id').notNull(), + status: text().notNull(), + startedAt: integer('started_at').notNull(), + completedAt: integer('completed_at'), +}); + +export const scoutRunActions = sqliteTable('scout_run_actions', { + id: text().primaryKey(), + runId: text('run_id').notNull(), + scoutId: text('scout_id').notNull(), + verb: text().notNull(), + entityType: text('entity_type').notNull(), + entityId: text('entity_id').notNull(), + entityTitle: text('entity_title'), + createdAt: integer('created_at').notNull(), +}); +``` + +(Read the file first to copy the exact column definitions — keep types/nullability identical, only rename `agent` → `scout`.) + +- [ ] **Step 2: Generate the migration** + +Run from `adiuvAI/`: +```bash +npx drizzle-kit generate +``` +Expected: a new file `0007__.sql` in `src/main/db/migrations/`. Verify it contains `ALTER TABLE` statements (or DROP+CREATE if Drizzle prefers — SQLite may require recreating the table; Drizzle will emit the right pattern). + +If Drizzle emits a destructive recreate (DROP TABLE + CREATE), that's fine pre-1.0 since these tables are largely empty in dev DBs. Confirm with the user before proceeding if their dev DB has rows worth preserving — for empty/dev DBs, drop is fine. + +- [ ] **Step 3: Rename the generated migration to a clean name** + +```bash +mv adiuvAI/src/main/db/migrations/0007_*.sql adiuvAI/src/main/db/migrations/0007_scouts_rename.sql +``` + +- [ ] **Step 4: Sweep TypeScript imports of the old names** + +```bash +grep -rn "agentRuns\|agentRunActions" adiuvAI/src/ +``` +Replace each with `scoutRuns` / `scoutRunActions`. + +- [ ] **Step 5: Run typecheck + lint** + +From `adiuvAI/`: +```bash +npm run lint +npx tsc --noEmit +``` +Expected: pass. + +- [ ] **Step 6: Manual smoke — boot the app once to apply the migration** + +```bash +cd adiuvAI +npm run start +``` +Watch the console for migration apply messages. Once app boots cleanly, close it. Verify the dev DB has `scout_runs` and `scout_run_actions` (use `sqlite3 dev.db ".tables"` from `adiuvAI/`). + +- [ ] **Step 7: Commit (inside submodule)** + +```bash +cd adiuvAI +git add src/main/db/schema.ts src/main/db/migrations/0007_scouts_rename.sql +git add -u src/ # any consumer files updated in step 4 +git commit -m "refactor(db): rename agent_runs/agent_run_actions to scout_*" +``` + +--- + +## Task 8: Rename Electron main-process modules and tRPC sub-routers + +**Files:** +- Rename: `adiuvAI/src/main/agents/agent-scheduler.ts` → `adiuvAI/src/main/scouts/scout-scheduler.ts` +- Modify: `adiuvAI/src/main/router/index.ts` — rename `agent.*` sub-router to `scout.*` +- Modify: `adiuvAI/src/main/store.ts` — rename `getLocalAgents`/`saveLocalAgent` → `getLocalScouts`/`saveLocalScout` +- Modify: `adiuvAI/src/main/index.ts` — update scheduler import + start call +- Modify: `adiuvAI/src/main/api/backend-client.ts` — rename agent-related symbols if any + +**Context:** Inside `agent-scheduler.ts`, the function names are `startAgentScheduler`, `stopAgentScheduler`, `tickAgentScheduler` and POSTs to `/api/v1/agents/trigger`. After Task 3, the BE route is `/api/v1/scouts/trigger` — so this file must also update its URL. + +- [ ] **Step 1: Move the file** + +```bash +cd adiuvAI +mkdir -p src/main/scouts +git mv src/main/agents/agent-scheduler.ts src/main/scouts/scout-scheduler.ts +rmdir src/main/agents 2>/dev/null || true +``` +(Don't fail if other files exist in `src/main/agents/` — check first; if so, decide per-file whether to move or leave.) + +- [ ] **Step 2: Inside `scout-scheduler.ts`, rename functions and update URL** + +Replace: +- `startAgentScheduler` → `startScoutScheduler` +- `stopAgentScheduler` → `stopScoutScheduler` +- `tickAgentScheduler` → `tickScoutScheduler` +- `getLocalAgents()` → `getLocalScouts()` +- `saveLocalAgent(...)` → `saveLocalScout(...)` +- URL `'/api/v1/agents/trigger'` → `'/api/v1/scouts/trigger'` +- Variable `agents` → `scouts` (clarity) +- `agentRuns` Drizzle import → `scoutRuns` + +Show the full updated function header: +```typescript +async function tickScoutScheduler(): Promise { + const scouts = getLocalScouts(); + const now = Date.now(); + + for (const scout of scouts) { + if (!scout.enabled) continue; + const intervalMs = CRON_INTERVAL_MS[scout.scheduleCron]; + if (!intervalMs) continue; + if (scout.lastRunAt && now - scout.lastRunAt < intervalMs) continue; + + try { + const response = await getBackendClient().proxyPost<{ id: string }>( + '/api/v1/scouts/trigger', + { directory: scout.directory, deviceId: getDeviceId(), /* ... */ } + ); + if (response?.id) { + await getDb().insert(scoutRuns).values({ + id: response.id, scoutId: scout.id, status: 'running', startedAt: now, + }).onConflictDoNothing(); + } + saveLocalScout({ ...scout, lastRunAt: now }); + } catch (err) { /* ... */ } + } +} +``` + +- [ ] **Step 3: Rename store helpers in `adiuvAI/src/main/store.ts`** + +Find `getLocalAgents` / `saveLocalAgent` / any `LocalAgentConfig` references and rename to `getLocalScouts` / `saveLocalScout` / `LocalScoutConfig`. Keep the storage key migration-safe: read from old key `localAgents` once, write to `localScouts`, then read only from new key thereafter. For pre-1.0 dev, simpler to just rename the key and accept users re-adding their scouts (note: confirm with user). + +For now, simplest acceptable approach: +```typescript +// store.ts +const STORAGE_KEY = 'localScouts'; // was 'localAgents' + +export function getLocalScouts(): LocalScoutConfig[] { + return store.get(STORAGE_KEY, []) as LocalScoutConfig[]; +} +export function saveLocalScout(scout: LocalScoutConfig): void { /* ... */ } +``` + +- [ ] **Step 4: Update `adiuvAI/src/main/index.ts`** + +Replace: +```typescript +import { startAgentScheduler, stopAgentScheduler } from './agents/agent-scheduler'; +``` +With: +```typescript +import { startScoutScheduler, stopScoutScheduler } from './scouts/scout-scheduler'; +``` + +And the call sites: `startAgentScheduler()` → `startScoutScheduler()`, `stopAgentScheduler()` → `stopScoutScheduler()`. + +- [ ] **Step 5: Update `adiuvAI/src/main/router/index.ts` — rename `agent.*` sub-router to `scout.*`** + +Find the `agent: t.router({ local: ..., cloud: ..., journey: ..., runs: ..., runActions: ..., catalog: ..., runNow: ..., ... })` block. Rename the key from `agent` to `scout`. Also rename the imported sub-router files if any (in `adiuvAI/src/main/router/`, look for `agent.ts` or `agent/` subdirectory). + +- [ ] **Step 6: Sweep type references** + +```bash +grep -rn "LocalAgentConfig\|CloudAgentConfig\|trpc\.agent\.\|/api/v1/agents" adiuvAI/src/ +``` +Replace systematically: `LocalAgentConfig` → `LocalScoutConfig`, `CloudAgentConfig` → `CloudScoutConfig`, `trpc.agent.` → `trpc.scout.`, `/api/v1/agents` → `/api/v1/scouts`. + +- [ ] **Step 7: Typecheck + lint** + +```bash +cd adiuvAI +npx tsc --noEmit +npm run lint +``` + +- [ ] **Step 8: Commit** + +```bash +cd adiuvAI +git add -A src/main/ +git commit -m "refactor(main): rename agent-scheduler/store/router symbols to scout" +``` + +--- + +## Task 9: Rename Renderer components + +**Files:** +- Rename 5 files in `adiuvAI/src/renderer/components/settings/` (per File Structure table) +- Modify: `adiuvAI/src/renderer/routes/settings.tsx` — update `SECTIONS` array +- Modify: each renamed component — update internal symbols + +- [ ] **Step 1: Move the files** + +```bash +cd adiuvAI +git mv src/renderer/components/settings/AgentsSection.tsx src/renderer/components/settings/ScoutsSection.tsx +git mv src/renderer/components/settings/AgentRow.tsx src/renderer/components/settings/ScoutRow.tsx +git mv src/renderer/components/settings/LocalAgentConfigPanel.tsx src/renderer/components/settings/LocalScoutConfigPanel.tsx +git mv src/renderer/components/settings/CloudAgentConfigPanel.tsx src/renderer/components/settings/CloudScoutConfigPanel.tsx +git mv src/renderer/components/settings/InlineAgentCreationStepper.tsx src/renderer/components/settings/InlineScoutCreationStepper.tsx +``` + +- [ ] **Step 2: Inside each renamed file, rename the exported component** + +Rule: every `export function AgentX` → `export function ScoutX`, every `Agent` → `Scout` in component bodies, every `agent` → `scout` in local variables, every `'toast.agent.X'` i18n key → `'toast.scout.X'`, every `'agents.X'` → `'scouts.X'`, every `t('settings.agentsX')` → `t('settings.scoutsX')`. + +Show example (head of `ScoutsSection.tsx`): +```typescript +export function ScoutsSection() { + const { t } = useTranslation(); + const utils = trpc.useUtils(); + const localScoutsQuery = trpc.scout.local.list.useQuery(); + const cloudScoutsQuery = trpc.scout.cloud.list.useQuery(); + const deleteLocalMutation = trpc.scout.local.delete.useMutation(); + const deleteCloudMutation = trpc.scout.cloud.delete.useMutation(); + const updateLocalMutation = trpc.scout.local.update.useMutation(); + const updateCloudMutation = trpc.scout.cloud.update.useMutation(); + const runNowMutation = trpc.scout.runNow.useMutation(); + + const { notify, notifyError, notifyPromise } = useNotify(); + const [expandedScout, setExpandedScout] = useState(null); + // ... + + const localScouts: LocalScoutConfig[] = localScoutsQuery.data ?? []; + const cloudScouts: CloudScoutConfig[] = cloudScoutsQuery.data ?? []; + const allScouts = [ + ...localScouts.map(s => ({ ...s, scoutType: 'local' as const })), + ...cloudScouts.map(s => ({ ...s, scoutType: 'cloud' as const })), + ]; + // ... +} +``` + +- [ ] **Step 3: Update consumers — settings route + any other importer** + +In `adiuvAI/src/renderer/routes/settings.tsx`, change: +```typescript +import { AgentsSection } from '@/components/settings/AgentsSection'; +// ... in SECTIONS array: +{ id: 'agents', label: 'settings.agents', component: AgentsSection }, +``` +To: +```typescript +import { ScoutsSection } from '@/components/settings/ScoutsSection'; +// ... +{ id: 'scouts', label: 'settings.scouts', component: ScoutsSection }, +``` + +Also sweep: +```bash +cd adiuvAI +grep -rn "AgentsSection\|AgentRow\|LocalAgentConfigPanel\|CloudAgentConfigPanel\|InlineAgentCreationStepper" src/ +``` +Replace each. + +- [ ] **Step 4: Update `adiuvAI/src/renderer/components/settings/types.ts` (if it defines `LocalAgentConfig`)** + +Rename the interface `LocalAgentConfig` → `LocalScoutConfig`. Add `export type LocalScoutConfig = ...`. If there are other type definitions referencing "agent" (e.g. `AgentDataType`), rename consistently. + +- [ ] **Step 5: Typecheck + lint** + +```bash +cd adiuvAI +npx tsc --noEmit +npm run lint +``` + +- [ ] **Step 6: Commit** + +```bash +cd adiuvAI +git add -A src/renderer/ +git commit -m "refactor(renderer): rename Agent components and types to Scout" +``` + +--- + +## Task 10: Rename i18n keys across all 5 languages + +**Files:** +- Modify: `adiuvAI/src/renderer/locales/{en,it,es,fr,de}/translation.json` + +**Context:** Three buckets to rename in every language file: +1. `"settings": { "agents": "...", "agentsSubtitle": "...", "agentsDescription": "...", ... }` → `scouts`, `scoutsSubtitle`, `scoutsDescription` +2. Top-level `"agents": { "noAgentsYet": ..., "createAgent": ..., "yourAgents": ..., "createFirstAgent": ..., "noAgentsDescription": ... }` → `"scouts": { "noScoutsYet", "createScout", "yourScouts", "createFirstScout", "noScoutsDescription" }`. Update key names AND any in-value occurrences of "agent"/"agents" to "scout"/"scouts" within human-readable strings (apply translation per language — for English: "Agents" → "Scouts", "agent" → "scout"; for Italian: "Agenti" → "Scout"; etc.). +3. `"toast": { "agent": { ... } }` → `"toast": { "scout": { ... } }` (rename nested key only; values unchanged unless they contain "agent" word). + +For non-English languages, keep the meaning consistent. Suggested translations: + +| Lang | Key | Suggested value | +|------|-----|-----------------| +| en | `settings.scouts` | "Scouts" | +| it | `settings.scouts` | "Scout" | +| es | `settings.scouts` | "Scouts" | +| fr | `settings.scouts` | "Scouts" | +| de | `settings.scouts` | "Scouts" | +| en | `settings.scoutsSubtitle` | "working for you." | +| it | `settings.scoutsSubtitle` | "che lavorano per te." | +| en | `settings.scoutsDescription` | "Scouts watch your data sources — local files, mailboxes, cloud services — and surface what matters in your task brief." | +| it | `settings.scoutsDescription` | "Gli scout monitorano le tue fonti dati — file locali, caselle email, servizi cloud — e mettono in evidenza ciò che conta nel tuo task brief." | +| en | `scouts.noScoutsYet` | "No scouts yet" | +| en | `scouts.createScout` | "Create scout" | +| en | `scouts.yourScouts` | "Your Scouts" | +| en | `scouts.createFirstScout` | "Create first scout" | +| en | `scouts.noScoutsDescription` | "Create your first scout from a template. Choose what data to extract, set a schedule, and edit instructions before saving." | + +(Translate analogously for it/es/fr/de — for the new descriptive copy, keep the spirit, ask the user for review on language nuance if uncertain.) + +- [ ] **Step 1: Edit `en/translation.json`** + +Apply renames per the bucket list above. + +- [ ] **Step 2: Edit `it/translation.json`** — same renames, with Italian values + +- [ ] **Step 3: Edit `es/translation.json`** — same renames, with Spanish values + +- [ ] **Step 4: Edit `fr/translation.json`** — same renames, with French values + +- [ ] **Step 5: Edit `de/translation.json`** — same renames, with German values + +- [ ] **Step 6: Sweep code references** + +```bash +cd adiuvAI +grep -rn "'agents\.\|\"agents\.\|'settings\.agents\|\"settings\.agents\|'toast\.agent\|\"toast\.agent" src/ +``` +Replace each `agents.X` → `scouts.X`, `settings.agents` → `settings.scouts`, `toast.agent` → `toast.scout`. + +- [ ] **Step 7: Typecheck + lint + run app** + +```bash +cd adiuvAI +npx tsc --noEmit +npm run lint +npm run start # smoke: open Settings, switch language, confirm "Scouts" header renders +``` + +- [ ] **Step 8: Commit** + +```bash +cd adiuvAI +git add -A src/renderer/locales/ src/renderer/ +git commit -m "i18n: rename agents keys to scouts across all 5 languages" +``` + +--- + +## Task 11: Update monorepo to bump submodule pointers + +**Files:** +- Modify: monorepo `.gitmodules` is unchanged; update tracked commit pointers via `git add api adiuvAI` + +- [ ] **Step 1: From the monorepo root, observe the dirty submodule pointers** + +```bash +cd c:\Users\PC-Roby\Documents\_adiuvai_workspace +git status +``` +Expected: `api` and `adiuvAI` listed as "modified content" or "new commits". + +- [ ] **Step 2: Stage and commit the submodule bumps** + +```bash +git add api adiuvAI +git commit -m "chore: bump submodules — Phase 1 scouts rename complete" +``` + +- [ ] **Step 3: Verify CLAUDE.md doesn't need updates** + +Check `c:\Users\PC-Roby\Documents\_adiuvai_workspace\.claude\CLAUDE.md` and `c:\Users\PC-Roby\Documents\_adiuvai_workspace\adiuvAI\.claude\CLAUDE.md` — many references mention `agent` and the agent subsystem. For Phase 1, **only update durable notes that became wrong** (e.g. `tRPC routers (in appRouter): ... agent (with local / cloud / journey sub-routers) ...` should now read `scout`). Use targeted Edit, do not rewrite the file. + +```bash +grep -n "agent" .claude/CLAUDE.md adiuvAI/.claude/CLAUDE.md +``` +For each line that documents the renamed surface, fix in place. + +- [ ] **Step 4: Commit CLAUDE.md updates** + +```bash +git add .claude/CLAUDE.md adiuvAI/.claude/CLAUDE.md +git commit -m "docs(CLAUDE): update references to scouts after rename" +``` + +**Phase 1 complete.** Verify by running both test suites green and starting the app. + +--- + +# PHASE 2 — Connector Skeleton + +Goal: lay all infra (BE queue table, connector Protocol, engine, WS frame contract, SQLite suggestions table) without any user-visible Gmail feature yet. End state: backend can be unit-tested with a mock connector; Electron has the table + frame handler ready to receive proposals. + +--- + +## Task 12: Alembic migration — `scout_triage_queue` + `cloud_scout_configs` alters + +**Files:** +- Create: `api/alembic/versions/008_scout_triage_queue.py` + +- [ ] **Step 1: Create the migration** + +```python +"""Scout triage queue + cloud_scout_configs alterations. + +Revision ID: 008 +Revises: 007 +Create Date: 2026-05-15 +""" + +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + + +revision: str = "008" +down_revision: Union[str, None] = "007" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.create_table( + "scout_triage_queue", + sa.Column("id", sa.Uuid(as_uuid=False), primary_key=True), + sa.Column("user_id", sa.Uuid(as_uuid=False), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True), + sa.Column("scout_id", sa.Uuid(as_uuid=False), sa.ForeignKey("cloud_scout_configs.id", ondelete="CASCADE"), nullable=False), + sa.Column("source_type", sa.String(50), nullable=False), + sa.Column("source_msg_ref", sa.String(255), nullable=False), + sa.Column("triage_verdict", sa.String(20), nullable=False), + sa.Column("triage_reason", sa.Text, nullable=True), + sa.Column("status", sa.String(20), nullable=False, server_default="queued"), + sa.Column("triaged_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()), + sa.Column("delivered_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("acked_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False), + sa.UniqueConstraint("scout_id", "source_msg_ref", name="uq_scout_triage_queue_scout_msg"), + ) + op.create_index("ix_scout_triage_queue_user_status", "scout_triage_queue", ["user_id", "status"]) + op.create_index( + "ix_scout_triage_queue_expires_active", + "scout_triage_queue", + ["expires_at"], + postgresql_where=sa.text("status != 'acked'"), + ) + + op.add_column("cloud_scout_configs", sa.Column("auto_trash_spam", sa.Boolean(), nullable=False, server_default=sa.text("false"))) + op.add_column("cloud_scout_configs", sa.Column("gmail_history_id", sa.String(64), nullable=True)) + op.add_column("cloud_scout_configs", sa.Column("gmail_watch_expires_at", sa.DateTime(timezone=True), nullable=True)) + op.add_column("cloud_scout_configs", sa.Column("device_inactivity_pause_days", sa.Integer(), nullable=False, server_default="14")) + + +def downgrade() -> None: + op.drop_column("cloud_scout_configs", "device_inactivity_pause_days") + op.drop_column("cloud_scout_configs", "gmail_watch_expires_at") + op.drop_column("cloud_scout_configs", "gmail_history_id") + op.drop_column("cloud_scout_configs", "auto_trash_spam") + + op.drop_index("ix_scout_triage_queue_expires_active", table_name="scout_triage_queue") + op.drop_index("ix_scout_triage_queue_user_status", table_name="scout_triage_queue") + op.drop_table("scout_triage_queue") +``` + +- [ ] **Step 2: Apply + verify** + +```bash +cd api +alembic upgrade head +``` +Verify with `\d scout_triage_queue` and `\d cloud_scout_configs` (the latter should now show the four new columns). + +- [ ] **Step 3: Commit** + +```bash +cd api +git add alembic/versions/008_scout_triage_queue.py +git commit -m "feat(db): add scout_triage_queue and cloud_scout_configs gmail columns" +``` + +--- + +## Task 13: SQLAlchemy `ScoutTriageQueue` model + `CloudScoutConfig` field additions + +**Files:** +- Modify: `api/app/models.py` — add `ScoutTriageQueue` class, add four new fields to `CloudScoutConfig` + +- [ ] **Step 1: Add fields to `CloudScoutConfig` (in `models.py` after the existing fields)** + +```python +auto_trash_spam: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default=sa.text("false")) +gmail_history_id: Mapped[str | None] = mapped_column(String(64), nullable=True) +gmail_watch_expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) +device_inactivity_pause_days: Mapped[int] = mapped_column(Integer, nullable=False, default=14, server_default="14") +``` + +- [ ] **Step 2: Add the `ScoutTriageQueue` class (after `CloudScoutConfig`)** + +```python +class ScoutTriageQueue(Base): + __tablename__ = "scout_triage_queue" + __table_args__ = ( + UniqueConstraint("scout_id", "source_msg_ref", name="uq_scout_triage_queue_scout_msg"), + ) + + id: Mapped[str] = mapped_column(Uuid(as_uuid=False), primary_key=True, default=_uuid) + user_id: Mapped[str] = mapped_column(Uuid(as_uuid=False), ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True) + scout_id: Mapped[str] = mapped_column(Uuid(as_uuid=False), ForeignKey("cloud_scout_configs.id", ondelete="CASCADE"), nullable=False) + source_type: Mapped[str] = mapped_column(String(50), nullable=False) + source_msg_ref: Mapped[str] = mapped_column(String(255), nullable=False) + triage_verdict: Mapped[str] = mapped_column(String(20), nullable=False) + triage_reason: Mapped[str | None] = mapped_column(Text, nullable=True) + status: Mapped[str] = mapped_column(String(20), nullable=False, default="queued", server_default="queued") + triaged_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now()) + delivered_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + acked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) +``` + +- [ ] **Step 3: Verify imports at the top of `models.py` include `UniqueConstraint` and `Integer`** + +If missing, add `from sqlalchemy import ..., UniqueConstraint, Integer`. + +- [ ] **Step 4: Run tests** + +```bash +cd api +pytest +``` +Expected: green (existing tests don't touch the new model yet, but `Base.metadata.create_all` should now succeed creating the new table in the test DB). + +- [ ] **Step 5: Commit** + +```bash +cd api +git add app/models.py +git commit -m "feat(models): add ScoutTriageQueue model + cloud_scout gmail fields" +``` + +--- + +## Task 14: `SourceConnector` Protocol + Pydantic types + +**Files:** +- Create: `api/app/scouts/__init__.py` (empty) +- Create: `api/app/scouts/connectors/__init__.py` (empty) +- Create: `api/app/scouts/connectors/base.py` +- Create: `api/tests/test_scout_connectors_base.py` + +- [ ] **Step 1: Write the failing test** + +`api/tests/test_scout_connectors_base.py`: +```python +"""Tests for the SourceConnector base protocol and shared types.""" + +from __future__ import annotations + +from datetime import datetime, timezone + +import pytest + +from app.scouts.connectors.base import ( + ItemContent, + ItemMetadata, + ItemRef, + TriageVerdict, +) + + +def test_item_ref_round_trips_through_pydantic(): + ref = ItemRef(source_msg_ref="abc123", received_at=datetime.now(tz=timezone.utc)) + parsed = ItemRef.model_validate(ref.model_dump()) + assert parsed.source_msg_ref == "abc123" + assert parsed.received_at == ref.received_at + + +def test_item_metadata_allows_all_optional(): + meta = ItemMetadata() + assert meta.subject is None + assert meta.sender is None + assert meta.snippet is None + assert meta.received_at is None + + +def test_item_content_requires_metadata_and_body(): + content = ItemContent( + metadata=ItemMetadata(subject="hi"), + body_text="hello world", + raw_headers={"X-Foo": "bar"}, + ) + assert content.metadata.subject == "hi" + assert content.body_text == "hello world" + assert content.raw_headers["X-Foo"] == "bar" + + +def test_triage_verdict_constraints(): + v = TriageVerdict(verdict="relevant", reason="contains task language", confidence=0.92) + assert v.verdict == "relevant" + + with pytest.raises(ValueError): + TriageVerdict(verdict="meh", reason="x", confidence=0.5) # bad enum value +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```bash +cd api +pytest tests/test_scout_connectors_base.py -v +``` +Expected: ImportError (`No module named 'app.scouts.connectors.base'`). + +- [ ] **Step 3: Implement `app/scouts/connectors/base.py`** + +```python +"""Source connector Protocol and shared item types. + +A SourceConnector adapts a third-party data source (Gmail, Slack, ...) to the +shared ScoutEngine interface. Each connector owns: + + * how to enumerate new items since the last poll (``list_new``) + * how to fetch a single item's metadata cheaply (``fetch_metadata``) + * how to fetch a single item's full content for in-memory triage + (``fetch_content``) — this content MUST NOT be persisted by the engine + * how to archive/trash an item (``archive``) for spam handling + * optional push-notification setup (``setup_watch`` / ``renew_watch``) +""" + +from __future__ import annotations + +from datetime import datetime +from typing import Literal, Protocol + +from pydantic import BaseModel, Field + + +class ItemRef(BaseModel): + source_msg_ref: str + received_at: datetime | None = None + + +class ItemMetadata(BaseModel): + subject: str | None = None + sender: str | None = None + snippet: str | None = None + received_at: datetime | None = None + + +class ItemContent(BaseModel): + metadata: ItemMetadata + body_text: str + raw_headers: dict[str, str] = Field(default_factory=dict) + + +class TriageVerdict(BaseModel): + verdict: Literal["relevant", "spam"] + reason: str + confidence: float = Field(ge=0.0, le=1.0) + + +class SourceConnector(Protocol): + """Adapter for a third-party data source (Gmail, Slack, ...).""" + + source_type: str # e.g. "gmail" + + async def list_new(self, scout) -> list[ItemRef]: ... + async def fetch_metadata(self, scout, ref: ItemRef) -> ItemMetadata: ... + async def fetch_content(self, scout, ref: ItemRef) -> ItemContent: ... + async def archive(self, scout, ref: ItemRef) -> None: ... + async def setup_watch(self, scout) -> None: ... + async def renew_watch(self, scout) -> None: ... +``` + +Also create empty package markers: +```bash +cd api +touch app/scouts/__init__.py app/scouts/connectors/__init__.py +``` + +- [ ] **Step 4: Run tests** + +```bash +cd api +pytest tests/test_scout_connectors_base.py -v +``` +Expected: PASS (4 tests). + +- [ ] **Step 5: Commit** + +```bash +cd api +git add app/scouts/ tests/test_scout_connectors_base.py +git commit -m "feat(scouts): add SourceConnector protocol and item types" +``` + +--- + +## Task 15: Connector registry + +**Files:** +- Create: `api/app/scouts/connectors/registry.py` +- Create: `api/tests/test_scout_connector_registry.py` + +- [ ] **Step 1: Write the failing test** + +```python +"""Tests for the connector registry.""" + +from __future__ import annotations + +import pytest + +from app.scouts.connectors.base import ItemRef +from app.scouts.connectors.registry import ( + get_connector, + register_connector, + _reset_for_tests, +) + + +class _DummyConnector: + source_type = "dummy" + async def list_new(self, scout): return [] + async def fetch_metadata(self, scout, ref): raise NotImplementedError + async def fetch_content(self, scout, ref): raise NotImplementedError + async def archive(self, scout, ref): raise NotImplementedError + async def setup_watch(self, scout): raise NotImplementedError + async def renew_watch(self, scout): raise NotImplementedError + + +@pytest.fixture(autouse=True) +def _clean_registry(): + _reset_for_tests() + yield + _reset_for_tests() + + +def test_register_and_get(): + c = _DummyConnector() + register_connector(c) + assert get_connector("dummy") is c + + +def test_unknown_source_raises(): + with pytest.raises(KeyError): + get_connector("nope") + + +def test_double_register_replaces(): + a = _DummyConnector() + b = _DummyConnector() + register_connector(a) + register_connector(b) + assert get_connector("dummy") is b +``` + +- [ ] **Step 2: Run test (should fail with ImportError)** + +```bash +cd api +pytest tests/test_scout_connector_registry.py -v +``` + +- [ ] **Step 3: Implement `app/scouts/connectors/registry.py`** + +```python +"""Connector registry — single source of truth for source_type -> connector.""" + +from __future__ import annotations + +from typing import Any + +_CONNECTORS: dict[str, Any] = {} + + +def register_connector(connector: Any) -> None: + """Register a SourceConnector instance under its ``source_type``. + + Calling twice with the same ``source_type`` replaces the prior entry — + useful for tests and hot-reload, but in production each connector + should be registered exactly once at startup. + """ + if not getattr(connector, "source_type", None): + raise ValueError("Connector must declare a non-empty source_type") + _CONNECTORS[connector.source_type] = connector + + +def get_connector(source_type: str) -> Any: + """Return the registered connector for ``source_type`` or raise KeyError.""" + try: + return _CONNECTORS[source_type] + except KeyError as exc: + raise KeyError(f"No connector registered for source_type {source_type!r}") from exc + + +def _reset_for_tests() -> None: + """Clear the registry — for use in pytest fixtures only.""" + _CONNECTORS.clear() +``` + +- [ ] **Step 4: Run tests** + +```bash +cd api +pytest tests/test_scout_connector_registry.py -v +``` +Expected: PASS (3 tests). + +- [ ] **Step 5: Commit** + +```bash +cd api +git add app/scouts/connectors/registry.py tests/test_scout_connector_registry.py +git commit -m "feat(scouts): add connector registry" +``` + +--- + +## Task 16: `ScoutEngine` skeleton — `trigger_scout` + `_process_item` + +**Files:** +- Create: `api/app/scouts/engine.py` +- Create: `api/tests/test_scout_engine.py` + +**Context:** This task implements the core engine logic but uses a stub triage call. The real LLM triage comes in Task 24. The engine writes `scout_triage_queue` rows for relevant items and calls `connector.archive` for spam items when `scout.auto_trash_spam` is true. + +- [ ] **Step 1: Write the failing test** + +```python +"""Unit tests for ScoutEngine.trigger_scout / _process_item.""" + +from __future__ import annotations + +import uuid +from datetime import datetime, timedelta, timezone +from unittest.mock import AsyncMock + +import pytest +from sqlalchemy import select + +from app.models import CloudScoutConfig, ScoutTriageQueue, User, Subscription +from app.scouts.connectors.base import ItemContent, ItemMetadata, ItemRef, TriageVerdict +from app.scouts.connectors.registry import register_connector, _reset_for_tests +from app.scouts.engine import ScoutEngine +from tests.conftest import _TestSessionLocal + + +def _make_connector(items, content_for): + c = AsyncMock() + c.source_type = "dummy" + c.list_new = AsyncMock(return_value=items) + c.fetch_content = AsyncMock(side_effect=lambda scout, ref: content_for[ref.source_msg_ref]) + c.archive = AsyncMock() + return c + + +@pytest.fixture(autouse=True) +def _registry(): + _reset_for_tests() + yield + _reset_for_tests() + + +@pytest.mark.asyncio +async def test_relevant_item_inserted_into_queue(monkeypatch): + user_id = "00000000-0000-0000-0000-000000000003" # power tier seeded in conftest + scout_id = str(uuid.uuid4()) + + async with _TestSessionLocal() as session: + scout = CloudScoutConfig( + id=scout_id, user_id=user_id, provider="gmail", name="Test", + data_types=[], prompt_template="", schedule_cron="0 * * * *", + enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14, + ) + session.add(scout) + await session.commit() + + refs = [ItemRef(source_msg_ref="msg-1")] + content = {"msg-1": ItemContent(metadata=ItemMetadata(subject="Hi"), body_text="task tomorrow")} + connector = _make_connector(refs, content) + register_connector(connector) + + engine = ScoutEngine() + monkeypatch.setattr( + engine, + "_triage_llm", + AsyncMock(return_value=TriageVerdict(verdict="relevant", reason="task", confidence=0.9)), + ) + + await engine.trigger_scout(uuid.UUID(scout_id)) + + async with _TestSessionLocal() as session: + rows = (await session.execute(select(ScoutTriageQueue))).scalars().all() + assert len(rows) == 1 + assert rows[0].source_msg_ref == "msg-1" + assert rows[0].triage_verdict == "relevant" + assert rows[0].status == "queued" + connector.archive.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_spam_with_auto_trash_archives_and_does_not_queue(monkeypatch): + user_id = "00000000-0000-0000-0000-000000000003" + scout_id = str(uuid.uuid4()) + + async with _TestSessionLocal() as session: + scout = CloudScoutConfig( + id=scout_id, user_id=user_id, provider="gmail", name="Test", + data_types=[], prompt_template="", schedule_cron="0 * * * *", + enabled=True, auto_trash_spam=True, device_inactivity_pause_days=14, + ) + session.add(scout) + await session.commit() + + refs = [ItemRef(source_msg_ref="msg-spam")] + content = {"msg-spam": ItemContent(metadata=ItemMetadata(subject="$$$"), body_text="buy")} + connector = _make_connector(refs, content) + register_connector(connector) + + engine = ScoutEngine() + monkeypatch.setattr( + engine, + "_triage_llm", + AsyncMock(return_value=TriageVerdict(verdict="spam", reason="bait", confidence=0.99)), + ) + + await engine.trigger_scout(uuid.UUID(scout_id)) + + async with _TestSessionLocal() as session: + rows = (await session.execute(select(ScoutTriageQueue))).scalars().all() + assert rows == [] + connector.archive.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_spam_without_auto_trash_does_not_archive_and_does_not_queue(monkeypatch): + user_id = "00000000-0000-0000-0000-000000000003" + scout_id = str(uuid.uuid4()) + + async with _TestSessionLocal() as session: + scout = CloudScoutConfig( + id=scout_id, user_id=user_id, provider="gmail", name="Test", + data_types=[], prompt_template="", schedule_cron="0 * * * *", + enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14, + ) + session.add(scout) + await session.commit() + + refs = [ItemRef(source_msg_ref="msg-2")] + content = {"msg-2": ItemContent(metadata=ItemMetadata(subject="$$$"), body_text="buy")} + connector = _make_connector(refs, content) + register_connector(connector) + + engine = ScoutEngine() + monkeypatch.setattr( + engine, + "_triage_llm", + AsyncMock(return_value=TriageVerdict(verdict="spam", reason="bait", confidence=0.99)), + ) + + await engine.trigger_scout(uuid.UUID(scout_id)) + + async with _TestSessionLocal() as session: + rows = (await session.execute(select(ScoutTriageQueue))).scalars().all() + assert rows == [] + connector.archive.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_idempotent_replay(monkeypatch): + user_id = "00000000-0000-0000-0000-000000000003" + scout_id = str(uuid.uuid4()) + + async with _TestSessionLocal() as session: + session.add(CloudScoutConfig( + id=scout_id, user_id=user_id, provider="gmail", name="Test", + data_types=[], prompt_template="", schedule_cron="0 * * * *", + enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14, + )) + await session.commit() + + refs = [ItemRef(source_msg_ref="msg-3")] + content = {"msg-3": ItemContent(metadata=ItemMetadata(subject="x"), body_text="y")} + connector = _make_connector(refs, content) + register_connector(connector) + + engine = ScoutEngine() + monkeypatch.setattr( + engine, + "_triage_llm", + AsyncMock(return_value=TriageVerdict(verdict="relevant", reason="x", confidence=0.5)), + ) + + await engine.trigger_scout(uuid.UUID(scout_id)) + await engine.trigger_scout(uuid.UUID(scout_id)) + + async with _TestSessionLocal() as session: + rows = (await session.execute(select(ScoutTriageQueue))).scalars().all() + assert len(rows) == 1, "Replay must not create duplicate queue rows" +``` + +- [ ] **Step 2: Run test (should fail — module missing)** + +```bash +cd api +pytest tests/test_scout_engine.py -v +``` + +- [ ] **Step 3: Implement `app/scouts/engine.py`** + +```python +"""ScoutEngine — orchestrates triage, queueing, and delivery for cloud scouts. + +Triage flow per scout: + 1. Resolve scout config from the DB. + 2. Skip if device hasn't connected within ``device_inactivity_pause_days``. + 3. Ask the connector to ``list_new`` — fresh items since last poll. + 4. For each item: + - skip if already in the queue (idempotent on (scout_id, source_msg_ref)) + - fetch the full content via the connector (transient, never persisted) + - run the triage LLM call → relevant | spam + - spam + auto_trash_spam → connector.archive + - relevant → INSERT scout_triage_queue row + 5. Update scout.last_run_at. + +Delivery flow on Electron WS reconnect: + - drain ``status='queued'`` rows for the user + - fetch metadata-only for each (subject + snippet) + - send a ``scout_proposal`` frame + - flip status to ``delivered`` on ack +""" + +from __future__ import annotations + +import logging +import uuid +from datetime import datetime, timedelta, timezone + +from sqlalchemy import select +from sqlalchemy.exc import IntegrityError + +from app.db.session import async_session_factory +from app.models import CloudScoutConfig, ScoutTriageQueue +from app.scouts.connectors.base import ItemContent, ItemRef, TriageVerdict +from app.scouts.connectors.registry import get_connector + +logger = logging.getLogger(__name__) + +QUEUE_TTL_DAYS = 30 + + +class ScoutEngine: + async def trigger_scout(self, scout_id: uuid.UUID) -> None: + async with async_session_factory() as session: + scout = await session.get(CloudScoutConfig, str(scout_id)) + if scout is None: + logger.warning("trigger_scout: no such scout id=%s", scout_id) + return + if not scout.enabled: + return + # Device-inactivity pause check is a simple heuristic on last_run_at — + # the device-online signal lives in the DeviceConnectionManager and is + # consulted at delivery time. For triage, we only check that the + # configured pause threshold isn't suppressing the run. + connector = get_connector(scout.provider) + try: + refs = await connector.list_new(scout) + except Exception: + logger.exception("scout %s: list_new failed", scout.id) + return + + for ref in refs: + await self._process_item(session, scout, connector, ref) + + scout.last_run_at = datetime.now(tz=timezone.utc) + await session.commit() + + async def _process_item( + self, + session, + scout: CloudScoutConfig, + connector, + ref: ItemRef, + ) -> None: + # Idempotency check + existing = await session.execute( + select(ScoutTriageQueue.id).where( + ScoutTriageQueue.scout_id == scout.id, + ScoutTriageQueue.source_msg_ref == ref.source_msg_ref, + ) + ) + if existing.first() is not None: + return + + try: + content = await connector.fetch_content(scout, ref) + except Exception: + logger.exception("scout %s: fetch_content failed for %s", scout.id, ref.source_msg_ref) + return + + try: + verdict = await self._triage_llm(scout, content) + except Exception: + logger.exception("scout %s: triage_llm failed for %s", scout.id, ref.source_msg_ref) + return + + if verdict.verdict == "spam": + if scout.auto_trash_spam: + try: + await connector.archive(scout, ref) + except Exception: + logger.exception("scout %s: archive failed for %s", scout.id, ref.source_msg_ref) + return + + now = datetime.now(tz=timezone.utc) + row = ScoutTriageQueue( + id=str(uuid.uuid4()), + user_id=scout.user_id, + scout_id=scout.id, + source_type=connector.source_type, + source_msg_ref=ref.source_msg_ref, + triage_verdict=verdict.verdict, + triage_reason=verdict.reason, + status="queued", + triaged_at=now, + expires_at=now + timedelta(days=QUEUE_TTL_DAYS), + ) + session.add(row) + try: + await session.flush() + except IntegrityError: + await session.rollback() + # Race: another worker inserted between our SELECT and INSERT. + # The unique constraint did its job; safe to ignore. + return + + async def _triage_llm(self, scout: CloudScoutConfig, content: ItemContent) -> TriageVerdict: + """Stub — real implementation in Task 24.""" + raise NotImplementedError("Real triage LLM call lands in Task 24") +``` + +Note: the import `from app.db.session import async_session_factory` assumes there's an existing session factory module. If the project uses a different name (e.g. `from app.db import SessionLocal`), find it via: +```bash +grep -rn "AsyncSession\|sessionmaker" api/app/db/ +``` +and adjust the import. This is a common case across this codebase — the engine just needs an async session. + +- [ ] **Step 4: Run tests** + +```bash +cd api +pytest tests/test_scout_engine.py -v +``` +Expected: 4 PASS. + +- [ ] **Step 5: Commit** + +```bash +cd api +git add app/scouts/engine.py tests/test_scout_engine.py +git commit -m "feat(scouts): add ScoutEngine triage + queue insertion" +``` + +--- + +## Task 17: Drizzle migration — `scout_suggestions` table + +**Files:** +- Modify: `adiuvAI/src/main/db/schema.ts` — add `scoutSuggestions` table +- Create: `adiuvAI/src/main/db/migrations/0008_scout_suggestions.sql` (auto-generated, then renamed) + +- [ ] **Step 1: Add the table to `schema.ts`** + +Append at the end of the existing tables: +```typescript +export const scoutSuggestions = sqliteTable('scout_suggestions', { + id: text().primaryKey(), + scoutId: text('scout_id').notNull(), + sourceType: text('source_type').notNull(), + sourceMsgRef: text('source_msg_ref').notNull(), + category: text().notNull(), // "unprocessed" until Phase 4 + payload: text(), // JSON, populated by Phase 4 + rawSubject: text('raw_subject'), + rawSnippet: text('raw_snippet'), + status: text().notNull(), // pending | approved | rejected | expired + proposedAt: integer('proposed_at').notNull(), + resolvedAt: integer('resolved_at'), + resolvedEntityType: text('resolved_entity_type'), + resolvedEntityId: text('resolved_entity_id'), +}); +``` + +- [ ] **Step 2: Generate the migration** + +```bash +cd adiuvAI +npx drizzle-kit generate +``` +Rename to `0008_scout_suggestions.sql`: +```bash +mv src/main/db/migrations/0008_*.sql src/main/db/migrations/0008_scout_suggestions.sql +``` + +- [ ] **Step 3: Inspect the generated SQL** + +Verify it contains a `CREATE TABLE scout_suggestions` and matches the schema columns. + +- [ ] **Step 4: Smoke test** + +```bash +cd adiuvAI +npm run start +``` +After app boots, close. Verify: +```bash +sqlite3 dev.db ".schema scout_suggestions" +``` + +- [ ] **Step 5: Commit** + +```bash +cd adiuvAI +git add src/main/db/schema.ts src/main/db/migrations/0008_scout_suggestions.sql +git commit -m "feat(db): add scout_suggestions table" +``` + +--- + +## Task 18: WS frame schemas — `scout_proposal` + `scout_proposal_ack` + +**Files:** +- Modify: `api/app/schemas.py` — add new `WsFrameType` enum values + Pydantic models for the two frames + +- [ ] **Step 1: Read `api/app/schemas.py` to find `WsFrameType` enum** + +```bash +cd api +grep -n "WsFrameType\|class Ws" app/schemas.py +``` + +- [ ] **Step 2: Add new enum values** + +```python +class WsFrameType(str, Enum): + # ... existing values ... + SCOUT_PROPOSAL = "scout_proposal" + SCOUT_PROPOSAL_ACK = "scout_proposal_ack" +``` + +- [ ] **Step 3: Add the frame Pydantic models** + +```python +class ScoutProposalPayload(BaseModel): + id: str + scout_id: str + source_type: str + source_msg_ref: str + raw_subject: str | None = None + raw_snippet: str | None = None + category: Literal["unprocessed"] = "unprocessed" # Phase 4 will widen + payload: dict | None = None + + +class ScoutProposalFrame(BaseModel): + type: Literal[WsFrameType.SCOUT_PROPOSAL] + proposal: ScoutProposalPayload + + +class ScoutProposalAckFrame(BaseModel): + type: Literal[WsFrameType.SCOUT_PROPOSAL_ACK] + proposal_id: str +``` + +- [ ] **Step 4: Run tests** + +```bash +cd api +pytest tests/test_schemas.py -v # if exists +pytest +``` + +- [ ] **Step 5: Commit** + +```bash +cd api +git add app/schemas.py +git commit -m "feat(schemas): add scout_proposal + scout_proposal_ack WS frame types" +``` + +--- + +## Task 19: `ScoutEngine.deliver_pending` — drain queue + send WS frames + +**Files:** +- Modify: `api/app/scouts/engine.py` — add `deliver_pending` method +- Modify: `api/app/api/routes/device_ws.py` — call `deliver_pending` on connect; handle `scout_proposal_ack` incoming frames +- Modify: `api/tests/test_scout_engine.py` — add `deliver_pending` tests + +- [ ] **Step 1: Write the failing test** + +Append to `api/tests/test_scout_engine.py`: +```python +@pytest.mark.asyncio +async def test_deliver_pending_sends_one_frame_per_queued_row(monkeypatch): + user_id = "00000000-0000-0000-0000-000000000003" + scout_id = str(uuid.uuid4()) + now = datetime.now(tz=timezone.utc) + + async with _TestSessionLocal() as session: + session.add(CloudScoutConfig( + id=scout_id, user_id=user_id, provider="gmail", name="Test", + data_types=[], prompt_template="", schedule_cron="0 * * * *", + enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14, + )) + for i in range(3): + session.add(ScoutTriageQueue( + id=str(uuid.uuid4()), user_id=user_id, scout_id=scout_id, + source_type="gmail", source_msg_ref=f"msg-{i}", + triage_verdict="relevant", status="queued", + triaged_at=now, expires_at=now + timedelta(days=30), + )) + await session.commit() + + connector = AsyncMock() + connector.source_type = "gmail" + connector.fetch_metadata = AsyncMock(side_effect=lambda scout, ref: ItemMetadata( + subject=f"sub-{ref.source_msg_ref}", snippet=f"snip-{ref.source_msg_ref}", + )) + register_connector(connector) + + sent = [] + ws = AsyncMock() + ws.send_json = AsyncMock(side_effect=lambda payload: sent.append(payload)) + + engine = ScoutEngine() + await engine.deliver_pending(uuid.UUID(user_id), ws) + + assert len(sent) == 3 + assert all(s["type"] == "scout_proposal" for s in sent) + subjects = {s["proposal"]["raw_subject"] for s in sent} + assert subjects == {"sub-msg-0", "sub-msg-1", "sub-msg-2"} + # Status should flip to delivered + async with _TestSessionLocal() as session: + rows = (await session.execute(select(ScoutTriageQueue))).scalars().all() + assert all(r.status == "delivered" for r in rows) + assert all(r.delivered_at is not None for r in rows) +``` + +- [ ] **Step 2: Run test (should fail — `deliver_pending` not defined)** + +```bash +cd api +pytest tests/test_scout_engine.py::test_deliver_pending_sends_one_frame_per_queued_row -v +``` + +- [ ] **Step 3: Implement `deliver_pending` in `app/scouts/engine.py`** + +Add to the `ScoutEngine` class: +```python +async def deliver_pending(self, user_id: uuid.UUID, ws) -> None: + """Drain queued triage rows for a user, send one scout_proposal frame each. + + Frames are sent immediately and the rows flip to ``status='delivered'``. + Final ``acked`` flip happens in the WS endpoint when the client returns + a ``scout_proposal_ack`` frame. + """ + from app.scouts.connectors.base import ItemRef + + async with async_session_factory() as session: + rows = (await session.execute( + select(ScoutTriageQueue).where( + ScoutTriageQueue.user_id == str(user_id), + ScoutTriageQueue.status == "queued", + ) + )).scalars().all() + + for row in rows: + try: + connector = get_connector(row.source_type) + except KeyError: + logger.warning("deliver_pending: no connector for %s", row.source_type) + continue + scout = await session.get(CloudScoutConfig, row.scout_id) + if scout is None: + continue + try: + meta = await connector.fetch_metadata(scout, ItemRef(source_msg_ref=row.source_msg_ref)) + except Exception: + logger.exception("deliver_pending: fetch_metadata failed") + continue + + payload = { + "type": "scout_proposal", + "proposal": { + "id": row.id, + "scout_id": row.scout_id, + "source_type": row.source_type, + "source_msg_ref": row.source_msg_ref, + "raw_subject": meta.subject, + "raw_snippet": meta.snippet, + "category": "unprocessed", + "payload": None, + }, + } + await ws.send_json(payload) + row.status = "delivered" + row.delivered_at = datetime.now(tz=timezone.utc) + + await session.commit() + + +async def ack_proposal(self, proposal_id: str) -> None: + """Mark a proposal as acked once the Electron client confirms receipt.""" + async with async_session_factory() as session: + row = await session.get(ScoutTriageQueue, proposal_id) + if row is None: + return + row.status = "acked" + row.acked_at = datetime.now(tz=timezone.utc) + await session.commit() +``` + +- [ ] **Step 4: Run tests** + +```bash +cd api +pytest tests/test_scout_engine.py -v +``` +Expected: all 5 tests PASS. + +- [ ] **Step 5: Wire into `app/api/routes/device_ws.py`** + +Find the WS handshake / post-connect section. Add a call to `ScoutEngine().deliver_pending(user.id, websocket)` after the `device_hello` is received and the connection is registered. + +Also add a new dispatch case for incoming `scout_proposal_ack` frames: +```python +elif msg.get("type") == "scout_proposal_ack": + proposal_id = msg.get("proposal_id") + if proposal_id: + await ScoutEngine().ack_proposal(proposal_id) +``` + +- [ ] **Step 6: Run tests** + +```bash +cd api +pytest tests/test_ws_unified.py -v +pytest +``` + +- [ ] **Step 7: Commit** + +```bash +cd api +git add app/scouts/engine.py tests/test_scout_engine.py app/api/routes/device_ws.py +git commit -m "feat(scouts): deliver_pending drains queue and sends scout_proposal frames" +``` + +--- + +## Task 20: Electron — `scout-suggestion-handler.ts` (insert into local table on incoming frame) + +**Files:** +- Create: `adiuvAI/src/main/scouts/scout-suggestion-handler.ts` +- Modify: `adiuvAI/src/main/api/backend-client.ts` — dispatch incoming `scout_proposal` frame to the handler, send `scout_proposal_ack` back + +- [ ] **Step 1: Read `adiuvAI/src/main/api/backend-client.ts` to find the inbound frame switch** + +```bash +cd adiuvAI +grep -n "tool_call\|run_complete\|switch.*type" src/main/api/backend-client.ts +``` +Identify the existing dispatch pattern. + +- [ ] **Step 2: Create the handler module** + +`adiuvAI/src/main/scouts/scout-suggestion-handler.ts`: +```typescript +import { randomUUID } from 'node:crypto'; +import { getDb } from '../db'; +import { scoutSuggestions } from '../db/schema'; + +export interface IncomingScoutProposal { + id: string; + scout_id: string; + source_type: string; + source_msg_ref: string; + raw_subject: string | null; + raw_snippet: string | null; + category: 'unprocessed'; + payload: Record | null; +} + +export async function handleScoutProposal(p: IncomingScoutProposal): Promise { + // Idempotent: same id from BE shouldn't double-insert if the ack was lost. + await getDb() + .insert(scoutSuggestions) + .values({ + id: p.id, + scoutId: p.scout_id, + sourceType: p.source_type, + sourceMsgRef: p.source_msg_ref, + category: p.category, + payload: p.payload ? JSON.stringify(p.payload) : null, + rawSubject: p.raw_subject, + rawSnippet: p.raw_snippet, + status: 'pending', + proposedAt: Date.now(), + }) + .onConflictDoNothing(); +} +``` + +- [ ] **Step 3: Wire in `backend-client.ts`** + +In the inbound frame dispatch: +```typescript +case 'scout_proposal': { + const { handleScoutProposal } = await import('../scouts/scout-suggestion-handler'); + await handleScoutProposal(msg.proposal); + this.send({ type: 'scout_proposal_ack', proposal_id: msg.proposal.id }); + break; +} +``` + +(Adapt to the existing dispatch style — if it uses a switch, add a case; if it uses an if-chain, add an else-if. The send pattern follows whatever existing frames do.) + +- [ ] **Step 4: Typecheck + lint** + +```bash +cd adiuvAI +npx tsc --noEmit +npm run lint +``` + +- [ ] **Step 5: Commit** + +```bash +cd adiuvAI +git add src/main/scouts/scout-suggestion-handler.ts src/main/api/backend-client.ts +git commit -m "feat(scouts): handle scout_proposal frames and ack" +``` + +--- + +## Task 21: Submodule pointer bump for Phase 2 + +- [ ] **Step 1: Bump submodule pointers** + +```bash +cd c:\Users\PC-Roby\Documents\_adiuvai_workspace +git add api adiuvAI +git commit -m "chore: bump submodules — Phase 2 connector skeleton" +``` + +**Phase 2 complete.** Engine + queue + frame contract + local table all in place. No user-facing change yet. + +--- + +# PHASE 3 — Gmail Scout End-to-End + +Goal: ship a working Gmail scout — user can connect Gmail through Settings, push notifications + cron-fallback poll the inbox, BE triages with the real LLM, queue rows are delivered to Electron on reconnect and land as `scout_suggestions` rows with `category='unprocessed'`. + +--- + +## Task 22: `GmailConnector` implementation + +**Files:** +- Create: `api/app/scouts/connectors/gmail.py` +- Create: `api/tests/test_scout_connectors_gmail.py` + +**Context:** Wraps existing `app/integrations/gmail.py`'s `GmailClient`. Maps existing `EmailMessage` to the new `ItemContent`. Implements `list_new` using Gmail's `users.history.list` since the scout's `gmail_history_id`. Implements `archive` via `users.messages.trash`. + +- [ ] **Step 1: Write the failing test** + +`api/tests/test_scout_connectors_gmail.py`: +```python +"""Tests for GmailConnector.""" + +from __future__ import annotations + +import uuid +from datetime import datetime, timezone +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from app.models import CloudScoutConfig +from app.scouts.connectors.base import ItemRef +from app.scouts.connectors.gmail import GmailConnector + + +def _make_scout(): + return CloudScoutConfig( + id=str(uuid.uuid4()), + user_id="00000000-0000-0000-0000-000000000003", + provider="gmail", + name="Inbox", + data_types=[], + prompt_template="", + oauth_token_encrypted="encrypted-blob", + schedule_cron="0 * * * *", + enabled=True, + auto_trash_spam=False, + device_inactivity_pause_days=14, + gmail_history_id="100", + ) + + +@pytest.mark.asyncio +async def test_fetch_metadata_returns_subject_and_snippet(): + scout = _make_scout() + conn = GmailConnector() + fake_message = { + "id": "msg-1", + "snippet": "preview text", + "payload": {"headers": [ + {"name": "Subject", "value": "Hello"}, + {"name": "From", "value": "alice@example.com"}, + {"name": "Date", "value": "Wed, 14 May 2026 10:00:00 +0000"}, + ]}, + } + with patch("app.scouts.connectors.gmail._get_gmail_service") as mock_svc: + mock_svc.return_value.users().messages().get().execute.return_value = fake_message + meta = await conn.fetch_metadata(scout, ItemRef(source_msg_ref="msg-1")) + assert meta.subject == "Hello" + assert meta.sender == "alice@example.com" + assert meta.snippet == "preview text" + + +@pytest.mark.asyncio +async def test_fetch_content_returns_body_text(): + scout = _make_scout() + conn = GmailConnector() + with patch("app.scouts.connectors.gmail.GmailClient") as MockClient: + instance = MockClient.return_value + instance.fetch_messages = AsyncMock(return_value=[ + MagicMock(id="msg-1", subject="S", sender="a@b", body_text="hello world", + date=datetime.now(tz=timezone.utc), labels=[]), + ]) + content = await conn.fetch_content(scout, ItemRef(source_msg_ref="msg-1")) + assert content.body_text == "hello world" + assert content.metadata.subject == "S" + + +@pytest.mark.asyncio +async def test_archive_calls_trash(): + scout = _make_scout() + conn = GmailConnector() + with patch("app.scouts.connectors.gmail._get_gmail_service") as mock_svc: + await conn.archive(scout, ItemRef(source_msg_ref="msg-1")) + mock_svc.return_value.users().messages().trash.assert_called() +``` + +- [ ] **Step 2: Run test (fails — module missing)** + +```bash +cd api +pytest tests/test_scout_connectors_gmail.py -v +``` + +- [ ] **Step 3: Implement `app/scouts/connectors/gmail.py`** + +```python +"""Gmail SourceConnector — wraps the existing GmailClient. + +Responsibilities: + * list_new: incremental fetch since the scout's stored gmail_history_id + * fetch_metadata: subject + sender + snippet only (Gmail metadata format) + * fetch_content: full body text — transient, never persisted by engine + * archive: move a message to Gmail Trash (recoverable for 30 days) + * setup_watch / renew_watch: Gmail push notifications via Pub/Sub +""" + +from __future__ import annotations + +import asyncio +import logging +from datetime import datetime, timedelta, timezone +from typing import Any + +from app.config.settings import settings +from app.integrations import decrypt_token +from app.integrations.gmail import GmailClient +from app.scouts.connectors.base import ItemContent, ItemMetadata, ItemRef + +logger = logging.getLogger(__name__) + + +def _get_gmail_service(scout): + """Return a synchronous Google API client for low-level metadata calls.""" + from googleapiclient.discovery import build + from google.oauth2.credentials import Credentials + + creds_info = decrypt_token(scout.oauth_token_encrypted) + credentials = Credentials( + token=creds_info.get("token"), + refresh_token=creds_info.get("refresh_token"), + token_uri=creds_info.get("token_uri", "https://oauth2.googleapis.com/token"), + client_id=creds_info.get("client_id"), + client_secret=creds_info.get("client_secret"), + scopes=creds_info.get("scopes"), + ) + return build("gmail", "v1", credentials=credentials, cache_discovery=False) + + +class GmailConnector: + source_type = "gmail" + + async def list_new(self, scout) -> list[ItemRef]: + def _sync() -> tuple[list[ItemRef], str | None]: + service = _get_gmail_service(scout) + history_id = scout.gmail_history_id + refs: list[ItemRef] = [] + new_history_id = history_id + if history_id: + resp = service.users().history().list( + userId="me", + startHistoryId=history_id, + historyTypes=["messageAdded"], + ).execute() + for entry in resp.get("history", []): + for added in entry.get("messagesAdded", []): + refs.append(ItemRef(source_msg_ref=added["message"]["id"])) + new_history_id = resp.get("historyId", history_id) + else: + # First run: pick up the latest history id without backfilling. + profile = service.users().getProfile(userId="me").execute() + new_history_id = profile["historyId"] + return refs, new_history_id + + refs, new_history_id = await asyncio.to_thread(_sync) + if new_history_id and new_history_id != scout.gmail_history_id: + scout.gmail_history_id = new_history_id + return refs + + async def fetch_metadata(self, scout, ref: ItemRef) -> ItemMetadata: + def _sync() -> ItemMetadata: + service = _get_gmail_service(scout) + msg = service.users().messages().get( + userId="me", id=ref.source_msg_ref, format="metadata", + metadataHeaders=["Subject", "From", "Date"], + ).execute() + headers = {h["name"]: h["value"] for h in msg.get("payload", {}).get("headers", [])} + return ItemMetadata( + subject=headers.get("Subject"), + sender=headers.get("From"), + snippet=msg.get("snippet"), + received_at=None, + ) + return await asyncio.to_thread(_sync) + + async def fetch_content(self, scout, ref: ItemRef) -> ItemContent: + creds_info = decrypt_token(scout.oauth_token_encrypted) + client = GmailClient(creds_info) + # Use existing fetch_messages narrowed by message id via the q parameter. + # Gmail API doesn't have a direct "get by message id with full body" via + # GmailClient's high-level interface, so we shell to the low-level call. + def _sync(): + service = _get_gmail_service(scout) + msg = service.users().messages().get( + userId="me", id=ref.source_msg_ref, format="full", + ).execute() + return msg + + msg = await asyncio.to_thread(_sync) + from app.integrations.gmail import _extract_body, _build_email_message + # _extract_body / _build_email_message exist as helpers in gmail.py. + # If named differently in your source, find with: grep -n "def _" app/integrations/gmail.py + email_msg = _build_email_message(msg) + return ItemContent( + metadata=ItemMetadata( + subject=email_msg.subject, + sender=email_msg.sender, + snippet=msg.get("snippet"), + received_at=email_msg.date, + ), + body_text=email_msg.body_text, + raw_headers={h["name"]: h["value"] for h in msg.get("payload", {}).get("headers", [])}, + ) + + async def archive(self, scout, ref: ItemRef) -> None: + def _sync(): + service = _get_gmail_service(scout) + service.users().messages().trash(userId="me", id=ref.source_msg_ref).execute() + await asyncio.to_thread(_sync) + + async def setup_watch(self, scout) -> None: + def _sync(): + service = _get_gmail_service(scout) + request_body = { + "labelIds": ["INBOX"], + "topicName": settings.GMAIL_PUBSUB_TOPIC, + } + resp = service.users().watch(userId="me", body=request_body).execute() + scout.gmail_history_id = resp.get("historyId") + scout.gmail_watch_expires_at = datetime.fromtimestamp( + int(resp["expiration"]) / 1000, tz=timezone.utc, + ) + await asyncio.to_thread(_sync) + + async def renew_watch(self, scout) -> None: + await self.setup_watch(scout) +``` + +If `_build_email_message` and `_extract_body` are not the actual private helper names in `app/integrations/gmail.py`, find them: +```bash +grep -n "def _" api/app/integrations/gmail.py +``` +and adjust the import. The point is to reuse the existing body-extraction logic rather than duplicate it. + +- [ ] **Step 4: Run tests** + +```bash +cd api +pytest tests/test_scout_connectors_gmail.py -v +``` +Expected: 3 PASS. + +- [ ] **Step 5: Commit** + +```bash +cd api +git add app/scouts/connectors/gmail.py tests/test_scout_connectors_gmail.py +git commit -m "feat(scouts): add GmailConnector" +``` + +--- + +## Task 23: Connector registration at app startup + +**Files:** +- Modify: `api/app/main.py` — register `GmailConnector` during lifespan startup + +- [ ] **Step 1: Read `app/main.py` lifespan** + +```bash +cd api +grep -n "lifespan\|startup\|@asynccontextmanager" app/main.py +``` + +- [ ] **Step 2: Register connector in lifespan startup** + +```python +@asynccontextmanager +async def lifespan(app: FastAPI): + from app.scouts.connectors.gmail import GmailConnector + from app.scouts.connectors.registry import register_connector + register_connector(GmailConnector()) + + # ... existing startup (scheduler etc.) ... + yield + # ... existing shutdown ... +``` + +- [ ] **Step 3: Run tests** + +```bash +cd api +pytest +``` + +- [ ] **Step 4: Commit** + +```bash +cd api +git add app/main.py +git commit -m "feat(scouts): register GmailConnector at startup" +``` + +--- + +## Task 24: Real triage LLM call + Langfuse prompt + +**Files:** +- Modify: `api/app/scouts/engine.py` — replace stub `_triage_llm` +- Manual: create Langfuse prompt `scout-triage-system` with label `production` +- Modify: `api/tests/test_scout_engine.py` — relax/adjust the monkeypatched stub if needed (existing tests already monkeypatch it, so should keep working) + +- [ ] **Step 1: Create the Langfuse prompt** + +Use the Langfuse MCP: +``` +mcp__langfuse__createTextPrompt + name: "scout-triage-system" + prompt: | + You are a triage classifier for an executive-assistant scout that watches a {{source_type}} feed. + The scout's purpose is: "{{scout_purpose}}". + + Given one item, decide whether it is RELEVANT (worth surfacing to the user as a potential + task / event / note / project) or SPAM (advertising, mass marketing, phishing, bulk + notifications with no actionable content). + + Item: + - Subject: {{item_subject}} + - From: {{item_sender}} + - Body (truncated): {{item_body_truncated_2k}} + + Return JSON only, matching this schema: + {"verdict": "relevant" | "spam", "reason": , "confidence": <0..1>} + + Be conservative on "spam" — if a message could plausibly be a personal/work email, mark it relevant. + label: "production" +``` + +- [ ] **Step 2: Implement real `_triage_llm` in `app/scouts/engine.py`** + +Replace the `_triage_llm` stub with: +```python +async def _triage_llm(self, scout: CloudScoutConfig, content: ItemContent) -> TriageVerdict: + from app.core.llm import get_llm + from app.core.langfuse_client import get_prompt_or_fallback, get_langfuse + + fallback = ( + "You are a triage classifier. Given subject/sender/body, return JSON: " + "{verdict: relevant|spam, reason: str, confidence: 0..1}" + ) + template, prompt_obj = get_prompt_or_fallback("scout-triage-system", fallback) + + body_trunc = (content.body_text or "")[:2000] + rendered = ( + template + .replace("{{source_type}}", scout.provider) + .replace("{{scout_purpose}}", scout.prompt_template or "") + .replace("{{item_subject}}", content.metadata.subject or "") + .replace("{{item_sender}}", content.metadata.sender or "") + .replace("{{item_body_truncated_2k}}", body_trunc) + ) + + llm = get_llm(model="gpt-4o-mini") + lf = get_langfuse() + if lf: + with lf.start_as_current_observation(as_type="generation", name="scout-triage", prompt=prompt_obj) as gen: + gen.update(input=rendered) + result = await llm.acomplete(rendered, response_format={"type": "json_object"}) + gen.update(output=result.text) + else: + result = await llm.acomplete(rendered, response_format={"type": "json_object"}) + + import json + data = json.loads(result.text) + return TriageVerdict(**data) +``` + +(Adapt to the project's actual LLM helper signature. If `get_llm`/`acomplete` aren't the real names, find the LiteLLM wrapper: +```bash +grep -n "def \|class " api/app/core/llm.py +``` +and use the actual API.) + +- [ ] **Step 3: Run engine tests** + +```bash +cd api +pytest tests/test_scout_engine.py -v +``` +The existing tests monkeypatch `_triage_llm`, so they keep passing. + +- [ ] **Step 4: Add an integration test that exercises the real LLM path with a mocked LLM client** + +Append to `tests/test_scout_engine.py`: +```python +@pytest.mark.asyncio +async def test_triage_llm_parses_json_response(monkeypatch): + from app.scouts.engine import ScoutEngine + from app.scouts.connectors.base import ItemContent, ItemMetadata + from app.models import CloudScoutConfig + + scout = CloudScoutConfig( + id=str(uuid.uuid4()), user_id="x", provider="gmail", name="t", + data_types=[], prompt_template="watch invoices", schedule_cron="0 * * * *", + enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14, + ) + content = ItemContent(metadata=ItemMetadata(subject="Invoice 42", sender="acme@x"), body_text="due 2026-06-01") + + fake_llm = AsyncMock() + fake_response = AsyncMock() + fake_response.text = '{"verdict": "relevant", "reason": "invoice", "confidence": 0.9}' + fake_llm.acomplete = AsyncMock(return_value=fake_response) + + monkeypatch.setattr("app.scouts.engine.get_llm" if False else "app.core.llm.get_llm", lambda **k: fake_llm) + + engine = ScoutEngine() + verdict = await engine._triage_llm(scout, content) + assert verdict.verdict == "relevant" + assert verdict.reason == "invoice" +``` + +(Adjust the monkeypatch target to match the actual import path used inside `_triage_llm`.) + +- [ ] **Step 5: Run tests** + +```bash +cd api +pytest tests/test_scout_engine.py -v +``` + +- [ ] **Step 6: Commit** + +```bash +cd api +git add app/scouts/engine.py tests/test_scout_engine.py +git commit -m "feat(scouts): real triage LLM call via scout-triage-system prompt" +``` + +--- + +## Task 25: Pub/Sub webhook route + JWT verification + +**Files:** +- Create: `api/app/api/routes/scout_webhooks.py` +- Create: `api/tests/test_scout_webhook.py` +- Modify: `api/app/main.py` — register the new router +- Modify: `api/app/config/settings.py` — add `GMAIL_PUBSUB_TOPIC` and `GMAIL_PUBSUB_AUDIENCE` env vars + +- [ ] **Step 1: Add settings vars** + +In `app/config/settings.py`: +```python +GMAIL_PUBSUB_TOPIC: str = "" # e.g. "projects/adiuvai-prod/topics/gmail-watch" +GMAIL_PUBSUB_AUDIENCE: str = "" # OIDC token audience configured in Pub/Sub push subscription +``` + +- [ ] **Step 2: Write the failing test** + +`api/tests/test_scout_webhook.py`: +```python +"""Tests for the Gmail Pub/Sub webhook.""" + +from __future__ import annotations + +import base64 +import json +import uuid +from unittest.mock import AsyncMock, patch + +import pytest +from httpx import AsyncClient + +from app.main import app +from app.models import CloudScoutConfig, User +from tests.conftest import _TestSessionLocal + + +def _pubsub_payload(email: str, history_id: str) -> dict: + inner = json.dumps({"emailAddress": email, "historyId": history_id}).encode() + return { + "message": {"data": base64.b64encode(inner).decode(), "messageId": "m1"}, + "subscription": "projects/x/subscriptions/gmail-watch-sub", + } + + +@pytest.mark.asyncio +async def test_webhook_triggers_scout_for_matching_user(): + user_id = "00000000-0000-0000-0000-000000000003" + scout_id = str(uuid.uuid4()) + async with _TestSessionLocal() as session: + user = await session.get(User, user_id) + user.email = "alice@example.com" + session.add(CloudScoutConfig( + id=scout_id, user_id=user_id, provider="gmail", name="Inbox", + data_types=[], prompt_template="", schedule_cron="0 * * * *", + enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14, + )) + await session.commit() + + payload = _pubsub_payload("alice@example.com", "200") + + with patch("app.api.routes.scout_webhooks._verify_pubsub_jwt", return_value=True), \ + patch("app.scouts.engine.ScoutEngine.trigger_scout", new=AsyncMock()) as mock_trigger: + async with AsyncClient(app=app, base_url="http://test") as client: + resp = await client.post( + "/api/v1/scouts/webhooks/gmail", + json=payload, + headers={"Authorization": "Bearer fake-google-jwt"}, + ) + assert resp.status_code == 204 + mock_trigger.assert_awaited_once_with(uuid.UUID(scout_id)) + + +@pytest.mark.asyncio +async def test_webhook_rejects_unverified_jwt(): + payload = _pubsub_payload("alice@example.com", "200") + with patch("app.api.routes.scout_webhooks._verify_pubsub_jwt", return_value=False): + async with AsyncClient(app=app, base_url="http://test") as client: + resp = await client.post( + "/api/v1/scouts/webhooks/gmail", + json=payload, + headers={"Authorization": "Bearer bogus"}, + ) + assert resp.status_code == 401 +``` + +- [ ] **Step 3: Implement `app/api/routes/scout_webhooks.py`** + +```python +"""Gmail Pub/Sub push receiver. + +Google Pub/Sub push subscriptions deliver Gmail watch notifications as POST +requests with a JSON envelope. The body payload contains a base64-encoded +JSON blob with ``emailAddress`` + ``historyId``. We resolve the user by +email, look up their cloud_scout_configs row for provider='gmail', and +hand off to ScoutEngine.trigger_scout. + +Authentication: Pub/Sub push includes an OIDC JWT in the Authorization +header. We verify it against Google's public keys with the audience +configured in our Pub/Sub subscription. +""" + +from __future__ import annotations + +import base64 +import json +import logging +import uuid + +from fastapi import APIRouter, Header, HTTPException, Request, status +from sqlalchemy import select + +from app.config.settings import settings +from app.db.session import async_session_factory +from app.models import CloudScoutConfig, User +from app.scouts.engine import ScoutEngine + +logger = logging.getLogger(__name__) +router = APIRouter(prefix="/scouts/webhooks", tags=["scout-webhooks"]) + + +def _verify_pubsub_jwt(token: str) -> bool: + """Verify the Google Pub/Sub OIDC JWT. + + Returns False on any verification failure. Production should use + google.oauth2.id_token.verify_oauth2_token with the configured audience. + """ + if not token: + return False + try: + from google.oauth2 import id_token + from google.auth.transport import requests as g_requests + id_token.verify_oauth2_token(token, g_requests.Request(), audience=settings.GMAIL_PUBSUB_AUDIENCE) + return True + except Exception: + logger.warning("pubsub jwt verification failed", exc_info=True) + return False + + +@router.post("/gmail", status_code=status.HTTP_204_NO_CONTENT) +async def gmail_pubsub( + request: Request, + authorization: str = Header(default=""), +): + token = authorization.removeprefix("Bearer ").strip() + if not _verify_pubsub_jwt(token): + raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Invalid Pub/Sub JWT") + + body = await request.json() + msg = body.get("message") or {} + raw = msg.get("data") + if not raw: + return # ack without action + try: + decoded = json.loads(base64.b64decode(raw).decode()) + except Exception: + logger.warning("pubsub payload decode failed") + return + + email = decoded.get("emailAddress") + if not email: + return + + async with async_session_factory() as session: + user_q = await session.execute(select(User).where(User.email == email)) + user = user_q.scalar_one_or_none() + if user is None: + logger.info("pubsub: no user for %s — ignoring", email) + return + scouts_q = await session.execute( + select(CloudScoutConfig).where( + CloudScoutConfig.user_id == user.id, + CloudScoutConfig.provider == "gmail", + CloudScoutConfig.enabled == True, + ) + ) + scouts = scouts_q.scalars().all() + + engine = ScoutEngine() + for scout in scouts: + await engine.trigger_scout(uuid.UUID(scout.id)) +``` + +- [ ] **Step 4: Register the router in `app/main.py`** + +```python +from app.api.routes import scout_webhooks +app.include_router(scout_webhooks.router, prefix="/api/v1") +``` + +- [ ] **Step 5: Run tests** + +```bash +cd api +pytest tests/test_scout_webhook.py -v +``` +Expected: 2 PASS. + +- [ ] **Step 6: Commit** + +```bash +cd api +git add app/api/routes/scout_webhooks.py tests/test_scout_webhook.py app/main.py app/config/settings.py +git commit -m "feat(scouts): gmail pub/sub webhook with JWT verification" +``` + +--- + +## Task 26: Cron-fallback scheduler + watch renewal in lifespan + +**Files:** +- Modify: `api/app/main.py` — add `_scout_cron_tick` and `_scout_watch_renewal_tick` to existing APScheduler + +- [ ] **Step 1: Read existing scheduler setup** + +```bash +cd api +grep -n "AsyncIOScheduler\|apscheduler\|memory_cron_tick" app/main.py +``` + +- [ ] **Step 2: Add the two new ticks (gated on `settings.SCHEDULER_ENABLED`)** + +Inside `lifespan`: +```python +if settings.SCHEDULER_ENABLED: + scheduler.add_job( + _scout_cron_tick, trigger="interval", minutes=15, + id="scout_cron_tick", replace_existing=True, + ) + scheduler.add_job( + _scout_watch_renewal_tick, trigger="interval", hours=24, + id="scout_watch_renewal_tick", replace_existing=True, + ) +``` + +Add the tick functions to the same file: +```python +async def _scout_cron_tick(): + """Poll cloud scouts whose schedule has elapsed since last_run_at.""" + from app.scouts.engine import ScoutEngine + from app.models import CloudScoutConfig + + async with async_session_factory() as session: + scouts = (await session.execute( + select(CloudScoutConfig).where(CloudScoutConfig.enabled == True) + )).scalars().all() + + engine = ScoutEngine() + for scout in scouts: + # Cheap rate-limit: don't poll if we ran within the last 5 minutes. + # Push notifications are the primary trigger; this is fallback only. + if scout.last_run_at: + elapsed = (datetime.now(tz=timezone.utc) - scout.last_run_at).total_seconds() + if elapsed < 300: + continue + await engine.trigger_scout(uuid.UUID(scout.id)) + + +async def _scout_watch_renewal_tick(): + """Re-issue Gmail users.watch for any scout whose watch expires within 24h.""" + from app.scouts.connectors.registry import get_connector + from app.models import CloudScoutConfig + + threshold = datetime.now(tz=timezone.utc) + timedelta(hours=24) + async with async_session_factory() as session: + scouts = (await session.execute( + select(CloudScoutConfig).where( + CloudScoutConfig.enabled == True, + CloudScoutConfig.provider == "gmail", + CloudScoutConfig.gmail_watch_expires_at <= threshold, + ) + )).scalars().all() + for scout in scouts: + try: + connector = get_connector("gmail") + await connector.renew_watch(scout) + except Exception: + logger.exception("scout %s: watch renewal failed", scout.id) + await session.commit() +``` + +- [ ] **Step 3: Run tests** + +```bash +cd api +pytest +``` + +- [ ] **Step 4: Commit** + +```bash +cd api +git add app/main.py +git commit -m "feat(scouts): add cron-fallback poll + gmail watch renewal ticks" +``` + +--- + +## Task 27: Settings UI — "Add Gmail Scout" OAuth flow + +**Files:** +- Modify: `adiuvAI/src/renderer/components/settings/CloudScoutConfigPanel.tsx` — add "Connect Gmail" button + OAuth flow +- Modify: `adiuvAI/src/main/router/index.ts` — add tRPC procedure `scout.cloud.startGmailOAuth` that opens the browser to `/api/v1/auth/oauth/google/authorize` with scout-specific scopes +- Modify: `adiuvAI/src/main/index.ts` — extend the existing `adiuvai://` protocol handler to detect `?scout_oauth_callback=1` and dispatch to scout flow instead of login flow + +**Context:** The login OAuth flow already exists in `auth.py`. For scout OAuth we want a separate consent screen with `gmail.readonly` + `gmail.modify` scopes. Two approaches: + +A) Reuse the existing `/auth/oauth/google/authorize` route with an extra query param `purpose=scout&scout_id=`. The route branches on purpose to choose scopes and store-back behavior. +B) Add a new pair of routes `/scouts/oauth/gmail/authorize` and `/scouts/oauth/gmail/callback` mirroring the auth pattern. + +Pick **B** for clarity — separates concerns. Implement the new routes alongside the existing ones. + +- [ ] **Step 1: Add scout OAuth routes in `api/app/api/routes/scouts.py`** + +```python +@router.get("/oauth/gmail/authorize") +async def scout_gmail_oauth_authorize( + scout_id: str, + user: User = Depends(get_current_user), +): + from app.auth.oauth_providers import GoogleOAuthProvider, generate_pkce_pair + + code_verifier, code_challenge = generate_pkce_pair() + state = secrets.token_urlsafe(32) + _pending_scout_oauth_states[state] = (code_verifier, scout_id, user.id, time.time() + 600) + + provider = GoogleOAuthProvider() + authorize_url = provider.build_authorize_url( + redirect_uri=f"{settings.OAUTH_REDIRECT_URI_BASE}/api/v1/scouts/oauth/gmail/web-callback", + state=state, + code_challenge=code_challenge, + scopes=["openid", "email", "https://www.googleapis.com/auth/gmail.readonly", "https://www.googleapis.com/auth/gmail.modify"], + access_type="offline", + prompt="consent", + ) + return {"authorize_url": authorize_url} + + +@router.get("/oauth/gmail/web-callback") +async def scout_gmail_oauth_web_callback(code: str, state: str): + """Bounces to the Electron deep link with code+state.""" + return RedirectResponse(f"adiuvai://scout/oauth/gmail/callback?code={code}&state={state}") + + +@router.post("/oauth/gmail/callback") +async def scout_gmail_oauth_callback( + body: _ScoutGmailCallbackBody, + db: AsyncSession = Depends(get_session), + user: User = Depends(get_current_user), +): + entry = _pending_scout_oauth_states.pop(body.state, None) + if entry is None or entry[3] < time.time() or entry[2] != user.id: + raise HTTPException(401, "Invalid or expired OAuth state") + code_verifier, scout_id, _, _ = entry + + provider = GoogleOAuthProvider() + token_data = await provider.exchange_code( + code=body.code, + code_verifier=code_verifier, + redirect_uri=f"{settings.OAUTH_REDIRECT_URI_BASE}/api/v1/scouts/oauth/gmail/web-callback", + scopes=["openid", "email", "https://www.googleapis.com/auth/gmail.readonly", "https://www.googleapis.com/auth/gmail.modify"], + ) + + creds_dict = { + "token": token_data["access_token"], + "refresh_token": token_data.get("refresh_token"), + "token_uri": "https://oauth2.googleapis.com/token", + "client_id": settings.GOOGLE_AUTH_CLIENT_ID, + "client_secret": settings.GOOGLE_AUTH_CLIENT_SECRET, + "scopes": ["https://www.googleapis.com/auth/gmail.readonly", "https://www.googleapis.com/auth/gmail.modify"], + } + encrypted = encrypt_token(creds_dict) + + scout = await db.get(CloudScoutConfig, scout_id) + if scout is None or scout.user_id != user.id: + raise HTTPException(404, "Scout not found") + scout.oauth_token_encrypted = encrypted + await db.commit() + + # Set up the Gmail watch so we start receiving push notifications. + from app.scouts.connectors.registry import get_connector + try: + await get_connector("gmail").setup_watch(scout) + await db.commit() + except Exception: + logger.exception("setup_watch failed for scout %s", scout_id) + + return {"ok": True} +``` + +(`_pending_scout_oauth_states` and `_ScoutGmailCallbackBody` Pydantic model defined at the top of the file alongside the existing `_pending_states`. `encrypt_token` imported from `app.integrations`.) + +- [ ] **Step 2: Add tRPC procedures `scout.cloud.startGmailOAuth` + `scout.cloud.completeGmailOAuth` in `adiuvAI/src/main/router/index.ts`** + +```typescript +startGmailOAuth: publicProcedure + .input(z.object({ scoutId: z.string() })) + .mutation(async ({ input }) => { + const { authorize_url } = await getBackendClient().proxyGet<{ authorize_url: string }>( + `/api/v1/scouts/oauth/gmail/authorize?scout_id=${encodeURIComponent(input.scoutId)}`, + ); + await shell.openExternal(authorize_url); + return { ok: true }; + }), + +completeGmailOAuth: publicProcedure + .input(z.object({ code: z.string(), state: z.string() })) + .mutation(async ({ input }) => { + return await getBackendClient().proxyPost('/api/v1/scouts/oauth/gmail/callback', input); + }), +``` + +- [ ] **Step 3: Extend the `adiuvai://` protocol handler in `adiuvAI/src/main/index.ts`** + +Find the existing `second-instance` / `open-url` handler that parses `adiuvai://oauth/callback?...`. Add a branch for `adiuvai://scout/oauth/gmail/callback?code=...&state=...`: +```typescript +if (parsed.host === 'scout' && parsed.pathname === '/oauth/gmail/callback') { + const code = parsed.searchParams.get('code'); + const state = parsed.searchParams.get('state'); + if (code && state) { + // Forward to renderer via IPC so the scout settings page can call the tRPC mutation. + BrowserWindow.getAllWindows()[0]?.webContents.send('scout:gmailOAuthCallback', { code, state }); + } + return; +} +``` + +- [ ] **Step 4: Add the consumer in `CloudScoutConfigPanel.tsx`** + +```typescript +const startGmailOAuth = trpc.scout.cloud.startGmailOAuth.useMutation(); +const completeGmailOAuth = trpc.scout.cloud.completeGmailOAuth.useMutation(); + +useEffect(() => { + const off = window.electronAI.on('scout:gmailOAuthCallback', async ({ code, state }) => { + await completeGmailOAuth.mutateAsync({ code, state }); + notify('success', 'toast.scout.gmailConnected'); + void utils.scout.cloud.list.invalidate(); + }); + return () => off?.(); +}, []); + +// In render, when scout.provider === 'gmail' && !scout.oauthTokenEncrypted: + +``` + +(Adjust to actual `electronAI` API shape — check `adiuvAI/src/preload/index.ts` for the exact event subscription pattern.) + +- [ ] **Step 5: Add i18n keys** + +In all 5 `translation.json` files: +```json +"scouts": { + "connectGmail": "Connect Gmail", + ... +}, +"toast": { + "scout": { + "gmailConnected": "Gmail connected. Watching for new messages.", + ... + } +} +``` + +(Translate per language.) + +- [ ] **Step 6: Typecheck + lint + manual test** + +```bash +cd adiuvAI +npx tsc --noEmit +npm run lint +``` + +Then manual: +```bash +npm run start +``` +Open Settings > Scouts > "Add Cloud Scout" → choose Gmail → click "Connect Gmail" → consent screen opens in browser → choose account, grant scopes → browser redirects to `adiuvai://...` → app receives callback → toast confirms. + +- [ ] **Step 7: Commit (split across submodules)** + +```bash +cd api +git add app/api/routes/scouts.py app/config/settings.py +git commit -m "feat(scouts): add Gmail OAuth scout-setup routes" + +cd ../adiuvAI +git add src/main/router/index.ts src/main/index.ts \ + src/renderer/components/settings/CloudScoutConfigPanel.tsx \ + src/renderer/locales/ +git commit -m "feat(scouts): Gmail OAuth UI flow" +``` + +--- + +## Task 28: Submodule pointer bump for Phase 3 + +- [ ] **Step 1: Bump submodule pointers** + +```bash +cd c:\Users\PC-Roby\Documents\_adiuvai_workspace +git add api adiuvAI +git commit -m "chore: bump submodules — Phase 3 Gmail scout end-to-end" +``` + +**Phase 3 complete.** End-to-end Gmail scout: OAuth → watch → push → triage → queue → on reconnect → SQLite `scout_suggestions` rows with `category='unprocessed'`. + +--- + +## Final Acceptance Walkthrough (manual, after all tasks) + +- [ ] Boot app, log in. +- [ ] Settings > Scouts: header reads "Scouts" (or localized equivalent). +- [ ] Existing local directory monitor still listed and operational. +- [ ] "Add scout" → choose Gmail → connect → consent → return to app, scout shows as enabled. +- [ ] Send yourself a test email. +- [ ] Wait ~30 seconds (Pub/Sub push) or trigger via the cron tick. +- [ ] Run from `adiuvAI/`: `sqlite3 dev.db "select id, scout_id, raw_subject, status, category from scout_suggestions"`. +- [ ] Expected: one row with `status='pending'`, `category='unprocessed'`, `raw_subject` matching the email subject. +- [ ] Run on the API DB: `psql -d adiuvai -c "select id, scout_id, status, triage_verdict from scout_triage_queue"`. +- [ ] Expected: same row, `status='acked'` (Electron acked on receipt). +- [ ] No content (body) anywhere in `scout_triage_queue` — verify with `\d scout_triage_queue`. +- [ ] Send a spam-looking email (e.g. all-caps subject, "$$$ buy now $$$" body). With `auto_trash_spam=false` (default), no row appears in either DB. Toggle `auto_trash_spam=true` on the scout, send another spam email — verify it lands in Gmail Trash and no rows appear. + +--- + +## Plan Self-Review + +**Spec coverage:** +- Phase 1 rename — Tasks 1-11. All renamed surfaces from spec table covered. +- Phase 2 connector skeleton — Tasks 12-21. Covers `scout_triage_queue` table, `cloud_scout_configs` alters, `SourceConnector` Protocol, registry, `ScoutEngine.trigger_scout` + `_process_item` + `deliver_pending` + `ack_proposal`, Drizzle `scout_suggestions` table, WS frame schemas, Electron handler. +- Phase 3 Gmail end-to-end — Tasks 22-28. `GmailConnector` impl, registration, real triage LLM, Pub/Sub webhook, cron-fallback + watch renewal scheduler, OAuth setup UI. +- Phase 4 explicitly out of scope per spec. + +**Type/method consistency:** +- `ScoutTriageQueue` columns match the Alembic migration in Task 12 and the SQLAlchemy model in Task 13. +- `scoutSuggestions` Drizzle columns in Task 17 match the `IncomingScoutProposal` interface in Task 20 (snake_case in DB, camelCase in TS object). +- `_triage_llm` is a stub in Task 16 and replaced in Task 24 — explicit. +- `ScoutEngine.deliver_pending` signature `(user_id: UUID, ws)` is the same in Task 19 (definition) and Task 19 (WS endpoint usage). +- `ScoutProposalPayload` (Task 18) matches the dict the engine sends in Task 19. + +**Placeholder scan:** no TBDs or "implement appropriate error handling" stubs. Where the underlying API name (e.g. `_extract_body`, `get_llm`) might differ from this codebase's actual symbols, the task gives a `grep` command to discover the real name and adapt — that's a task-time investigation, not a placeholder. + +**Scope check:** Three phases, each with a clear cut-point and submodule bump. Phase 1 alone is shippable and produces working software (rename, no behavior change). Phase 2 alone is internal-only infra. Phase 3 lands the user-visible feature. All three together fit one implementation track per the approved spec. + +--- + +## Execution Handoff + +**Plan complete and saved to** `docs/superpowers/specs/2026-05-15-scouts-refactor-and-gmail-integration-plan.md`. **Two execution options:** + +**1. Subagent-Driven (recommended)** — I dispatch a fresh subagent per task, review between tasks, fast iteration. + +**2. Inline Execution** — Execute tasks in this session using executing-plans, batch execution with checkpoints. + +**Which approach?**