adiuvAI/workspace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
445a4cbbf94fae3a2d28abebfc0cb37a5c717f2a
workspace/docs/superpowers/specs/2026-05-15-scouts-refactor-and-gmail-integration-plan.md
Roberto 445a4cbbf9 docs: add scouts refactor + gmail scout implementation plan 
Covers Phases 1-3 (rename, connector skeleton, Gmail end-to-end) as
28 TDD tasks. Phase 4 (Stage 2 categorization + brief HITL) deferred
to separate spec.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-15 23:32:40 +02:00

3081 lines
108 KiB
Markdown
Raw Blame History

# Scouts Refactor + Gmail Integration Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Rename the entire "Agents" subsystem to "Scouts" across UI/code/Postgres/SQLite/Langfuse, then add a Gmail cloud scout with two-stage pipeline (BE triage + on-connect delivery) and a reusable `SourceConnector` abstraction. Phase 4 (Stage 2 categorization + HITL surface) is deferred to a future spec.
**Architecture:** Phase 1 is a single atomic rename PR (no behavior change). Phase 2 lays connector skeleton, queue table, WS frame contract, and SQLite suggestion table — no user-visible feature yet. Phase 3 ships Gmail end-to-end: OAuth setup UI, push (`users.watch`) + cron-fallback polling, BE-side spam triage with opt-in auto-trash, deliver-on-reconnect drain into local `scoutSuggestions` table with `category='unprocessed'` stub.
**Tech Stack:** Python 3.12 / FastAPI / SQLAlchemy 2.0 (async) / Alembic / Pydantic v2 / Langfuse for prompts / Google API Client for Gmail / Pub/Sub for push notifications. Electron / TypeScript / React 19 / tRPC v11 / Drizzle ORM / better-sqlite3 / i18next.
**Spec:** [docs/superpowers/specs/2026-05-15-scouts-refactor-and-gmail-integration-design.md](2026-05-15-scouts-refactor-and-gmail-integration-design.md)
**Repository layout:** monorepo with submodules `api/` (FastAPI) and `adiuvAI/` (Electron). All tasks operate on absolute paths under `c:\Users\PC-Roby\Documents\_adiuvai_workspace`.
---
## File Structure
### New files (BE — `api/`)
| Path | Responsibility |
|------|----------------|
| `api/alembic/versions/007_rename_agents_to_scouts.py` | Rename Postgres tables/columns from `*_agent_*` to `*_scout_*`. |
| `api/alembic/versions/008_scout_triage_queue.py` | Create `scout_triage_queue` + alter `cloud_scout_configs`. |
| `api/app/scouts/__init__.py` | Package marker. |
| `api/app/scouts/engine.py` | `ScoutEngine` orchestrator (triage, queue, deliver). |
| `api/app/scouts/connectors/__init__.py` | Package marker. |
| `api/app/scouts/connectors/base.py` | `SourceConnector` Protocol + `ItemRef`/`ItemMetadata`/`ItemContent`/`TriageVerdict` Pydantic models. |
| `api/app/scouts/connectors/registry.py` | Connector registration + lookup. |
| `api/app/scouts/connectors/gmail.py` | `GmailConnector` impl (wraps existing `app/integrations/gmail.py`). |
| `api/app/api/routes/scout_webhooks.py` | `POST /api/v1/scouts/webhooks/gmail` — Pub/Sub receiver. |
| `api/tests/test_scout_engine.py` | Unit tests for `ScoutEngine` w/ mocked connector. |
| `api/tests/test_scout_connectors_gmail.py` | Tests for `GmailConnector` (mocked Gmail API). |
| `api/tests/test_scout_webhook.py` | Webhook + Pub/Sub JWT verification. |
### Renamed files (BE)
| Before | After |
|--------|-------|
| `api/app/api/routes/agents.py` | `api/app/api/routes/scouts.py` |
| `api/app/api/routes/agent_setup.py` | `api/app/api/routes/scout_setup.py` |
| `api/app/core/agent_runner.py` | `api/app/core/scout_runner.py` |
| `api/app/core/agent_session_buffer.py` | `api/app/core/scout_session_buffer.py` |
| `api/app/core/agent_registry.py` | `api/app/core/scout_registry.py` |
### New files (Electron — `adiuvAI/`)
| Path | Responsibility |
|------|----------------|
| `adiuvAI/src/main/db/migrations/0007_scouts_rename.sql` | Drizzle SQL: rename `agent_runs` → `scout_runs`, `agent_run_actions` → `scout_run_actions`. |
| `adiuvAI/src/main/db/migrations/0008_scout_suggestions.sql` | Drizzle SQL: create `scout_suggestions` table. |
| `adiuvAI/src/main/scouts/scout-suggestion-handler.ts` | Handles incoming `scout_proposal` WS frame, inserts into `scoutSuggestions`, sends `scout_proposal_ack`. |
### Renamed files (Electron)
| Before | After |
|--------|-------|
| `adiuvAI/src/main/agents/agent-scheduler.ts` | `adiuvAI/src/main/scouts/scout-scheduler.ts` |
| `adiuvAI/src/renderer/components/settings/AgentsSection.tsx` | `adiuvAI/src/renderer/components/settings/ScoutsSection.tsx` |
| `adiuvAI/src/renderer/components/settings/AgentRow.tsx` | `adiuvAI/src/renderer/components/settings/ScoutRow.tsx` |
| `adiuvAI/src/renderer/components/settings/LocalAgentConfigPanel.tsx` | `adiuvAI/src/renderer/components/settings/LocalScoutConfigPanel.tsx` |
| `adiuvAI/src/renderer/components/settings/CloudAgentConfigPanel.tsx` | `adiuvAI/src/renderer/components/settings/CloudScoutConfigPanel.tsx` |
| `adiuvAI/src/renderer/components/settings/InlineAgentCreationStepper.tsx` | `adiuvAI/src/renderer/components/settings/InlineScoutCreationStepper.tsx` |
### Modified files (high-impact, full list per task)
- `api/app/models.py` — class renames + table/column renames + relationship updates.
- `api/app/main.py` — register new router, lifespan crons.
- `api/app/schemas.py` — `WsFrameType` enum + new frame Pydantic models.
- `api/app/config/settings.py` — add Gmail Pub/Sub topic env vars.
- `adiuvAI/src/main/db/schema.ts` — rename tables + add `scoutSuggestions`.
- `adiuvAI/src/main/router/index.ts` — rename `agent.*` sub-routers → `scout.*`.
- `adiuvAI/src/main/api/backend-client.ts` — handle new WS frames.
- `adiuvAI/src/renderer/locales/{en,it,es,fr,de}/translation.json` — rename i18n keys.
- `adiuvAI/src/renderer/routes/settings.tsx` — update SECTIONS array.
---
# PHASE 1 — Rename Only
Goal: pure rename, no behavior change, all tests green at end of phase.
**Important caveat:** the project uses git submodules. Each task touches files in `api/` or `adiuvAI/` submodule — commits happen inside the submodule, then a separate commit in the monorepo updates the submodule pointer. Each task's commit step shows both.
---
## Task 1: Postgres rename — Alembic migration
**Files:**
- Create: `api/alembic/versions/007_rename_agents_to_scouts.py`
**Context:** Latest existing migration is `d6e3f4a5b6c7_folder_index_tables.py` with `down_revision = "006"`. New migration uses revision `"007"` and depends on `d6e3f4a5b6c7`. Tables to rename: `local_agent_configs`, `cloud_agent_configs`, `agent_run_logs`. Columns to rename: `agent_config` (in `local_agent_configs`), `agent_id` and `agent_type` (in `agent_run_logs`).
- [ ] **Step 1: Create the migration file**
```python
"""Rename agents to scouts.
Revision ID: 007
Revises: d6e3f4a5b6c7
Create Date: 2026-05-15
Renames the entire agents subsystem identifiers to scouts.
Pre-1.0 — no data preservation concerns beyond ALTER TABLE rename.
"""
from typing import Sequence, Union
from alembic import op
revision: str = "007"
down_revision: Union[str, None] = "d6e3f4a5b6c7"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Tables
op.rename_table("local_agent_configs", "local_scout_configs")
op.rename_table("cloud_agent_configs", "cloud_scout_configs")
op.rename_table("agent_run_logs", "scout_run_logs")
# Columns
op.alter_column("local_scout_configs", "agent_config", new_column_name="scout_config")
op.alter_column("scout_run_logs", "agent_id", new_column_name="scout_id")
op.alter_column("scout_run_logs", "agent_type", new_column_name="scout_type")
def downgrade() -> None:
op.alter_column("scout_run_logs", "scout_type", new_column_name="agent_type")
op.alter_column("scout_run_logs", "scout_id", new_column_name="agent_id")
op.alter_column("local_scout_configs", "scout_config", new_column_name="agent_config")
op.rename_table("scout_run_logs", "agent_run_logs")
op.rename_table("cloud_scout_configs", "cloud_agent_configs")
op.rename_table("local_scout_configs", "local_agent_configs")
```
- [ ] **Step 2: Run migration against a fresh test DB to verify it applies cleanly**
Run from `api/` directory:
```bash
alembic upgrade head
```
Expected: no errors. `\d local_scout_configs` shows the renamed table. `\d scout_run_logs` shows the `scout_id` column (not `agent_id`).
- [ ] **Step 3: Verify downgrade works**
```bash
alembic downgrade -1
```
Expected: tables go back to `local_agent_configs` etc. Then re-run `alembic upgrade head`.
- [ ] **Step 4: Commit (inside submodule)**
```bash
cd api
git add alembic/versions/007_rename_agents_to_scouts.py
git commit -m "feat(db): rename agents to scouts (alembic 007)"
```
---
## Task 2: SQLAlchemy model rename — `LocalScoutConfig`, `CloudScoutConfig`, `ScoutRunLog`
**Files:**
- Modify: `api/app/models.py:161-227` (and the `AgentRunLog` class — find via grep)
**Context:** Renames Python class names to match the new tables. Must update `__tablename__` and `mapped_column` for renamed columns. Relationships and `back_populates` strings must reference the new class names.
- [ ] **Step 1: Read current models around `LocalAgentConfig`, `CloudAgentConfig`, `AgentRunLog`**
Use Read tool on `api/app/models.py` to confirm the exact line ranges (the `AgentRunLog` class + relationships use string-based `primaryjoin` + `back_populates="local_agent"` / `back_populates="cloud_agent"` patterns).
- [ ] **Step 2: Apply the renames in `models.py`**
Replace class names and tablenames:
```python
class LocalScoutConfig(Base):
__tablename__ = "local_scout_configs"
# ... id/user_id/etc unchanged ...
scout_config: Mapped[dict | None] = mapped_column(JSON, nullable=True)
# ... rest unchanged ...
run_logs: Mapped[list["ScoutRunLog"]] = relationship(
back_populates="local_scout",
primaryjoin="and_(ScoutRunLog.scout_id == LocalScoutConfig.id, ScoutRunLog.scout_type == 'local')",
foreign_keys="ScoutRunLog.scout_id",
cascade="all, delete-orphan",
overlaps="run_logs,cloud_scout",
)
class CloudScoutConfig(Base):
__tablename__ = "cloud_scout_configs"
# ... unchanged columns ...
run_logs: Mapped[list["ScoutRunLog"]] = relationship(
back_populates="cloud_scout",
primaryjoin="and_(ScoutRunLog.scout_id == CloudScoutConfig.id, ScoutRunLog.scout_type == 'cloud')",
foreign_keys="ScoutRunLog.scout_id",
cascade="all, delete-orphan",
overlaps="run_logs,local_scout",
)
class ScoutRunLog(Base):
__tablename__ = "scout_run_logs"
# ... id, user_id, started_at, finished_at, status, error etc. unchanged ...
scout_id: Mapped[str] = mapped_column(...)
scout_type: Mapped[str] = mapped_column(...)
local_scout: Mapped["LocalScoutConfig | None"] = relationship(
back_populates="run_logs",
primaryjoin="and_(ScoutRunLog.scout_id == LocalScoutConfig.id, ScoutRunLog.scout_type == 'local')",
foreign_keys="ScoutRunLog.scout_id",
overlaps="run_logs,cloud_scout",
)
cloud_scout: Mapped["CloudScoutConfig | None"] = relationship(
back_populates="run_logs",
primaryjoin="and_(ScoutRunLog.scout_id == CloudScoutConfig.id, ScoutRunLog.scout_type == 'cloud')",
foreign_keys="ScoutRunLog.scout_id",
overlaps="run_logs,local_scout",
)
```
- [ ] **Step 3: Sweep all imports of the old class names**
Run from `api/`:
```bash
grep -rn "AgentRunLog\|LocalAgentConfig\|CloudAgentConfig" app/ tests/ --include="*.py"
```
For each hit, replace with the corresponding `Scout*` name. Do not skip test files.
- [ ] **Step 4: Run the test suite — expect failures and fix incrementally**
Run from `api/`:
```bash
pytest -x
```
Fix import errors and reference errors as they surface. The test fixture at `api/tests/conftest.py` does `Base.metadata.create_all` so renamed tables will be created automatically. Don't worry about migrations in tests — they use `create_all`.
- [ ] **Step 5: Verify the suite is green**
```bash
pytest
```
Expected: all tests pass.
- [ ] **Step 6: Commit (inside submodule)**
```bash
cd api
git add app/models.py
git add -u app/ tests/ # any files changed by the import sweep
git commit -m "refactor(models): rename Agent classes to Scout"
```
---
## Task 3: Rename BE routes — `routes/agents.py` → `routes/scouts.py`, `agent_setup.py` → `scout_setup.py`
**Files:**
- Rename: `api/app/api/routes/agents.py` → `api/app/api/routes/scouts.py`
- Rename: `api/app/api/routes/agent_setup.py` → `api/app/api/routes/scout_setup.py`
- Modify: `api/app/main.py` — update router registration
- Modify: `api/app/api/routes/device_ws.py:30-31` — update imports
- [ ] **Step 1: Rename the files via git**
```bash
cd api
git mv app/api/routes/agents.py app/api/routes/scouts.py
git mv app/api/routes/agent_setup.py app/api/routes/scout_setup.py
```
- [ ] **Step 2: Inside both renamed files, change `APIRouter(prefix="/agents", ...)` to `prefix="/scouts"` and `prefix="/agent-setup"` to `prefix="/scout-setup"`**
In `app/api/routes/scouts.py` find and replace:
```python
router = APIRouter(prefix="/agents", tags=["agents"])
```
With:
```python
router = APIRouter(prefix="/scouts", tags=["scouts"])
```
In `app/api/routes/scout_setup.py` find and replace:
```python
router = APIRouter(prefix="/agent-setup", tags=["agent-setup"])
```
With:
```python
router = APIRouter(prefix="/scout-setup", tags=["scout-setup"])
```
- [ ] **Step 3: Update `app/main.py` to import the new module names**
Find the imports of `agents` and `agent_setup` routers and replace with `scouts` and `scout_setup`.
- [ ] **Step 4: Update `app/api/routes/device_ws.py:30-31` import**
Change:
```python
from app.api.routes.agent_setup import handle_journey_message, handle_journey_start
```
To:
```python
from app.api.routes.scout_setup import handle_journey_message, handle_journey_start
```
- [ ] **Step 5: Sweep remaining import references**
```bash
grep -rn "from app.api.routes.agents\|from app.api.routes.agent_setup\|app\.api\.routes\.agents\|app\.api\.routes\.agent_setup" app/ tests/
```
Replace each.
- [ ] **Step 6: Run tests**
```bash
pytest
```
Expected: pass. URL-based tests must use new paths (`/api/v1/scouts/...`, `/api/v1/scout-setup/...`).
- [ ] **Step 7: Commit (inside submodule)**
```bash
cd api
git add -A app/ tests/
git commit -m "refactor(routes): rename /agents and /agent-setup to /scouts and /scout-setup"
```
---
## Task 4: Rename `core/agent_runner.py`, `agent_session_buffer.py`, `agent_registry.py`
**Files:**
- Rename: `api/app/core/agent_runner.py` → `api/app/core/scout_runner.py`
- Rename: `api/app/core/agent_session_buffer.py` → `api/app/core/scout_session_buffer.py`
- Rename: `api/app/core/agent_registry.py` → `api/app/core/scout_registry.py`
- [ ] **Step 1: Rename via git**
```bash
cd api
git mv app/core/agent_runner.py app/core/scout_runner.py
git mv app/core/agent_session_buffer.py app/core/scout_session_buffer.py
git mv app/core/agent_registry.py app/core/scout_registry.py
```
- [ ] **Step 2: Update top-of-file references in each renamed file**
Inside each file, rename all references to `agent_*` symbols that are now defined here. For example, if `agent_runner.py` had `def trigger_agent_run(...)`, change to `def trigger_scout_run(...)`. Do NOT rename names that belong to `app/agents/*` (those LLM helpers stay).
- [ ] **Step 3: Sweep external imports**
```bash
grep -rn "from app.core.agent_runner\|from app.core.agent_session_buffer\|from app.core.agent_registry" app/ tests/
```
Replace each `agent_runner` → `scout_runner`, etc.
Also sweep symbol references — for example if you renamed `trigger_agent_run` → `trigger_scout_run`:
```bash
grep -rn "trigger_agent_run\|agent_session_buffer\|agent_registry" app/ tests/
```
Replace.
- [ ] **Step 4: Run tests**
```bash
pytest
```
Expected: green.
- [ ] **Step 5: Commit (inside submodule)**
```bash
cd api
git add -A app/ tests/
git commit -m "refactor(core): rename agent_runner/session_buffer/registry to scout_*"
```
---
## Task 5: Rename Pydantic schemas + WS frame names referring to "agent"
**Files:**
- Modify: `api/app/schemas.py` (or wherever `WsFrameType` enum + Agent-related schemas live)
**Context:** Pydantic schemas like `AgentRunStatus`, `AgentTriggerRequest`, etc. need rename. The `WsFrameType` enum may have values like `"agent_run_started"` — rename to `"scout_run_started"`. Frontend will be updated to match in later tasks.
- [ ] **Step 1: Find all Pydantic models + enum values containing "agent"**
```bash
cd api
grep -n "class .*Agent\|\"agent_" app/schemas.py
```
- [ ] **Step 2: Rename each — class names and enum values**
Examples (apply each found in step 1):
- `class AgentRunStatus(str, Enum)` → `class ScoutRunStatus(str, Enum)`
- `class AgentTriggerRequest(BaseModel)` → `class ScoutTriggerRequest(BaseModel)`
- `WsFrameType.AGENT_RUN_STARTED = "agent_run_started"` → `SCOUT_RUN_STARTED = "scout_run_started"`
- [ ] **Step 3: Sweep all references**
```bash
grep -rn "AgentRunStatus\|AgentTriggerRequest\|agent_run_started\|agent_run_completed" app/ tests/
```
Replace.
- [ ] **Step 4: Run tests**
```bash
pytest
```
- [ ] **Step 5: Commit**
```bash
cd api
git add -A
git commit -m "refactor(schemas): rename Agent* schemas and WS frame types to Scout*"
```
---
## Task 6: Rename Langfuse prompts (user-facing scout prompts only)
**Files:**
- Manual operation against Langfuse via MCP tools (no source files unless prompt names are referenced in code).
**Context:** Langfuse prompts whose names start with `agent-` and refer to the user-facing scout subsystem (e.g. `agent-runner-system`, `agent-setup-journey-system`, `cloud-agent-trigger`, `local-agent-extraction`). Prompts for `app/agents/*` LLM helpers (e.g. `task-agent-system`, `note-agent-summarize`) STAY.
- [ ] **Step 1: List all Langfuse prompts**
Use the Langfuse MCP tool:
```
mcp__langfuse__listPrompts (no arguments)
```
- [ ] **Step 2: Identify scout-subsystem prompts to rename**
From the listing, pick prompts whose names begin with `agent-` AND whose content/usage maps to the user-facing scout subsystem. If unsure for a given prompt, grep its name in `api/app/`:
```bash
grep -rn "get_prompt_or_fallback(\"agent-" api/app/ api/tests/
```
Names that appear in the renamed `scout_*.py` files are scout-subsystem prompts.
- [ ] **Step 3: For each scout prompt, create a new prompt with the renamed name**
Use `mcp__langfuse__getPromptUnresolved` to fetch the current prompt body, then `mcp__langfuse__createTextPrompt` (or `createChatPrompt` if it's a chat prompt) with the new name (e.g. `scout-runner-system`) and label `production`.
- [ ] **Step 4: Update code references to use the new prompt names**
Find each `get_prompt_or_fallback("agent-..."` call and rename the literal:
```bash
grep -rn 'get_prompt_or_fallback("agent-' api/app/
```
Replace each `"agent-X"` with `"scout-X"`.
- [ ] **Step 5: Run tests + manual smoke**
```bash
cd api
pytest
```
Then start dev server (`uvicorn app.main:app --reload`), trigger a known scout-subsystem agent invocation (use the agent setup journey or a scout trigger endpoint via curl) and confirm the prompt loads without falling back.
- [ ] **Step 6: Delete old prompts from Langfuse**
For each old `agent-*` prompt confirmed superseded, delete via Langfuse UI or API (Langfuse MCP doesn't expose delete; use the Langfuse web console).
- [ ] **Step 7: Commit code changes**
```bash
cd api
git add -A
git commit -m "refactor(prompts): rename scout-subsystem Langfuse prompt references to scout-*"
```
---
## Task 7: Drizzle migration — rename `agent_runs` → `scout_runs`, `agent_run_actions` → `scout_run_actions`
**Files:**
- Create: `adiuvAI/src/main/db/migrations/0007_scouts_rename.sql`
- Modify: `adiuvAI/src/main/db/schema.ts`
**Context:** Drizzle migrations are SQL files in `adiuvAI/src/main/db/migrations/`. Latest is `0006_misty_cammi.sql`. The auto-generated migration name suffix isn't strictly required but follow the pattern (drizzle-kit generates one from CHANGELOG entries). For a manual rename, a hand-written name is fine.
- [ ] **Step 1: Update `adiuvAI/src/main/db/schema.ts` first — change the table name in the Drizzle definition**
Find:
```typescript
export const agentRuns = sqliteTable('agent_runs', { ... });
export const agentRunActions = sqliteTable('agent_run_actions', { ... });
```
Replace with:
```typescript
export const scoutRuns = sqliteTable('scout_runs', {
id: text().primaryKey(),
scoutId: text('scout_id').notNull(),
status: text().notNull(),
startedAt: integer('started_at').notNull(),
completedAt: integer('completed_at'),
});
export const scoutRunActions = sqliteTable('scout_run_actions', {
id: text().primaryKey(),
runId: text('run_id').notNull(),
scoutId: text('scout_id').notNull(),
verb: text().notNull(),
entityType: text('entity_type').notNull(),
entityId: text('entity_id').notNull(),
entityTitle: text('entity_title'),
createdAt: integer('created_at').notNull(),
});
```
(Read the file first to copy the exact column definitions — keep types/nullability identical, only rename `agent` → `scout`.)
- [ ] **Step 2: Generate the migration**
Run from `adiuvAI/`:
```bash
npx drizzle-kit generate
```
Expected: a new file `0007_<adjective>_<noun>.sql` in `src/main/db/migrations/`. Verify it contains `ALTER TABLE` statements (or DROP+CREATE if Drizzle prefers — SQLite may require recreating the table; Drizzle will emit the right pattern).
If Drizzle emits a destructive recreate (DROP TABLE + CREATE), that's fine pre-1.0 since these tables are largely empty in dev DBs. Confirm with the user before proceeding if their dev DB has rows worth preserving — for empty/dev DBs, drop is fine.
- [ ] **Step 3: Rename the generated migration to a clean name**
```bash
mv adiuvAI/src/main/db/migrations/0007_*.sql adiuvAI/src/main/db/migrations/0007_scouts_rename.sql
```
- [ ] **Step 4: Sweep TypeScript imports of the old names**
```bash
grep -rn "agentRuns\|agentRunActions" adiuvAI/src/
```
Replace each with `scoutRuns` / `scoutRunActions`.
- [ ] **Step 5: Run typecheck + lint**
From `adiuvAI/`:
```bash
npm run lint
npx tsc --noEmit
```
Expected: pass.
- [ ] **Step 6: Manual smoke — boot the app once to apply the migration**
```bash
cd adiuvAI
npm run start
```
Watch the console for migration apply messages. Once app boots cleanly, close it. Verify the dev DB has `scout_runs` and `scout_run_actions` (use `sqlite3 dev.db ".tables"` from `adiuvAI/`).
- [ ] **Step 7: Commit (inside submodule)**
```bash
cd adiuvAI
git add src/main/db/schema.ts src/main/db/migrations/0007_scouts_rename.sql
git add -u src/ # any consumer files updated in step 4
git commit -m "refactor(db): rename agent_runs/agent_run_actions to scout_*"
```
---
## Task 8: Rename Electron main-process modules and tRPC sub-routers
**Files:**
- Rename: `adiuvAI/src/main/agents/agent-scheduler.ts` → `adiuvAI/src/main/scouts/scout-scheduler.ts`
- Modify: `adiuvAI/src/main/router/index.ts` — rename `agent.*` sub-router to `scout.*`
- Modify: `adiuvAI/src/main/store.ts` — rename `getLocalAgents`/`saveLocalAgent` → `getLocalScouts`/`saveLocalScout`
- Modify: `adiuvAI/src/main/index.ts` — update scheduler import + start call
- Modify: `adiuvAI/src/main/api/backend-client.ts` — rename agent-related symbols if any
**Context:** Inside `agent-scheduler.ts`, the function names are `startAgentScheduler`, `stopAgentScheduler`, `tickAgentScheduler` and POSTs to `/api/v1/agents/trigger`. After Task 3, the BE route is `/api/v1/scouts/trigger` — so this file must also update its URL.
- [ ] **Step 1: Move the file**
```bash
cd adiuvAI
mkdir -p src/main/scouts
git mv src/main/agents/agent-scheduler.ts src/main/scouts/scout-scheduler.ts
rmdir src/main/agents 2>/dev/null || true
```
(Don't fail if other files exist in `src/main/agents/` — check first; if so, decide per-file whether to move or leave.)
- [ ] **Step 2: Inside `scout-scheduler.ts`, rename functions and update URL**
Replace:
- `startAgentScheduler` → `startScoutScheduler`
- `stopAgentScheduler` → `stopScoutScheduler`
- `tickAgentScheduler` → `tickScoutScheduler`
- `getLocalAgents()` → `getLocalScouts()`
- `saveLocalAgent(...)` → `saveLocalScout(...)`
- URL `'/api/v1/agents/trigger'` → `'/api/v1/scouts/trigger'`
- Variable `agents` → `scouts` (clarity)
- `agentRuns` Drizzle import → `scoutRuns`
Show the full updated function header:
```typescript
async function tickScoutScheduler(): Promise<void> {
const scouts = getLocalScouts();
const now = Date.now();
for (const scout of scouts) {
if (!scout.enabled) continue;
const intervalMs = CRON_INTERVAL_MS[scout.scheduleCron];
if (!intervalMs) continue;
if (scout.lastRunAt && now - scout.lastRunAt < intervalMs) continue;
try {
const response = await getBackendClient().proxyPost<{ id: string }>(
'/api/v1/scouts/trigger',
{ directory: scout.directory, deviceId: getDeviceId(), /* ... */ }
);
if (response?.id) {
await getDb().insert(scoutRuns).values({
id: response.id, scoutId: scout.id, status: 'running', startedAt: now,
}).onConflictDoNothing();
}
saveLocalScout({ ...scout, lastRunAt: now });
} catch (err) { /* ... */ }
}
}
```
- [ ] **Step 3: Rename store helpers in `adiuvAI/src/main/store.ts`**
Find `getLocalAgents` / `saveLocalAgent` / any `LocalAgentConfig` references and rename to `getLocalScouts` / `saveLocalScout` / `LocalScoutConfig`. Keep the storage key migration-safe: read from old key `localAgents` once, write to `localScouts`, then read only from new key thereafter. For pre-1.0 dev, simpler to just rename the key and accept users re-adding their scouts (note: confirm with user).
For now, simplest acceptable approach:
```typescript
// store.ts
const STORAGE_KEY = 'localScouts'; // was 'localAgents'
export function getLocalScouts(): LocalScoutConfig[] {
return store.get(STORAGE_KEY, []) as LocalScoutConfig[];
}
export function saveLocalScout(scout: LocalScoutConfig): void { /* ... */ }
```
- [ ] **Step 4: Update `adiuvAI/src/main/index.ts`**
Replace:
```typescript
import { startAgentScheduler, stopAgentScheduler } from './agents/agent-scheduler';
```
With:
```typescript
import { startScoutScheduler, stopScoutScheduler } from './scouts/scout-scheduler';
```
And the call sites: `startAgentScheduler()` → `startScoutScheduler()`, `stopAgentScheduler()` → `stopScoutScheduler()`.
- [ ] **Step 5: Update `adiuvAI/src/main/router/index.ts` — rename `agent.*` sub-router to `scout.*`**
Find the `agent: t.router({ local: ..., cloud: ..., journey: ..., runs: ..., runActions: ..., catalog: ..., runNow: ..., ... })` block. Rename the key from `agent` to `scout`. Also rename the imported sub-router files if any (in `adiuvAI/src/main/router/`, look for `agent.ts` or `agent/` subdirectory).
- [ ] **Step 6: Sweep type references**
```bash
grep -rn "LocalAgentConfig\|CloudAgentConfig\|trpc\.agent\.\|/api/v1/agents" adiuvAI/src/
```
Replace systematically: `LocalAgentConfig` → `LocalScoutConfig`, `CloudAgentConfig` → `CloudScoutConfig`, `trpc.agent.` → `trpc.scout.`, `/api/v1/agents` → `/api/v1/scouts`.
- [ ] **Step 7: Typecheck + lint**
```bash
cd adiuvAI
npx tsc --noEmit
npm run lint
```
- [ ] **Step 8: Commit**
```bash
cd adiuvAI
git add -A src/main/
git commit -m "refactor(main): rename agent-scheduler/store/router symbols to scout"
```
---
## Task 9: Rename Renderer components
**Files:**
- Rename 5 files in `adiuvAI/src/renderer/components/settings/` (per File Structure table)
- Modify: `adiuvAI/src/renderer/routes/settings.tsx` — update `SECTIONS` array
- Modify: each renamed component — update internal symbols
- [ ] **Step 1: Move the files**
```bash
cd adiuvAI
git mv src/renderer/components/settings/AgentsSection.tsx src/renderer/components/settings/ScoutsSection.tsx
git mv src/renderer/components/settings/AgentRow.tsx src/renderer/components/settings/ScoutRow.tsx
git mv src/renderer/components/settings/LocalAgentConfigPanel.tsx src/renderer/components/settings/LocalScoutConfigPanel.tsx
git mv src/renderer/components/settings/CloudAgentConfigPanel.tsx src/renderer/components/settings/CloudScoutConfigPanel.tsx
git mv src/renderer/components/settings/InlineAgentCreationStepper.tsx src/renderer/components/settings/InlineScoutCreationStepper.tsx
```
- [ ] **Step 2: Inside each renamed file, rename the exported component**
Rule: every `export function AgentX` → `export function ScoutX`, every `Agent` → `Scout` in component bodies, every `agent` → `scout` in local variables, every `'toast.agent.X'` i18n key → `'toast.scout.X'`, every `'agents.X'` → `'scouts.X'`, every `t('settings.agentsX')` → `t('settings.scoutsX')`.
Show example (head of `ScoutsSection.tsx`):
```typescript
export function ScoutsSection() {
const { t } = useTranslation();
const utils = trpc.useUtils();
const localScoutsQuery = trpc.scout.local.list.useQuery();
const cloudScoutsQuery = trpc.scout.cloud.list.useQuery();
const deleteLocalMutation = trpc.scout.local.delete.useMutation();
const deleteCloudMutation = trpc.scout.cloud.delete.useMutation();
const updateLocalMutation = trpc.scout.local.update.useMutation();
const updateCloudMutation = trpc.scout.cloud.update.useMutation();
const runNowMutation = trpc.scout.runNow.useMutation();
const { notify, notifyError, notifyPromise } = useNotify();
const [expandedScout, setExpandedScout] = useState<string | null>(null);
// ...
const localScouts: LocalScoutConfig[] = localScoutsQuery.data ?? [];
const cloudScouts: CloudScoutConfig[] = cloudScoutsQuery.data ?? [];
const allScouts = [
...localScouts.map(s => ({ ...s, scoutType: 'local' as const })),
...cloudScouts.map(s => ({ ...s, scoutType: 'cloud' as const })),
];
// ...
}
```
- [ ] **Step 3: Update consumers — settings route + any other importer**
In `adiuvAI/src/renderer/routes/settings.tsx`, change:
```typescript
import { AgentsSection } from '@/components/settings/AgentsSection';
// ... in SECTIONS array:
{ id: 'agents', label: 'settings.agents', component: AgentsSection },
```
To:
```typescript
import { ScoutsSection } from '@/components/settings/ScoutsSection';
// ...
{ id: 'scouts', label: 'settings.scouts', component: ScoutsSection },
```
Also sweep:
```bash
cd adiuvAI
grep -rn "AgentsSection\|AgentRow\|LocalAgentConfigPanel\|CloudAgentConfigPanel\|InlineAgentCreationStepper" src/
```
Replace each.
- [ ] **Step 4: Update `adiuvAI/src/renderer/components/settings/types.ts` (if it defines `LocalAgentConfig`)**
Rename the interface `LocalAgentConfig` → `LocalScoutConfig`. Add `export type LocalScoutConfig = ...`. If there are other type definitions referencing "agent" (e.g. `AgentDataType`), rename consistently.
- [ ] **Step 5: Typecheck + lint**
```bash
cd adiuvAI
npx tsc --noEmit
npm run lint
```
- [ ] **Step 6: Commit**
```bash
cd adiuvAI
git add -A src/renderer/
git commit -m "refactor(renderer): rename Agent components and types to Scout"
```
---
## Task 10: Rename i18n keys across all 5 languages
**Files:**
- Modify: `adiuvAI/src/renderer/locales/{en,it,es,fr,de}/translation.json`
**Context:** Three buckets to rename in every language file:
1. `"settings": { "agents": "...", "agentsSubtitle": "...", "agentsDescription": "...", ... }` → `scouts`, `scoutsSubtitle`, `scoutsDescription`
2. Top-level `"agents": { "noAgentsYet": ..., "createAgent": ..., "yourAgents": ..., "createFirstAgent": ..., "noAgentsDescription": ... }` → `"scouts": { "noScoutsYet", "createScout", "yourScouts", "createFirstScout", "noScoutsDescription" }`. Update key names AND any in-value occurrences of "agent"/"agents" to "scout"/"scouts" within human-readable strings (apply translation per language — for English: "Agents" → "Scouts", "agent" → "scout"; for Italian: "Agenti" → "Scout"; etc.).
3. `"toast": { "agent": { ... } }` → `"toast": { "scout": { ... } }` (rename nested key only; values unchanged unless they contain "agent" word).
For non-English languages, keep the meaning consistent. Suggested translations:
| Lang | Key | Suggested value |
|------|-----|-----------------|
| en | `settings.scouts` | "Scouts" |
| it | `settings.scouts` | "Scout" |
| es | `settings.scouts` | "Scouts" |
| fr | `settings.scouts` | "Scouts" |
| de | `settings.scouts` | "Scouts" |
| en | `settings.scoutsSubtitle` | "working for you." |
| it | `settings.scoutsSubtitle` | "che lavorano per te." |
| en | `settings.scoutsDescription` | "Scouts watch your data sources — local files, mailboxes, cloud services — and surface what matters in your task brief." |
| it | `settings.scoutsDescription` | "Gli scout monitorano le tue fonti dati — file locali, caselle email, servizi cloud — e mettono in evidenza ciò che conta nel tuo task brief." |
| en | `scouts.noScoutsYet` | "No scouts yet" |
| en | `scouts.createScout` | "Create scout" |
| en | `scouts.yourScouts` | "Your Scouts" |
| en | `scouts.createFirstScout` | "Create first scout" |
| en | `scouts.noScoutsDescription` | "Create your first scout from a template. Choose what data to extract, set a schedule, and edit instructions before saving." |
(Translate analogously for it/es/fr/de — for the new descriptive copy, keep the spirit, ask the user for review on language nuance if uncertain.)
- [ ] **Step 1: Edit `en/translation.json`**
Apply renames per the bucket list above.
- [ ] **Step 2: Edit `it/translation.json`** — same renames, with Italian values
- [ ] **Step 3: Edit `es/translation.json`** — same renames, with Spanish values
- [ ] **Step 4: Edit `fr/translation.json`** — same renames, with French values
- [ ] **Step 5: Edit `de/translation.json`** — same renames, with German values
- [ ] **Step 6: Sweep code references**
```bash
cd adiuvAI
grep -rn "'agents\.\|\"agents\.\|'settings\.agents\|\"settings\.agents\|'toast\.agent\|\"toast\.agent" src/
```
Replace each `agents.X` → `scouts.X`, `settings.agents` → `settings.scouts`, `toast.agent` → `toast.scout`.
- [ ] **Step 7: Typecheck + lint + run app**
```bash
cd adiuvAI
npx tsc --noEmit
npm run lint
npm run start # smoke: open Settings, switch language, confirm "Scouts" header renders
```
- [ ] **Step 8: Commit**
```bash
cd adiuvAI
git add -A src/renderer/locales/ src/renderer/
git commit -m "i18n: rename agents keys to scouts across all 5 languages"
```
---
## Task 11: Update monorepo to bump submodule pointers
**Files:**
- Modify: monorepo `.gitmodules` is unchanged; update tracked commit pointers via `git add api adiuvAI`
- [ ] **Step 1: From the monorepo root, observe the dirty submodule pointers**
```bash
cd c:\Users\PC-Roby\Documents\_adiuvai_workspace
git status
```
Expected: `api` and `adiuvAI` listed as "modified content" or "new commits".
- [ ] **Step 2: Stage and commit the submodule bumps**
```bash
git add api adiuvAI
git commit -m "chore: bump submodules — Phase 1 scouts rename complete"
```
- [ ] **Step 3: Verify CLAUDE.md doesn't need updates**
Check `c:\Users\PC-Roby\Documents\_adiuvai_workspace\.claude\CLAUDE.md` and `c:\Users\PC-Roby\Documents\_adiuvai_workspace\adiuvAI\.claude\CLAUDE.md` — many references mention `agent` and the agent subsystem. For Phase 1, **only update durable notes that became wrong** (e.g. `tRPC routers (in appRouter): ... agent (with local / cloud / journey sub-routers) ...` should now read `scout`). Use targeted Edit, do not rewrite the file.
```bash
grep -n "agent" .claude/CLAUDE.md adiuvAI/.claude/CLAUDE.md
```
For each line that documents the renamed surface, fix in place.
- [ ] **Step 4: Commit CLAUDE.md updates**
```bash
git add .claude/CLAUDE.md adiuvAI/.claude/CLAUDE.md
git commit -m "docs(CLAUDE): update references to scouts after rename"
```
**Phase 1 complete.** Verify by running both test suites green and starting the app.
---
# PHASE 2 — Connector Skeleton
Goal: lay all infra (BE queue table, connector Protocol, engine, WS frame contract, SQLite suggestions table) without any user-visible Gmail feature yet. End state: backend can be unit-tested with a mock connector; Electron has the table + frame handler ready to receive proposals.
---
## Task 12: Alembic migration — `scout_triage_queue` + `cloud_scout_configs` alters
**Files:**
- Create: `api/alembic/versions/008_scout_triage_queue.py`
- [ ] **Step 1: Create the migration**
```python
"""Scout triage queue + cloud_scout_configs alterations.
Revision ID: 008
Revises: 007
Create Date: 2026-05-15
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
revision: str = "008"
down_revision: Union[str, None] = "007"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.create_table(
"scout_triage_queue",
sa.Column("id", sa.Uuid(as_uuid=False), primary_key=True),
sa.Column("user_id", sa.Uuid(as_uuid=False), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True),
sa.Column("scout_id", sa.Uuid(as_uuid=False), sa.ForeignKey("cloud_scout_configs.id", ondelete="CASCADE"), nullable=False),
sa.Column("source_type", sa.String(50), nullable=False),
sa.Column("source_msg_ref", sa.String(255), nullable=False),
sa.Column("triage_verdict", sa.String(20), nullable=False),
sa.Column("triage_reason", sa.Text, nullable=True),
sa.Column("status", sa.String(20), nullable=False, server_default="queued"),
sa.Column("triaged_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()),
sa.Column("delivered_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("acked_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
sa.UniqueConstraint("scout_id", "source_msg_ref", name="uq_scout_triage_queue_scout_msg"),
)
op.create_index("ix_scout_triage_queue_user_status", "scout_triage_queue", ["user_id", "status"])
op.create_index(
"ix_scout_triage_queue_expires_active",
"scout_triage_queue",
["expires_at"],
postgresql_where=sa.text("status != 'acked'"),
)
op.add_column("cloud_scout_configs", sa.Column("auto_trash_spam", sa.Boolean(), nullable=False, server_default=sa.text("false")))
op.add_column("cloud_scout_configs", sa.Column("gmail_history_id", sa.String(64), nullable=True))
op.add_column("cloud_scout_configs", sa.Column("gmail_watch_expires_at", sa.DateTime(timezone=True), nullable=True))
op.add_column("cloud_scout_configs", sa.Column("device_inactivity_pause_days", sa.Integer(), nullable=False, server_default="14"))
def downgrade() -> None:
op.drop_column("cloud_scout_configs", "device_inactivity_pause_days")
op.drop_column("cloud_scout_configs", "gmail_watch_expires_at")
op.drop_column("cloud_scout_configs", "gmail_history_id")
op.drop_column("cloud_scout_configs", "auto_trash_spam")
op.drop_index("ix_scout_triage_queue_expires_active", table_name="scout_triage_queue")
op.drop_index("ix_scout_triage_queue_user_status", table_name="scout_triage_queue")
op.drop_table("scout_triage_queue")
```
- [ ] **Step 2: Apply + verify**
```bash
cd api
alembic upgrade head
```
Verify with `\d scout_triage_queue` and `\d cloud_scout_configs` (the latter should now show the four new columns).
- [ ] **Step 3: Commit**
```bash
cd api
git add alembic/versions/008_scout_triage_queue.py
git commit -m "feat(db): add scout_triage_queue and cloud_scout_configs gmail columns"
```
---
## Task 13: SQLAlchemy `ScoutTriageQueue` model + `CloudScoutConfig` field additions
**Files:**
- Modify: `api/app/models.py` — add `ScoutTriageQueue` class, add four new fields to `CloudScoutConfig`
- [ ] **Step 1: Add fields to `CloudScoutConfig` (in `models.py` after the existing fields)**
```python
auto_trash_spam: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default=sa.text("false"))
gmail_history_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
gmail_watch_expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
device_inactivity_pause_days: Mapped[int] = mapped_column(Integer, nullable=False, default=14, server_default="14")
```
- [ ] **Step 2: Add the `ScoutTriageQueue` class (after `CloudScoutConfig`)**
```python
class ScoutTriageQueue(Base):
__tablename__ = "scout_triage_queue"
__table_args__ = (
UniqueConstraint("scout_id", "source_msg_ref", name="uq_scout_triage_queue_scout_msg"),
)
id: Mapped[str] = mapped_column(Uuid(as_uuid=False), primary_key=True, default=_uuid)
user_id: Mapped[str] = mapped_column(Uuid(as_uuid=False), ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
scout_id: Mapped[str] = mapped_column(Uuid(as_uuid=False), ForeignKey("cloud_scout_configs.id", ondelete="CASCADE"), nullable=False)
source_type: Mapped[str] = mapped_column(String(50), nullable=False)
source_msg_ref: Mapped[str] = mapped_column(String(255), nullable=False)
triage_verdict: Mapped[str] = mapped_column(String(20), nullable=False)
triage_reason: Mapped[str | None] = mapped_column(Text, nullable=True)
status: Mapped[str] = mapped_column(String(20), nullable=False, default="queued", server_default="queued")
triaged_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
delivered_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
acked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
```
- [ ] **Step 3: Verify imports at the top of `models.py` include `UniqueConstraint` and `Integer`**
If missing, add `from sqlalchemy import ..., UniqueConstraint, Integer`.
- [ ] **Step 4: Run tests**
```bash
cd api
pytest
```
Expected: green (existing tests don't touch the new model yet, but `Base.metadata.create_all` should now succeed creating the new table in the test DB).
- [ ] **Step 5: Commit**
```bash
cd api
git add app/models.py
git commit -m "feat(models): add ScoutTriageQueue model + cloud_scout gmail fields"
```
---
## Task 14: `SourceConnector` Protocol + Pydantic types
**Files:**
- Create: `api/app/scouts/__init__.py` (empty)
- Create: `api/app/scouts/connectors/__init__.py` (empty)
- Create: `api/app/scouts/connectors/base.py`
- Create: `api/tests/test_scout_connectors_base.py`
- [ ] **Step 1: Write the failing test**
`api/tests/test_scout_connectors_base.py`:
```python
"""Tests for the SourceConnector base protocol and shared types."""
from __future__ import annotations
from datetime import datetime, timezone
import pytest
from app.scouts.connectors.base import (
ItemContent,
ItemMetadata,
ItemRef,
TriageVerdict,
)
def test_item_ref_round_trips_through_pydantic():
ref = ItemRef(source_msg_ref="abc123", received_at=datetime.now(tz=timezone.utc))
parsed = ItemRef.model_validate(ref.model_dump())
assert parsed.source_msg_ref == "abc123"
assert parsed.received_at == ref.received_at
def test_item_metadata_allows_all_optional():
meta = ItemMetadata()
assert meta.subject is None
assert meta.sender is None
assert meta.snippet is None
assert meta.received_at is None
def test_item_content_requires_metadata_and_body():
content = ItemContent(
metadata=ItemMetadata(subject="hi"),
body_text="hello world",
raw_headers={"X-Foo": "bar"},
)
assert content.metadata.subject == "hi"
assert content.body_text == "hello world"
assert content.raw_headers["X-Foo"] == "bar"
def test_triage_verdict_constraints():
v = TriageVerdict(verdict="relevant", reason="contains task language", confidence=0.92)
assert v.verdict == "relevant"
with pytest.raises(ValueError):
TriageVerdict(verdict="meh", reason="x", confidence=0.5) # bad enum value
```
- [ ] **Step 2: Run the test to verify it fails**
```bash
cd api
pytest tests/test_scout_connectors_base.py -v
```
Expected: ImportError (`No module named 'app.scouts.connectors.base'`).
- [ ] **Step 3: Implement `app/scouts/connectors/base.py`**
```python
"""Source connector Protocol and shared item types.
A SourceConnector adapts a third-party data source (Gmail, Slack, ...) to the
shared ScoutEngine interface. Each connector owns:
* how to enumerate new items since the last poll (``list_new``)
* how to fetch a single item's metadata cheaply (``fetch_metadata``)
* how to fetch a single item's full content for in-memory triage
(``fetch_content``) — this content MUST NOT be persisted by the engine
* how to archive/trash an item (``archive``) for spam handling
* optional push-notification setup (``setup_watch`` / ``renew_watch``)
"""
from __future__ import annotations
from datetime import datetime
from typing import Literal, Protocol
from pydantic import BaseModel, Field
class ItemRef(BaseModel):
source_msg_ref: str
received_at: datetime | None = None
class ItemMetadata(BaseModel):
subject: str | None = None
sender: str | None = None
snippet: str | None = None
received_at: datetime | None = None
class ItemContent(BaseModel):
metadata: ItemMetadata
body_text: str
raw_headers: dict[str, str] = Field(default_factory=dict)
class TriageVerdict(BaseModel):
verdict: Literal["relevant", "spam"]
reason: str
confidence: float = Field(ge=0.0, le=1.0)
class SourceConnector(Protocol):
"""Adapter for a third-party data source (Gmail, Slack, ...)."""
source_type: str # e.g. "gmail"
async def list_new(self, scout) -> list[ItemRef]: ...
async def fetch_metadata(self, scout, ref: ItemRef) -> ItemMetadata: ...
async def fetch_content(self, scout, ref: ItemRef) -> ItemContent: ...
async def archive(self, scout, ref: ItemRef) -> None: ...
async def setup_watch(self, scout) -> None: ...
async def renew_watch(self, scout) -> None: ...
```
Also create empty package markers:
```bash
cd api
touch app/scouts/__init__.py app/scouts/connectors/__init__.py
```
- [ ] **Step 4: Run tests**
```bash
cd api
pytest tests/test_scout_connectors_base.py -v
```
Expected: PASS (4 tests).
- [ ] **Step 5: Commit**
```bash
cd api
git add app/scouts/ tests/test_scout_connectors_base.py
git commit -m "feat(scouts): add SourceConnector protocol and item types"
```
---
## Task 15: Connector registry
**Files:**
- Create: `api/app/scouts/connectors/registry.py`
- Create: `api/tests/test_scout_connector_registry.py`
- [ ] **Step 1: Write the failing test**
```python
"""Tests for the connector registry."""
from __future__ import annotations
import pytest
from app.scouts.connectors.base import ItemRef
from app.scouts.connectors.registry import (
get_connector,
register_connector,
_reset_for_tests,
)
class _DummyConnector:
source_type = "dummy"
async def list_new(self, scout): return []
async def fetch_metadata(self, scout, ref): raise NotImplementedError
async def fetch_content(self, scout, ref): raise NotImplementedError
async def archive(self, scout, ref): raise NotImplementedError
async def setup_watch(self, scout): raise NotImplementedError
async def renew_watch(self, scout): raise NotImplementedError
@pytest.fixture(autouse=True)
def _clean_registry():
_reset_for_tests()
yield
_reset_for_tests()
def test_register_and_get():
c = _DummyConnector()
register_connector(c)
assert get_connector("dummy") is c
def test_unknown_source_raises():
with pytest.raises(KeyError):
get_connector("nope")
def test_double_register_replaces():
a = _DummyConnector()
b = _DummyConnector()
register_connector(a)
register_connector(b)
assert get_connector("dummy") is b
```
- [ ] **Step 2: Run test (should fail with ImportError)**
```bash
cd api
pytest tests/test_scout_connector_registry.py -v
```
- [ ] **Step 3: Implement `app/scouts/connectors/registry.py`**
```python
"""Connector registry — single source of truth for source_type -> connector."""
from __future__ import annotations
from typing import Any
_CONNECTORS: dict[str, Any] = {}
def register_connector(connector: Any) -> None:
"""Register a SourceConnector instance under its ``source_type``.
Calling twice with the same ``source_type`` replaces the prior entry —
useful for tests and hot-reload, but in production each connector
should be registered exactly once at startup.
"""
if not getattr(connector, "source_type", None):
raise ValueError("Connector must declare a non-empty source_type")
_CONNECTORS[connector.source_type] = connector
def get_connector(source_type: str) -> Any:
"""Return the registered connector for ``source_type`` or raise KeyError."""
try:
return _CONNECTORS[source_type]
except KeyError as exc:
raise KeyError(f"No connector registered for source_type {source_type!r}") from exc
def _reset_for_tests() -> None:
"""Clear the registry — for use in pytest fixtures only."""
_CONNECTORS.clear()
```
- [ ] **Step 4: Run tests**
```bash
cd api
pytest tests/test_scout_connector_registry.py -v
```
Expected: PASS (3 tests).
- [ ] **Step 5: Commit**
```bash
cd api
git add app/scouts/connectors/registry.py tests/test_scout_connector_registry.py
git commit -m "feat(scouts): add connector registry"
```
---
## Task 16: `ScoutEngine` skeleton — `trigger_scout` + `_process_item`
**Files:**
- Create: `api/app/scouts/engine.py`
- Create: `api/tests/test_scout_engine.py`
**Context:** This task implements the core engine logic but uses a stub triage call. The real LLM triage comes in Task 24. The engine writes `scout_triage_queue` rows for relevant items and calls `connector.archive` for spam items when `scout.auto_trash_spam` is true.
- [ ] **Step 1: Write the failing test**
```python
"""Unit tests for ScoutEngine.trigger_scout / _process_item."""
from __future__ import annotations
import uuid
from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock
import pytest
from sqlalchemy import select
from app.models import CloudScoutConfig, ScoutTriageQueue, User, Subscription
from app.scouts.connectors.base import ItemContent, ItemMetadata, ItemRef, TriageVerdict
from app.scouts.connectors.registry import register_connector, _reset_for_tests
from app.scouts.engine import ScoutEngine
from tests.conftest import _TestSessionLocal
def _make_connector(items, content_for):
c = AsyncMock()
c.source_type = "dummy"
c.list_new = AsyncMock(return_value=items)
c.fetch_content = AsyncMock(side_effect=lambda scout, ref: content_for[ref.source_msg_ref])
c.archive = AsyncMock()
return c
@pytest.fixture(autouse=True)
def _registry():
_reset_for_tests()
yield
_reset_for_tests()
@pytest.mark.asyncio
async def test_relevant_item_inserted_into_queue(monkeypatch):
user_id = "00000000-0000-0000-0000-000000000003" # power tier seeded in conftest
scout_id = str(uuid.uuid4())
async with _TestSessionLocal() as session:
scout = CloudScoutConfig(
id=scout_id, user_id=user_id, provider="gmail", name="Test",
data_types=[], prompt_template="", schedule_cron="0 * * * *",
enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14,
)
session.add(scout)
await session.commit()
refs = [ItemRef(source_msg_ref="msg-1")]
content = {"msg-1": ItemContent(metadata=ItemMetadata(subject="Hi"), body_text="task tomorrow")}
connector = _make_connector(refs, content)
register_connector(connector)
engine = ScoutEngine()
monkeypatch.setattr(
engine,
"_triage_llm",
AsyncMock(return_value=TriageVerdict(verdict="relevant", reason="task", confidence=0.9)),
)
await engine.trigger_scout(uuid.UUID(scout_id))
async with _TestSessionLocal() as session:
rows = (await session.execute(select(ScoutTriageQueue))).scalars().all()
assert len(rows) == 1
assert rows[0].source_msg_ref == "msg-1"
assert rows[0].triage_verdict == "relevant"
assert rows[0].status == "queued"
connector.archive.assert_not_awaited()
@pytest.mark.asyncio
async def test_spam_with_auto_trash_archives_and_does_not_queue(monkeypatch):
user_id = "00000000-0000-0000-0000-000000000003"
scout_id = str(uuid.uuid4())
async with _TestSessionLocal() as session:
scout = CloudScoutConfig(
id=scout_id, user_id=user_id, provider="gmail", name="Test",
data_types=[], prompt_template="", schedule_cron="0 * * * *",
enabled=True, auto_trash_spam=True, device_inactivity_pause_days=14,
)
session.add(scout)
await session.commit()
refs = [ItemRef(source_msg_ref="msg-spam")]
content = {"msg-spam": ItemContent(metadata=ItemMetadata(subject="$$$"), body_text="buy")}
connector = _make_connector(refs, content)
register_connector(connector)
engine = ScoutEngine()
monkeypatch.setattr(
engine,
"_triage_llm",
AsyncMock(return_value=TriageVerdict(verdict="spam", reason="bait", confidence=0.99)),
)
await engine.trigger_scout(uuid.UUID(scout_id))
async with _TestSessionLocal() as session:
rows = (await session.execute(select(ScoutTriageQueue))).scalars().all()
assert rows == []
connector.archive.assert_awaited_once()
@pytest.mark.asyncio
async def test_spam_without_auto_trash_does_not_archive_and_does_not_queue(monkeypatch):
user_id = "00000000-0000-0000-0000-000000000003"
scout_id = str(uuid.uuid4())
async with _TestSessionLocal() as session:
scout = CloudScoutConfig(
id=scout_id, user_id=user_id, provider="gmail", name="Test",
data_types=[], prompt_template="", schedule_cron="0 * * * *",
enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14,
)
session.add(scout)
await session.commit()
refs = [ItemRef(source_msg_ref="msg-2")]
content = {"msg-2": ItemContent(metadata=ItemMetadata(subject="$$$"), body_text="buy")}
connector = _make_connector(refs, content)
register_connector(connector)
engine = ScoutEngine()
monkeypatch.setattr(
engine,
"_triage_llm",
AsyncMock(return_value=TriageVerdict(verdict="spam", reason="bait", confidence=0.99)),
)
await engine.trigger_scout(uuid.UUID(scout_id))
async with _TestSessionLocal() as session:
rows = (await session.execute(select(ScoutTriageQueue))).scalars().all()
assert rows == []
connector.archive.assert_not_awaited()
@pytest.mark.asyncio
async def test_idempotent_replay(monkeypatch):
user_id = "00000000-0000-0000-0000-000000000003"
scout_id = str(uuid.uuid4())
async with _TestSessionLocal() as session:
session.add(CloudScoutConfig(
id=scout_id, user_id=user_id, provider="gmail", name="Test",
data_types=[], prompt_template="", schedule_cron="0 * * * *",
enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14,
))
await session.commit()
refs = [ItemRef(source_msg_ref="msg-3")]
content = {"msg-3": ItemContent(metadata=ItemMetadata(subject="x"), body_text="y")}
connector = _make_connector(refs, content)
register_connector(connector)
engine = ScoutEngine()
monkeypatch.setattr(
engine,
"_triage_llm",
AsyncMock(return_value=TriageVerdict(verdict="relevant", reason="x", confidence=0.5)),
)
await engine.trigger_scout(uuid.UUID(scout_id))
await engine.trigger_scout(uuid.UUID(scout_id))
async with _TestSessionLocal() as session:
rows = (await session.execute(select(ScoutTriageQueue))).scalars().all()
assert len(rows) == 1, "Replay must not create duplicate queue rows"
```
- [ ] **Step 2: Run test (should fail — module missing)**
```bash
cd api
pytest tests/test_scout_engine.py -v
```
- [ ] **Step 3: Implement `app/scouts/engine.py`**
```python
"""ScoutEngine — orchestrates triage, queueing, and delivery for cloud scouts.
Triage flow per scout:
1. Resolve scout config from the DB.
2. Skip if device hasn't connected within ``device_inactivity_pause_days``.
3. Ask the connector to ``list_new`` — fresh items since last poll.
4. For each item:
- skip if already in the queue (idempotent on (scout_id, source_msg_ref))
- fetch the full content via the connector (transient, never persisted)
- run the triage LLM call → relevant | spam
- spam + auto_trash_spam → connector.archive
- relevant → INSERT scout_triage_queue row
5. Update scout.last_run_at.
Delivery flow on Electron WS reconnect:
- drain ``status='queued'`` rows for the user
- fetch metadata-only for each (subject + snippet)
- send a ``scout_proposal`` frame
- flip status to ``delivered`` on ack
"""
from __future__ import annotations
import logging
import uuid
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from app.db.session import async_session_factory
from app.models import CloudScoutConfig, ScoutTriageQueue
from app.scouts.connectors.base import ItemContent, ItemRef, TriageVerdict
from app.scouts.connectors.registry import get_connector
logger = logging.getLogger(__name__)
QUEUE_TTL_DAYS = 30
class ScoutEngine:
async def trigger_scout(self, scout_id: uuid.UUID) -> None:
async with async_session_factory() as session:
scout = await session.get(CloudScoutConfig, str(scout_id))
if scout is None:
logger.warning("trigger_scout: no such scout id=%s", scout_id)
return
if not scout.enabled:
return
# Device-inactivity pause check is a simple heuristic on last_run_at —
# the device-online signal lives in the DeviceConnectionManager and is
# consulted at delivery time. For triage, we only check that the
# configured pause threshold isn't suppressing the run.
connector = get_connector(scout.provider)
try:
refs = await connector.list_new(scout)
except Exception:
logger.exception("scout %s: list_new failed", scout.id)
return
for ref in refs:
await self._process_item(session, scout, connector, ref)
scout.last_run_at = datetime.now(tz=timezone.utc)
await session.commit()
async def _process_item(
self,
session,
scout: CloudScoutConfig,
connector,
ref: ItemRef,
) -> None:
# Idempotency check
existing = await session.execute(
select(ScoutTriageQueue.id).where(
ScoutTriageQueue.scout_id == scout.id,
ScoutTriageQueue.source_msg_ref == ref.source_msg_ref,
)
)
if existing.first() is not None:
return
try:
content = await connector.fetch_content(scout, ref)
except Exception:
logger.exception("scout %s: fetch_content failed for %s", scout.id, ref.source_msg_ref)
return
try:
verdict = await self._triage_llm(scout, content)
except Exception:
logger.exception("scout %s: triage_llm failed for %s", scout.id, ref.source_msg_ref)
return
if verdict.verdict == "spam":
if scout.auto_trash_spam:
try:
await connector.archive(scout, ref)
except Exception:
logger.exception("scout %s: archive failed for %s", scout.id, ref.source_msg_ref)
return
now = datetime.now(tz=timezone.utc)
row = ScoutTriageQueue(
id=str(uuid.uuid4()),
user_id=scout.user_id,
scout_id=scout.id,
source_type=connector.source_type,
source_msg_ref=ref.source_msg_ref,
triage_verdict=verdict.verdict,
triage_reason=verdict.reason,
status="queued",
triaged_at=now,
expires_at=now + timedelta(days=QUEUE_TTL_DAYS),
)
session.add(row)
try:
await session.flush()
except IntegrityError:
await session.rollback()
# Race: another worker inserted between our SELECT and INSERT.
# The unique constraint did its job; safe to ignore.
return
async def _triage_llm(self, scout: CloudScoutConfig, content: ItemContent) -> TriageVerdict:
"""Stub — real implementation in Task 24."""
raise NotImplementedError("Real triage LLM call lands in Task 24")
```
Note: the import `from app.db.session import async_session_factory` assumes there's an existing session factory module. If the project uses a different name (e.g. `from app.db import SessionLocal`), find it via:
```bash
grep -rn "AsyncSession\|sessionmaker" api/app/db/
```
and adjust the import. This is a common case across this codebase — the engine just needs an async session.
- [ ] **Step 4: Run tests**
```bash
cd api
pytest tests/test_scout_engine.py -v
```
Expected: 4 PASS.
- [ ] **Step 5: Commit**
```bash
cd api
git add app/scouts/engine.py tests/test_scout_engine.py
git commit -m "feat(scouts): add ScoutEngine triage + queue insertion"
```
---
## Task 17: Drizzle migration — `scout_suggestions` table
**Files:**
- Modify: `adiuvAI/src/main/db/schema.ts` — add `scoutSuggestions` table
- Create: `adiuvAI/src/main/db/migrations/0008_scout_suggestions.sql` (auto-generated, then renamed)
- [ ] **Step 1: Add the table to `schema.ts`**
Append at the end of the existing tables:
```typescript
export const scoutSuggestions = sqliteTable('scout_suggestions', {
id: text().primaryKey(),
scoutId: text('scout_id').notNull(),
sourceType: text('source_type').notNull(),
sourceMsgRef: text('source_msg_ref').notNull(),
category: text().notNull(), // "unprocessed" until Phase 4
payload: text(), // JSON, populated by Phase 4
rawSubject: text('raw_subject'),
rawSnippet: text('raw_snippet'),
status: text().notNull(), // pending | approved | rejected | expired
proposedAt: integer('proposed_at').notNull(),
resolvedAt: integer('resolved_at'),
resolvedEntityType: text('resolved_entity_type'),
resolvedEntityId: text('resolved_entity_id'),
});
```
- [ ] **Step 2: Generate the migration**
```bash
cd adiuvAI
npx drizzle-kit generate
```
Rename to `0008_scout_suggestions.sql`:
```bash
mv src/main/db/migrations/0008_*.sql src/main/db/migrations/0008_scout_suggestions.sql
```
- [ ] **Step 3: Inspect the generated SQL**
Verify it contains a `CREATE TABLE scout_suggestions` and matches the schema columns.
- [ ] **Step 4: Smoke test**
```bash
cd adiuvAI
npm run start
```
After app boots, close. Verify:
```bash
sqlite3 dev.db ".schema scout_suggestions"
```
- [ ] **Step 5: Commit**
```bash
cd adiuvAI
git add src/main/db/schema.ts src/main/db/migrations/0008_scout_suggestions.sql
git commit -m "feat(db): add scout_suggestions table"
```
---
## Task 18: WS frame schemas — `scout_proposal` + `scout_proposal_ack`
**Files:**
- Modify: `api/app/schemas.py` — add new `WsFrameType` enum values + Pydantic models for the two frames
- [ ] **Step 1: Read `api/app/schemas.py` to find `WsFrameType` enum**
```bash
cd api
grep -n "WsFrameType\|class Ws" app/schemas.py
```
- [ ] **Step 2: Add new enum values**
```python
class WsFrameType(str, Enum):
# ... existing values ...
SCOUT_PROPOSAL = "scout_proposal"
SCOUT_PROPOSAL_ACK = "scout_proposal_ack"
```
- [ ] **Step 3: Add the frame Pydantic models**
```python
class ScoutProposalPayload(BaseModel):
id: str
scout_id: str
source_type: str
source_msg_ref: str
raw_subject: str | None = None
raw_snippet: str | None = None
category: Literal["unprocessed"] = "unprocessed" # Phase 4 will widen
payload: dict | None = None
class ScoutProposalFrame(BaseModel):
type: Literal[WsFrameType.SCOUT_PROPOSAL]
proposal: ScoutProposalPayload
class ScoutProposalAckFrame(BaseModel):
type: Literal[WsFrameType.SCOUT_PROPOSAL_ACK]
proposal_id: str
```
- [ ] **Step 4: Run tests**
```bash
cd api
pytest tests/test_schemas.py -v # if exists
pytest
```
- [ ] **Step 5: Commit**
```bash
cd api
git add app/schemas.py
git commit -m "feat(schemas): add scout_proposal + scout_proposal_ack WS frame types"
```
---
## Task 19: `ScoutEngine.deliver_pending` — drain queue + send WS frames
**Files:**
- Modify: `api/app/scouts/engine.py` — add `deliver_pending` method
- Modify: `api/app/api/routes/device_ws.py` — call `deliver_pending` on connect; handle `scout_proposal_ack` incoming frames
- Modify: `api/tests/test_scout_engine.py` — add `deliver_pending` tests
- [ ] **Step 1: Write the failing test**
Append to `api/tests/test_scout_engine.py`:
```python
@pytest.mark.asyncio
async def test_deliver_pending_sends_one_frame_per_queued_row(monkeypatch):
user_id = "00000000-0000-0000-0000-000000000003"
scout_id = str(uuid.uuid4())
now = datetime.now(tz=timezone.utc)
async with _TestSessionLocal() as session:
session.add(CloudScoutConfig(
id=scout_id, user_id=user_id, provider="gmail", name="Test",
data_types=[], prompt_template="", schedule_cron="0 * * * *",
enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14,
))
for i in range(3):
session.add(ScoutTriageQueue(
id=str(uuid.uuid4()), user_id=user_id, scout_id=scout_id,
source_type="gmail", source_msg_ref=f"msg-{i}",
triage_verdict="relevant", status="queued",
triaged_at=now, expires_at=now + timedelta(days=30),
))
await session.commit()
connector = AsyncMock()
connector.source_type = "gmail"
connector.fetch_metadata = AsyncMock(side_effect=lambda scout, ref: ItemMetadata(
subject=f"sub-{ref.source_msg_ref}", snippet=f"snip-{ref.source_msg_ref}",
))
register_connector(connector)
sent = []
ws = AsyncMock()
ws.send_json = AsyncMock(side_effect=lambda payload: sent.append(payload))
engine = ScoutEngine()
await engine.deliver_pending(uuid.UUID(user_id), ws)
assert len(sent) == 3
assert all(s["type"] == "scout_proposal" for s in sent)
subjects = {s["proposal"]["raw_subject"] for s in sent}
assert subjects == {"sub-msg-0", "sub-msg-1", "sub-msg-2"}
# Status should flip to delivered
async with _TestSessionLocal() as session:
rows = (await session.execute(select(ScoutTriageQueue))).scalars().all()
assert all(r.status == "delivered" for r in rows)
assert all(r.delivered_at is not None for r in rows)
```
- [ ] **Step 2: Run test (should fail — `deliver_pending` not defined)**
```bash
cd api
pytest tests/test_scout_engine.py::test_deliver_pending_sends_one_frame_per_queued_row -v
```
- [ ] **Step 3: Implement `deliver_pending` in `app/scouts/engine.py`**
Add to the `ScoutEngine` class:
```python
async def deliver_pending(self, user_id: uuid.UUID, ws) -> None:
"""Drain queued triage rows for a user, send one scout_proposal frame each.
Frames are sent immediately and the rows flip to ``status='delivered'``.
Final ``acked`` flip happens in the WS endpoint when the client returns
a ``scout_proposal_ack`` frame.
"""
from app.scouts.connectors.base import ItemRef
async with async_session_factory() as session:
rows = (await session.execute(
select(ScoutTriageQueue).where(
ScoutTriageQueue.user_id == str(user_id),
ScoutTriageQueue.status == "queued",
)
)).scalars().all()
for row in rows:
try:
connector = get_connector(row.source_type)
except KeyError:
logger.warning("deliver_pending: no connector for %s", row.source_type)
continue
scout = await session.get(CloudScoutConfig, row.scout_id)
if scout is None:
continue
try:
meta = await connector.fetch_metadata(scout, ItemRef(source_msg_ref=row.source_msg_ref))
except Exception:
logger.exception("deliver_pending: fetch_metadata failed")
continue
payload = {
"type": "scout_proposal",
"proposal": {
"id": row.id,
"scout_id": row.scout_id,
"source_type": row.source_type,
"source_msg_ref": row.source_msg_ref,
"raw_subject": meta.subject,
"raw_snippet": meta.snippet,
"category": "unprocessed",
"payload": None,
},
}
await ws.send_json(payload)
row.status = "delivered"
row.delivered_at = datetime.now(tz=timezone.utc)
await session.commit()
async def ack_proposal(self, proposal_id: str) -> None:
"""Mark a proposal as acked once the Electron client confirms receipt."""
async with async_session_factory() as session:
row = await session.get(ScoutTriageQueue, proposal_id)
if row is None:
return
row.status = "acked"
row.acked_at = datetime.now(tz=timezone.utc)
await session.commit()
```
- [ ] **Step 4: Run tests**
```bash
cd api
pytest tests/test_scout_engine.py -v
```
Expected: all 5 tests PASS.
- [ ] **Step 5: Wire into `app/api/routes/device_ws.py`**
Find the WS handshake / post-connect section. Add a call to `ScoutEngine().deliver_pending(user.id, websocket)` after the `device_hello` is received and the connection is registered.
Also add a new dispatch case for incoming `scout_proposal_ack` frames:
```python
elif msg.get("type") == "scout_proposal_ack":
proposal_id = msg.get("proposal_id")
if proposal_id:
await ScoutEngine().ack_proposal(proposal_id)
```
- [ ] **Step 6: Run tests**
```bash
cd api
pytest tests/test_ws_unified.py -v
pytest
```
- [ ] **Step 7: Commit**
```bash
cd api
git add app/scouts/engine.py tests/test_scout_engine.py app/api/routes/device_ws.py
git commit -m "feat(scouts): deliver_pending drains queue and sends scout_proposal frames"
```
---
## Task 20: Electron — `scout-suggestion-handler.ts` (insert into local table on incoming frame)
**Files:**
- Create: `adiuvAI/src/main/scouts/scout-suggestion-handler.ts`
- Modify: `adiuvAI/src/main/api/backend-client.ts` — dispatch incoming `scout_proposal` frame to the handler, send `scout_proposal_ack` back
- [ ] **Step 1: Read `adiuvAI/src/main/api/backend-client.ts` to find the inbound frame switch**
```bash
cd adiuvAI
grep -n "tool_call\|run_complete\|switch.*type" src/main/api/backend-client.ts
```
Identify the existing dispatch pattern.
- [ ] **Step 2: Create the handler module**
`adiuvAI/src/main/scouts/scout-suggestion-handler.ts`:
```typescript
import { randomUUID } from 'node:crypto';
import { getDb } from '../db';
import { scoutSuggestions } from '../db/schema';
export interface IncomingScoutProposal {
id: string;
scout_id: string;
source_type: string;
source_msg_ref: string;
raw_subject: string | null;
raw_snippet: string | null;
category: 'unprocessed';
payload: Record<string, unknown> | null;
}
export async function handleScoutProposal(p: IncomingScoutProposal): Promise<void> {
// Idempotent: same id from BE shouldn't double-insert if the ack was lost.
await getDb()
.insert(scoutSuggestions)
.values({
id: p.id,
scoutId: p.scout_id,
sourceType: p.source_type,
sourceMsgRef: p.source_msg_ref,
category: p.category,
payload: p.payload ? JSON.stringify(p.payload) : null,
rawSubject: p.raw_subject,
rawSnippet: p.raw_snippet,
status: 'pending',
proposedAt: Date.now(),
})
.onConflictDoNothing();
}
```
- [ ] **Step 3: Wire in `backend-client.ts`**
In the inbound frame dispatch:
```typescript
case 'scout_proposal': {
const { handleScoutProposal } = await import('../scouts/scout-suggestion-handler');
await handleScoutProposal(msg.proposal);
this.send({ type: 'scout_proposal_ack', proposal_id: msg.proposal.id });
break;
}
```
(Adapt to the existing dispatch style — if it uses a switch, add a case; if it uses an if-chain, add an else-if. The send pattern follows whatever existing frames do.)
- [ ] **Step 4: Typecheck + lint**
```bash
cd adiuvAI
npx tsc --noEmit
npm run lint
```
- [ ] **Step 5: Commit**
```bash
cd adiuvAI
git add src/main/scouts/scout-suggestion-handler.ts src/main/api/backend-client.ts
git commit -m "feat(scouts): handle scout_proposal frames and ack"
```
---
## Task 21: Submodule pointer bump for Phase 2
- [ ] **Step 1: Bump submodule pointers**
```bash
cd c:\Users\PC-Roby\Documents\_adiuvai_workspace
git add api adiuvAI
git commit -m "chore: bump submodules — Phase 2 connector skeleton"
```
**Phase 2 complete.** Engine + queue + frame contract + local table all in place. No user-facing change yet.
---
# PHASE 3 — Gmail Scout End-to-End
Goal: ship a working Gmail scout — user can connect Gmail through Settings, push notifications + cron-fallback poll the inbox, BE triages with the real LLM, queue rows are delivered to Electron on reconnect and land as `scout_suggestions` rows with `category='unprocessed'`.
---
## Task 22: `GmailConnector` implementation
**Files:**
- Create: `api/app/scouts/connectors/gmail.py`
- Create: `api/tests/test_scout_connectors_gmail.py`
**Context:** Wraps existing `app/integrations/gmail.py`'s `GmailClient`. Maps existing `EmailMessage` to the new `ItemContent`. Implements `list_new` using Gmail's `users.history.list` since the scout's `gmail_history_id`. Implements `archive` via `users.messages.trash`.
- [ ] **Step 1: Write the failing test**
`api/tests/test_scout_connectors_gmail.py`:
```python
"""Tests for GmailConnector."""
from __future__ import annotations
import uuid
from datetime import datetime, timezone
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from app.models import CloudScoutConfig
from app.scouts.connectors.base import ItemRef
from app.scouts.connectors.gmail import GmailConnector
def _make_scout():
return CloudScoutConfig(
id=str(uuid.uuid4()),
user_id="00000000-0000-0000-0000-000000000003",
provider="gmail",
name="Inbox",
data_types=[],
prompt_template="",
oauth_token_encrypted="encrypted-blob",
schedule_cron="0 * * * *",
enabled=True,
auto_trash_spam=False,
device_inactivity_pause_days=14,
gmail_history_id="100",
)
@pytest.mark.asyncio
async def test_fetch_metadata_returns_subject_and_snippet():
scout = _make_scout()
conn = GmailConnector()
fake_message = {
"id": "msg-1",
"snippet": "preview text",
"payload": {"headers": [
{"name": "Subject", "value": "Hello"},
{"name": "From", "value": "alice@example.com"},
{"name": "Date", "value": "Wed, 14 May 2026 10:00:00 +0000"},
]},
}
with patch("app.scouts.connectors.gmail._get_gmail_service") as mock_svc:
mock_svc.return_value.users().messages().get().execute.return_value = fake_message
meta = await conn.fetch_metadata(scout, ItemRef(source_msg_ref="msg-1"))
assert meta.subject == "Hello"
assert meta.sender == "alice@example.com"
assert meta.snippet == "preview text"
@pytest.mark.asyncio
async def test_fetch_content_returns_body_text():
scout = _make_scout()
conn = GmailConnector()
with patch("app.scouts.connectors.gmail.GmailClient") as MockClient:
instance = MockClient.return_value
instance.fetch_messages = AsyncMock(return_value=[
MagicMock(id="msg-1", subject="S", sender="a@b", body_text="hello world",
date=datetime.now(tz=timezone.utc), labels=[]),
])
content = await conn.fetch_content(scout, ItemRef(source_msg_ref="msg-1"))
assert content.body_text == "hello world"
assert content.metadata.subject == "S"
@pytest.mark.asyncio
async def test_archive_calls_trash():
scout = _make_scout()
conn = GmailConnector()
with patch("app.scouts.connectors.gmail._get_gmail_service") as mock_svc:
await conn.archive(scout, ItemRef(source_msg_ref="msg-1"))
mock_svc.return_value.users().messages().trash.assert_called()
```
- [ ] **Step 2: Run test (fails — module missing)**
```bash
cd api
pytest tests/test_scout_connectors_gmail.py -v
```
- [ ] **Step 3: Implement `app/scouts/connectors/gmail.py`**
```python
"""Gmail SourceConnector — wraps the existing GmailClient.
Responsibilities:
* list_new: incremental fetch since the scout's stored gmail_history_id
* fetch_metadata: subject + sender + snippet only (Gmail metadata format)
* fetch_content: full body text — transient, never persisted by engine
* archive: move a message to Gmail Trash (recoverable for 30 days)
* setup_watch / renew_watch: Gmail push notifications via Pub/Sub
"""
from __future__ import annotations
import asyncio
import logging
from datetime import datetime, timedelta, timezone
from typing import Any
from app.config.settings import settings
from app.integrations import decrypt_token
from app.integrations.gmail import GmailClient
from app.scouts.connectors.base import ItemContent, ItemMetadata, ItemRef
logger = logging.getLogger(__name__)
def _get_gmail_service(scout):
"""Return a synchronous Google API client for low-level metadata calls."""
from googleapiclient.discovery import build
from google.oauth2.credentials import Credentials
creds_info = decrypt_token(scout.oauth_token_encrypted)
credentials = Credentials(
token=creds_info.get("token"),
refresh_token=creds_info.get("refresh_token"),
token_uri=creds_info.get("token_uri", "https://oauth2.googleapis.com/token"),
client_id=creds_info.get("client_id"),
client_secret=creds_info.get("client_secret"),
scopes=creds_info.get("scopes"),
)
return build("gmail", "v1", credentials=credentials, cache_discovery=False)
class GmailConnector:
source_type = "gmail"
async def list_new(self, scout) -> list[ItemRef]:
def _sync() -> tuple[list[ItemRef], str | None]:
service = _get_gmail_service(scout)
history_id = scout.gmail_history_id
refs: list[ItemRef] = []
new_history_id = history_id
if history_id:
resp = service.users().history().list(
userId="me",
startHistoryId=history_id,
historyTypes=["messageAdded"],
).execute()
for entry in resp.get("history", []):
for added in entry.get("messagesAdded", []):
refs.append(ItemRef(source_msg_ref=added["message"]["id"]))
new_history_id = resp.get("historyId", history_id)
else:
# First run: pick up the latest history id without backfilling.
profile = service.users().getProfile(userId="me").execute()
new_history_id = profile["historyId"]
return refs, new_history_id
refs, new_history_id = await asyncio.to_thread(_sync)
if new_history_id and new_history_id != scout.gmail_history_id:
scout.gmail_history_id = new_history_id
return refs
async def fetch_metadata(self, scout, ref: ItemRef) -> ItemMetadata:
def _sync() -> ItemMetadata:
service = _get_gmail_service(scout)
msg = service.users().messages().get(
userId="me", id=ref.source_msg_ref, format="metadata",
metadataHeaders=["Subject", "From", "Date"],
).execute()
headers = {h["name"]: h["value"] for h in msg.get("payload", {}).get("headers", [])}
return ItemMetadata(
subject=headers.get("Subject"),
sender=headers.get("From"),
snippet=msg.get("snippet"),
received_at=None,
)
return await asyncio.to_thread(_sync)
async def fetch_content(self, scout, ref: ItemRef) -> ItemContent:
creds_info = decrypt_token(scout.oauth_token_encrypted)
client = GmailClient(creds_info)
# Use existing fetch_messages narrowed by message id via the q parameter.
# Gmail API doesn't have a direct "get by message id with full body" via
# GmailClient's high-level interface, so we shell to the low-level call.
def _sync():
service = _get_gmail_service(scout)
msg = service.users().messages().get(
userId="me", id=ref.source_msg_ref, format="full",
).execute()
return msg
msg = await asyncio.to_thread(_sync)
from app.integrations.gmail import _extract_body, _build_email_message
# _extract_body / _build_email_message exist as helpers in gmail.py.
# If named differently in your source, find with: grep -n "def _" app/integrations/gmail.py
email_msg = _build_email_message(msg)
return ItemContent(
metadata=ItemMetadata(
subject=email_msg.subject,
sender=email_msg.sender,
snippet=msg.get("snippet"),
received_at=email_msg.date,
),
body_text=email_msg.body_text,
raw_headers={h["name"]: h["value"] for h in msg.get("payload", {}).get("headers", [])},
)
async def archive(self, scout, ref: ItemRef) -> None:
def _sync():
service = _get_gmail_service(scout)
service.users().messages().trash(userId="me", id=ref.source_msg_ref).execute()
await asyncio.to_thread(_sync)
async def setup_watch(self, scout) -> None:
def _sync():
service = _get_gmail_service(scout)
request_body = {
"labelIds": ["INBOX"],
"topicName": settings.GMAIL_PUBSUB_TOPIC,
}
resp = service.users().watch(userId="me", body=request_body).execute()
scout.gmail_history_id = resp.get("historyId")
scout.gmail_watch_expires_at = datetime.fromtimestamp(
int(resp["expiration"]) / 1000, tz=timezone.utc,
)
await asyncio.to_thread(_sync)
async def renew_watch(self, scout) -> None:
await self.setup_watch(scout)
```
If `_build_email_message` and `_extract_body` are not the actual private helper names in `app/integrations/gmail.py`, find them:
```bash
grep -n "def _" api/app/integrations/gmail.py
```
and adjust the import. The point is to reuse the existing body-extraction logic rather than duplicate it.
- [ ] **Step 4: Run tests**
```bash
cd api
pytest tests/test_scout_connectors_gmail.py -v
```
Expected: 3 PASS.
- [ ] **Step 5: Commit**
```bash
cd api
git add app/scouts/connectors/gmail.py tests/test_scout_connectors_gmail.py
git commit -m "feat(scouts): add GmailConnector"
```
---
## Task 23: Connector registration at app startup
**Files:**
- Modify: `api/app/main.py` — register `GmailConnector` during lifespan startup
- [ ] **Step 1: Read `app/main.py` lifespan**
```bash
cd api
grep -n "lifespan\|startup\|@asynccontextmanager" app/main.py
```
- [ ] **Step 2: Register connector in lifespan startup**
```python
@asynccontextmanager
async def lifespan(app: FastAPI):
from app.scouts.connectors.gmail import GmailConnector
from app.scouts.connectors.registry import register_connector
register_connector(GmailConnector())
# ... existing startup (scheduler etc.) ...
yield
# ... existing shutdown ...
```
- [ ] **Step 3: Run tests**
```bash
cd api
pytest
```
- [ ] **Step 4: Commit**
```bash
cd api
git add app/main.py
git commit -m "feat(scouts): register GmailConnector at startup"
```
---
## Task 24: Real triage LLM call + Langfuse prompt
**Files:**
- Modify: `api/app/scouts/engine.py` — replace stub `_triage_llm`
- Manual: create Langfuse prompt `scout-triage-system` with label `production`
- Modify: `api/tests/test_scout_engine.py` — relax/adjust the monkeypatched stub if needed (existing tests already monkeypatch it, so should keep working)
- [ ] **Step 1: Create the Langfuse prompt**
Use the Langfuse MCP:
```
mcp__langfuse__createTextPrompt
name: "scout-triage-system"
prompt: |
You are a triage classifier for an executive-assistant scout that watches a {{source_type}} feed.
The scout's purpose is: "{{scout_purpose}}".
Given one item, decide whether it is RELEVANT (worth surfacing to the user as a potential
task / event / note / project) or SPAM (advertising, mass marketing, phishing, bulk
notifications with no actionable content).
Item:
- Subject: {{item_subject}}
- From: {{item_sender}}
- Body (truncated): {{item_body_truncated_2k}}
Return JSON only, matching this schema:
{"verdict": "relevant" | "spam", "reason": <short string>, "confidence": <0..1>}
Be conservative on "spam" — if a message could plausibly be a personal/work email, mark it relevant.
label: "production"
```
- [ ] **Step 2: Implement real `_triage_llm` in `app/scouts/engine.py`**
Replace the `_triage_llm` stub with:
```python
async def _triage_llm(self, scout: CloudScoutConfig, content: ItemContent) -> TriageVerdict:
from app.core.llm import get_llm
from app.core.langfuse_client import get_prompt_or_fallback, get_langfuse
fallback = (
"You are a triage classifier. Given subject/sender/body, return JSON: "
"{verdict: relevant|spam, reason: str, confidence: 0..1}"
)
template, prompt_obj = get_prompt_or_fallback("scout-triage-system", fallback)
body_trunc = (content.body_text or "")[:2000]
rendered = (
template
.replace("{{source_type}}", scout.provider)
.replace("{{scout_purpose}}", scout.prompt_template or "")
.replace("{{item_subject}}", content.metadata.subject or "")
.replace("{{item_sender}}", content.metadata.sender or "")
.replace("{{item_body_truncated_2k}}", body_trunc)
)
llm = get_llm(model="gpt-4o-mini")
lf = get_langfuse()
if lf:
with lf.start_as_current_observation(as_type="generation", name="scout-triage", prompt=prompt_obj) as gen:
gen.update(input=rendered)
result = await llm.acomplete(rendered, response_format={"type": "json_object"})
gen.update(output=result.text)
else:
result = await llm.acomplete(rendered, response_format={"type": "json_object"})
import json
data = json.loads(result.text)
return TriageVerdict(**data)
```
(Adapt to the project's actual LLM helper signature. If `get_llm`/`acomplete` aren't the real names, find the LiteLLM wrapper:
```bash
grep -n "def \|class " api/app/core/llm.py
```
and use the actual API.)
- [ ] **Step 3: Run engine tests**
```bash
cd api
pytest tests/test_scout_engine.py -v
```
The existing tests monkeypatch `_triage_llm`, so they keep passing.
- [ ] **Step 4: Add an integration test that exercises the real LLM path with a mocked LLM client**
Append to `tests/test_scout_engine.py`:
```python
@pytest.mark.asyncio
async def test_triage_llm_parses_json_response(monkeypatch):
from app.scouts.engine import ScoutEngine
from app.scouts.connectors.base import ItemContent, ItemMetadata
from app.models import CloudScoutConfig
scout = CloudScoutConfig(
id=str(uuid.uuid4()), user_id="x", provider="gmail", name="t",
data_types=[], prompt_template="watch invoices", schedule_cron="0 * * * *",
enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14,
)
content = ItemContent(metadata=ItemMetadata(subject="Invoice 42", sender="acme@x"), body_text="due 2026-06-01")
fake_llm = AsyncMock()
fake_response = AsyncMock()
fake_response.text = '{"verdict": "relevant", "reason": "invoice", "confidence": 0.9}'
fake_llm.acomplete = AsyncMock(return_value=fake_response)
monkeypatch.setattr("app.scouts.engine.get_llm" if False else "app.core.llm.get_llm", lambda **k: fake_llm)
engine = ScoutEngine()
verdict = await engine._triage_llm(scout, content)
assert verdict.verdict == "relevant"
assert verdict.reason == "invoice"
```
(Adjust the monkeypatch target to match the actual import path used inside `_triage_llm`.)
- [ ] **Step 5: Run tests**
```bash
cd api
pytest tests/test_scout_engine.py -v
```
- [ ] **Step 6: Commit**
```bash
cd api
git add app/scouts/engine.py tests/test_scout_engine.py
git commit -m "feat(scouts): real triage LLM call via scout-triage-system prompt"
```
---
## Task 25: Pub/Sub webhook route + JWT verification
**Files:**
- Create: `api/app/api/routes/scout_webhooks.py`
- Create: `api/tests/test_scout_webhook.py`
- Modify: `api/app/main.py` — register the new router
- Modify: `api/app/config/settings.py` — add `GMAIL_PUBSUB_TOPIC` and `GMAIL_PUBSUB_AUDIENCE` env vars
- [ ] **Step 1: Add settings vars**
In `app/config/settings.py`:
```python
GMAIL_PUBSUB_TOPIC: str = "" # e.g. "projects/adiuvai-prod/topics/gmail-watch"
GMAIL_PUBSUB_AUDIENCE: str = "" # OIDC token audience configured in Pub/Sub push subscription
```
- [ ] **Step 2: Write the failing test**
`api/tests/test_scout_webhook.py`:
```python
"""Tests for the Gmail Pub/Sub webhook."""
from __future__ import annotations
import base64
import json
import uuid
from unittest.mock import AsyncMock, patch
import pytest
from httpx import AsyncClient
from app.main import app
from app.models import CloudScoutConfig, User
from tests.conftest import _TestSessionLocal
def _pubsub_payload(email: str, history_id: str) -> dict:
inner = json.dumps({"emailAddress": email, "historyId": history_id}).encode()
return {
"message": {"data": base64.b64encode(inner).decode(), "messageId": "m1"},
"subscription": "projects/x/subscriptions/gmail-watch-sub",
}
@pytest.mark.asyncio
async def test_webhook_triggers_scout_for_matching_user():
user_id = "00000000-0000-0000-0000-000000000003"
scout_id = str(uuid.uuid4())
async with _TestSessionLocal() as session:
user = await session.get(User, user_id)
user.email = "alice@example.com"
session.add(CloudScoutConfig(
id=scout_id, user_id=user_id, provider="gmail", name="Inbox",
data_types=[], prompt_template="", schedule_cron="0 * * * *",
enabled=True, auto_trash_spam=False, device_inactivity_pause_days=14,
))
await session.commit()
payload = _pubsub_payload("alice@example.com", "200")
with patch("app.api.routes.scout_webhooks._verify_pubsub_jwt", return_value=True), \
patch("app.scouts.engine.ScoutEngine.trigger_scout", new=AsyncMock()) as mock_trigger:
async with AsyncClient(app=app, base_url="http://test") as client:
resp = await client.post(
"/api/v1/scouts/webhooks/gmail",
json=payload,
headers={"Authorization": "Bearer fake-google-jwt"},
)
assert resp.status_code == 204
mock_trigger.assert_awaited_once_with(uuid.UUID(scout_id))
@pytest.mark.asyncio
async def test_webhook_rejects_unverified_jwt():
payload = _pubsub_payload("alice@example.com", "200")
with patch("app.api.routes.scout_webhooks._verify_pubsub_jwt", return_value=False):
async with AsyncClient(app=app, base_url="http://test") as client:
resp = await client.post(
"/api/v1/scouts/webhooks/gmail",
json=payload,
headers={"Authorization": "Bearer bogus"},
)
assert resp.status_code == 401
```
- [ ] **Step 3: Implement `app/api/routes/scout_webhooks.py`**
```python
"""Gmail Pub/Sub push receiver.
Google Pub/Sub push subscriptions deliver Gmail watch notifications as POST
requests with a JSON envelope. The body payload contains a base64-encoded
JSON blob with ``emailAddress`` + ``historyId``. We resolve the user by
email, look up their cloud_scout_configs row for provider='gmail', and
hand off to ScoutEngine.trigger_scout.
Authentication: Pub/Sub push includes an OIDC JWT in the Authorization
header. We verify it against Google's public keys with the audience
configured in our Pub/Sub subscription.
"""
from __future__ import annotations
import base64
import json
import logging
import uuid
from fastapi import APIRouter, Header, HTTPException, Request, status
from sqlalchemy import select
from app.config.settings import settings
from app.db.session import async_session_factory
from app.models import CloudScoutConfig, User
from app.scouts.engine import ScoutEngine
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/scouts/webhooks", tags=["scout-webhooks"])
def _verify_pubsub_jwt(token: str) -> bool:
"""Verify the Google Pub/Sub OIDC JWT.
Returns False on any verification failure. Production should use
google.oauth2.id_token.verify_oauth2_token with the configured audience.
"""
if not token:
return False
try:
from google.oauth2 import id_token
from google.auth.transport import requests as g_requests
id_token.verify_oauth2_token(token, g_requests.Request(), audience=settings.GMAIL_PUBSUB_AUDIENCE)
return True
except Exception:
logger.warning("pubsub jwt verification failed", exc_info=True)
return False
@router.post("/gmail", status_code=status.HTTP_204_NO_CONTENT)
async def gmail_pubsub(
request: Request,
authorization: str = Header(default=""),
):
token = authorization.removeprefix("Bearer ").strip()
if not _verify_pubsub_jwt(token):
raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Invalid Pub/Sub JWT")
body = await request.json()
msg = body.get("message") or {}
raw = msg.get("data")
if not raw:
return # ack without action
try:
decoded = json.loads(base64.b64decode(raw).decode())
except Exception:
logger.warning("pubsub payload decode failed")
return
email = decoded.get("emailAddress")
if not email:
return
async with async_session_factory() as session:
user_q = await session.execute(select(User).where(User.email == email))
user = user_q.scalar_one_or_none()
if user is None:
logger.info("pubsub: no user for %s — ignoring", email)
return
scouts_q = await session.execute(
select(CloudScoutConfig).where(
CloudScoutConfig.user_id == user.id,
CloudScoutConfig.provider == "gmail",
CloudScoutConfig.enabled == True,
)
)
scouts = scouts_q.scalars().all()
engine = ScoutEngine()
for scout in scouts:
await engine.trigger_scout(uuid.UUID(scout.id))
```
- [ ] **Step 4: Register the router in `app/main.py`**
```python
from app.api.routes import scout_webhooks
app.include_router(scout_webhooks.router, prefix="/api/v1")
```
- [ ] **Step 5: Run tests**
```bash
cd api
pytest tests/test_scout_webhook.py -v
```
Expected: 2 PASS.
- [ ] **Step 6: Commit**
```bash
cd api
git add app/api/routes/scout_webhooks.py tests/test_scout_webhook.py app/main.py app/config/settings.py
git commit -m "feat(scouts): gmail pub/sub webhook with JWT verification"
```
---
## Task 26: Cron-fallback scheduler + watch renewal in lifespan
**Files:**
- Modify: `api/app/main.py` — add `_scout_cron_tick` and `_scout_watch_renewal_tick` to existing APScheduler
- [ ] **Step 1: Read existing scheduler setup**
```bash
cd api
grep -n "AsyncIOScheduler\|apscheduler\|memory_cron_tick" app/main.py
```
- [ ] **Step 2: Add the two new ticks (gated on `settings.SCHEDULER_ENABLED`)**
Inside `lifespan`:
```python
if settings.SCHEDULER_ENABLED:
scheduler.add_job(
_scout_cron_tick, trigger="interval", minutes=15,
id="scout_cron_tick", replace_existing=True,
)
scheduler.add_job(
_scout_watch_renewal_tick, trigger="interval", hours=24,
id="scout_watch_renewal_tick", replace_existing=True,
)
```
Add the tick functions to the same file:
```python
async def _scout_cron_tick():
"""Poll cloud scouts whose schedule has elapsed since last_run_at."""
from app.scouts.engine import ScoutEngine
from app.models import CloudScoutConfig
async with async_session_factory() as session:
scouts = (await session.execute(
select(CloudScoutConfig).where(CloudScoutConfig.enabled == True)
)).scalars().all()
engine = ScoutEngine()
for scout in scouts:
# Cheap rate-limit: don't poll if we ran within the last 5 minutes.
# Push notifications are the primary trigger; this is fallback only.
if scout.last_run_at:
elapsed = (datetime.now(tz=timezone.utc) - scout.last_run_at).total_seconds()
if elapsed < 300:
continue
await engine.trigger_scout(uuid.UUID(scout.id))
async def _scout_watch_renewal_tick():
"""Re-issue Gmail users.watch for any scout whose watch expires within 24h."""
from app.scouts.connectors.registry import get_connector
from app.models import CloudScoutConfig
threshold = datetime.now(tz=timezone.utc) + timedelta(hours=24)
async with async_session_factory() as session:
scouts = (await session.execute(
select(CloudScoutConfig).where(
CloudScoutConfig.enabled == True,
CloudScoutConfig.provider == "gmail",
CloudScoutConfig.gmail_watch_expires_at <= threshold,
)
)).scalars().all()
for scout in scouts:
try:
connector = get_connector("gmail")
await connector.renew_watch(scout)
except Exception:
logger.exception("scout %s: watch renewal failed", scout.id)
await session.commit()
```
- [ ] **Step 3: Run tests**
```bash
cd api
pytest
```
- [ ] **Step 4: Commit**
```bash
cd api
git add app/main.py
git commit -m "feat(scouts): add cron-fallback poll + gmail watch renewal ticks"
```
---
## Task 27: Settings UI — "Add Gmail Scout" OAuth flow
**Files:**
- Modify: `adiuvAI/src/renderer/components/settings/CloudScoutConfigPanel.tsx` — add "Connect Gmail" button + OAuth flow
- Modify: `adiuvAI/src/main/router/index.ts` — add tRPC procedure `scout.cloud.startGmailOAuth` that opens the browser to `/api/v1/auth/oauth/google/authorize` with scout-specific scopes
- Modify: `adiuvAI/src/main/index.ts` — extend the existing `adiuvai://` protocol handler to detect `?scout_oauth_callback=1` and dispatch to scout flow instead of login flow
**Context:** The login OAuth flow already exists in `auth.py`. For scout OAuth we want a separate consent screen with `gmail.readonly` + `gmail.modify` scopes. Two approaches:
A) Reuse the existing `/auth/oauth/google/authorize` route with an extra query param `purpose=scout&scout_id=<id>`. The route branches on purpose to choose scopes and store-back behavior.
B) Add a new pair of routes `/scouts/oauth/gmail/authorize` and `/scouts/oauth/gmail/callback` mirroring the auth pattern.
Pick **B** for clarity — separates concerns. Implement the new routes alongside the existing ones.
- [ ] **Step 1: Add scout OAuth routes in `api/app/api/routes/scouts.py`**
```python
@router.get("/oauth/gmail/authorize")
async def scout_gmail_oauth_authorize(
scout_id: str,
user: User = Depends(get_current_user),
):
from app.auth.oauth_providers import GoogleOAuthProvider, generate_pkce_pair
code_verifier, code_challenge = generate_pkce_pair()
state = secrets.token_urlsafe(32)
_pending_scout_oauth_states[state] = (code_verifier, scout_id, user.id, time.time() + 600)
provider = GoogleOAuthProvider()
authorize_url = provider.build_authorize_url(
redirect_uri=f"{settings.OAUTH_REDIRECT_URI_BASE}/api/v1/scouts/oauth/gmail/web-callback",
state=state,
code_challenge=code_challenge,
scopes=["openid", "email", "https://www.googleapis.com/auth/gmail.readonly", "https://www.googleapis.com/auth/gmail.modify"],
access_type="offline",
prompt="consent",
)
return {"authorize_url": authorize_url}
@router.get("/oauth/gmail/web-callback")
async def scout_gmail_oauth_web_callback(code: str, state: str):
"""Bounces to the Electron deep link with code+state."""
return RedirectResponse(f"adiuvai://scout/oauth/gmail/callback?code={code}&state={state}")
@router.post("/oauth/gmail/callback")
async def scout_gmail_oauth_callback(
body: _ScoutGmailCallbackBody,
db: AsyncSession = Depends(get_session),
user: User = Depends(get_current_user),
):
entry = _pending_scout_oauth_states.pop(body.state, None)
if entry is None or entry[3] < time.time() or entry[2] != user.id:
raise HTTPException(401, "Invalid or expired OAuth state")
code_verifier, scout_id, _, _ = entry
provider = GoogleOAuthProvider()
token_data = await provider.exchange_code(
code=body.code,
code_verifier=code_verifier,
redirect_uri=f"{settings.OAUTH_REDIRECT_URI_BASE}/api/v1/scouts/oauth/gmail/web-callback",
scopes=["openid", "email", "https://www.googleapis.com/auth/gmail.readonly", "https://www.googleapis.com/auth/gmail.modify"],
)
creds_dict = {
"token": token_data["access_token"],
"refresh_token": token_data.get("refresh_token"),
"token_uri": "https://oauth2.googleapis.com/token",
"client_id": settings.GOOGLE_AUTH_CLIENT_ID,
"client_secret": settings.GOOGLE_AUTH_CLIENT_SECRET,
"scopes": ["https://www.googleapis.com/auth/gmail.readonly", "https://www.googleapis.com/auth/gmail.modify"],
}
encrypted = encrypt_token(creds_dict)
scout = await db.get(CloudScoutConfig, scout_id)
if scout is None or scout.user_id != user.id:
raise HTTPException(404, "Scout not found")
scout.oauth_token_encrypted = encrypted
await db.commit()
# Set up the Gmail watch so we start receiving push notifications.
from app.scouts.connectors.registry import get_connector
try:
await get_connector("gmail").setup_watch(scout)
await db.commit()
except Exception:
logger.exception("setup_watch failed for scout %s", scout_id)
return {"ok": True}
```
(`_pending_scout_oauth_states` and `_ScoutGmailCallbackBody` Pydantic model defined at the top of the file alongside the existing `_pending_states`. `encrypt_token` imported from `app.integrations`.)
- [ ] **Step 2: Add tRPC procedures `scout.cloud.startGmailOAuth` + `scout.cloud.completeGmailOAuth` in `adiuvAI/src/main/router/index.ts`**
```typescript
startGmailOAuth: publicProcedure
.input(z.object({ scoutId: z.string() }))
.mutation(async ({ input }) => {
const { authorize_url } = await getBackendClient().proxyGet<{ authorize_url: string }>(
`/api/v1/scouts/oauth/gmail/authorize?scout_id=${encodeURIComponent(input.scoutId)}`,
);
await shell.openExternal(authorize_url);
return { ok: true };
}),
completeGmailOAuth: publicProcedure
.input(z.object({ code: z.string(), state: z.string() }))
.mutation(async ({ input }) => {
return await getBackendClient().proxyPost('/api/v1/scouts/oauth/gmail/callback', input);
}),
```
- [ ] **Step 3: Extend the `adiuvai://` protocol handler in `adiuvAI/src/main/index.ts`**
Find the existing `second-instance` / `open-url` handler that parses `adiuvai://oauth/callback?...`. Add a branch for `adiuvai://scout/oauth/gmail/callback?code=...&state=...`:
```typescript
if (parsed.host === 'scout' && parsed.pathname === '/oauth/gmail/callback') {
const code = parsed.searchParams.get('code');
const state = parsed.searchParams.get('state');
if (code && state) {
// Forward to renderer via IPC so the scout settings page can call the tRPC mutation.
BrowserWindow.getAllWindows()[0]?.webContents.send('scout:gmailOAuthCallback', { code, state });
}
return;
}
```
- [ ] **Step 4: Add the consumer in `CloudScoutConfigPanel.tsx`**
```typescript
const startGmailOAuth = trpc.scout.cloud.startGmailOAuth.useMutation();
const completeGmailOAuth = trpc.scout.cloud.completeGmailOAuth.useMutation();
useEffect(() => {
const off = window.electronAI.on('scout:gmailOAuthCallback', async ({ code, state }) => {
await completeGmailOAuth.mutateAsync({ code, state });
notify('success', 'toast.scout.gmailConnected');
void utils.scout.cloud.list.invalidate();
});
return () => off?.();
}, []);
// In render, when scout.provider === 'gmail' && !scout.oauthTokenEncrypted:
<Button onClick={() => startGmailOAuth.mutate({ scoutId: scout.id })}>
{t('scouts.connectGmail')}
</Button>
```
(Adjust to actual `electronAI` API shape — check `adiuvAI/src/preload/index.ts` for the exact event subscription pattern.)
- [ ] **Step 5: Add i18n keys**
In all 5 `translation.json` files:
```json
"scouts": {
"connectGmail": "Connect Gmail",
...
},
"toast": {
"scout": {
"gmailConnected": "Gmail connected. Watching for new messages.",
...
}
}
```
(Translate per language.)
- [ ] **Step 6: Typecheck + lint + manual test**
```bash
cd adiuvAI
npx tsc --noEmit
npm run lint
```
Then manual:
```bash
npm run start
```
Open Settings > Scouts > "Add Cloud Scout" → choose Gmail → click "Connect Gmail" → consent screen opens in browser → choose account, grant scopes → browser redirects to `adiuvai://...` → app receives callback → toast confirms.
- [ ] **Step 7: Commit (split across submodules)**
```bash
cd api
git add app/api/routes/scouts.py app/config/settings.py
git commit -m "feat(scouts): add Gmail OAuth scout-setup routes"
cd ../adiuvAI
git add src/main/router/index.ts src/main/index.ts \
src/renderer/components/settings/CloudScoutConfigPanel.tsx \
src/renderer/locales/
git commit -m "feat(scouts): Gmail OAuth UI flow"
```
---
## Task 28: Submodule pointer bump for Phase 3
- [ ] **Step 1: Bump submodule pointers**
```bash
cd c:\Users\PC-Roby\Documents\_adiuvai_workspace
git add api adiuvAI
git commit -m "chore: bump submodules — Phase 3 Gmail scout end-to-end"
```
**Phase 3 complete.** End-to-end Gmail scout: OAuth → watch → push → triage → queue → on reconnect → SQLite `scout_suggestions` rows with `category='unprocessed'`.
---
## Final Acceptance Walkthrough (manual, after all tasks)
- [ ] Boot app, log in.
- [ ] Settings > Scouts: header reads "Scouts" (or localized equivalent).
- [ ] Existing local directory monitor still listed and operational.
- [ ] "Add scout" → choose Gmail → connect → consent → return to app, scout shows as enabled.
- [ ] Send yourself a test email.
- [ ] Wait ~30 seconds (Pub/Sub push) or trigger via the cron tick.
- [ ] Run from `adiuvAI/`: `sqlite3 dev.db "select id, scout_id, raw_subject, status, category from scout_suggestions"`.
- [ ] Expected: one row with `status='pending'`, `category='unprocessed'`, `raw_subject` matching the email subject.
- [ ] Run on the API DB: `psql -d adiuvai -c "select id, scout_id, status, triage_verdict from scout_triage_queue"`.
- [ ] Expected: same row, `status='acked'` (Electron acked on receipt).
- [ ] No content (body) anywhere in `scout_triage_queue` — verify with `\d scout_triage_queue`.
- [ ] Send a spam-looking email (e.g. all-caps subject, "$$$ buy now $$$" body). With `auto_trash_spam=false` (default), no row appears in either DB. Toggle `auto_trash_spam=true` on the scout, send another spam email — verify it lands in Gmail Trash and no rows appear.
---
## Plan Self-Review
**Spec coverage:**
- Phase 1 rename — Tasks 1-11. All renamed surfaces from spec table covered.
- Phase 2 connector skeleton — Tasks 12-21. Covers `scout_triage_queue` table, `cloud_scout_configs` alters, `SourceConnector` Protocol, registry, `ScoutEngine.trigger_scout` + `_process_item` + `deliver_pending` + `ack_proposal`, Drizzle `scout_suggestions` table, WS frame schemas, Electron handler.
- Phase 3 Gmail end-to-end — Tasks 22-28. `GmailConnector` impl, registration, real triage LLM, Pub/Sub webhook, cron-fallback + watch renewal scheduler, OAuth setup UI.
- Phase 4 explicitly out of scope per spec.
**Type/method consistency:**
- `ScoutTriageQueue` columns match the Alembic migration in Task 12 and the SQLAlchemy model in Task 13.
- `scoutSuggestions` Drizzle columns in Task 17 match the `IncomingScoutProposal` interface in Task 20 (snake_case in DB, camelCase in TS object).
- `_triage_llm` is a stub in Task 16 and replaced in Task 24 — explicit.
- `ScoutEngine.deliver_pending` signature `(user_id: UUID, ws)` is the same in Task 19 (definition) and Task 19 (WS endpoint usage).
- `ScoutProposalPayload` (Task 18) matches the dict the engine sends in Task 19.
**Placeholder scan:** no TBDs or "implement appropriate error handling" stubs. Where the underlying API name (e.g. `_extract_body`, `get_llm`) might differ from this codebase's actual symbols, the task gives a `grep` command to discover the real name and adapt — that's a task-time investigation, not a placeholder.
**Scope check:** Three phases, each with a clear cut-point and submodule bump. Phase 1 alone is shippable and produces working software (rename, no behavior change). Phase 2 alone is internal-only infra. Phase 3 lands the user-visible feature. All three together fit one implementation track per the approved spec.
---
## Execution Handoff
**Plan complete and saved to** `docs/superpowers/specs/2026-05-15-scouts-refactor-and-gmail-integration-plan.md`. **Two execution options:**
**1. Subagent-Driven (recommended)** — I dispatch a fresh subagent per task, review between tasks, fast iteration.
**2. Inline Execution** — Execute tasks in this session using executing-plans, batch execution with checkpoints.
**Which approach?**
Reference in New Issue View Git Blame Copy Permalink