updated plan

update refactor plan
remove unused file
2026-03-02 17:57:02 +01:00 · 2026-03-02 14:06:38 +01:00 · 2026-03-02 00:07:32 +01:00
2 changed files with 191 additions and 412 deletions
--- a/AI_REFACTOR_PLAN.md
+++ b/AI_REFACTOR_PLAN.md
@@ -1,8 +1,8 @@
 # AI Refactor Plan — Adiuva Electron App
-> **Objective:** Transform the Electron app from a single-process AI integration into a local-first multi-agent client with plugin-based batch agents, multi-provider LLM support, E2E encrypted backup, granular permissions, and cloud backend integration.
+> **Objective:** Transform the Electron app into a hybrid-first multi-agent client. The user controls where data is stored (local / cloud / sync), which AI provider to use (BYOK multi-provider), and which automations to run — either custom batch agents built with the LLM-powered Batch Builder, or pre-built plugins from the marketplace. All data access is opt-in, transparent, and auditable.
 >
-> **Backend:** Lives in a separate repository. See `BACKEND_PLAN.md` for the API contract and backend implementation guide.
+> **Backend:** Lives in a separate repository. See `../adiuva-api/BACKEND_PLAN.md` for the API contract and backend implementation guide.
 >
 > **Protocol:** Execute steps sequentially. Each step is atomic and committable. Mark `[x]` when done.
@@ -12,7 +12,7 @@
 ### Step 0.1 — Define backend API contract types
 - [ ] Create `src/shared/api-types.ts` with all interfaces the Electron app needs to communicate with the backend:
-  - `ExecutionPlan`, `PlanStep`, `PlanAction` (action types: `create_record`, `update_record`, `delete_record`, `index_document`, `send_notification`)
+  - `ExecutionPlan`, `PlanStep`, `PlanAction` (action types: `create_record`, `update_record`, `delete_record`, `index_document`, `send_notification`, `call_agent`)
  - `ChatRequest` (message, context, execution_mode: `'direct'` | `'plan'`)
  - `ChatResponse` (response, actions)
  - `ChatContext` (user_profile, relevant_documents, recent_tasks, conversation_history)
@@ -22,9 +22,25 @@
  - `BillingTier` enum (`free`, `pro`, `power`, `team`)
  - `AuthTokens` (access_token, refresh_token, expires_at)
  - `UserProfile` (id, email, tier)
 - [ ] Create `src/shared/batch-types.ts` with all types for the batch builder and storage layer:
  - `StorageTarget` — `'local'` | `'cloud'` | `'sync'` | `'none'`
  - `ConnectorType` — `'imap'` | `'filesystem'` | `'calendar'` | `'api'` | `'gmail'` | `'gdrive'` | `'outlook'`
  - `BatchActionType` — `'create_record'` | `'update_record'` | `'delete_record'` | `'index_document'` | `'send_notification'` | `'call_agent'`
  - `BatchSource` — `{ connector: ConnectorType, config: Record<string, unknown> }`
  - `BatchTrigger` — `{ type: 'cron' | 'event', schedule?: string, timezone?: string }`
  - `BatchAnalysis` — `{ prompt: string, model_override?: string, output_schema?: object }`
  - `BatchAction` — `{ type: BatchActionType, table?: string, mapping?: Record<string, string> }`
  - `BatchStorage` — `{ records: StorageTarget, vectors: StorageTarget, raw_data: StorageTarget }`
  - `BatchConfig` — full config object: `id`, `name`, `description`, `enabled`, `source`, `trigger`, `analysis`, `actions`, `storage`, `permissions`
  - `BatchStatus` — `'idle'` | `'running'` | `'error'` | `'disabled'`
  - `BatchRunResult` — `{ batchId, runAt, status, itemsProcessed, errors }`
  - `PluginListing` — `{ id, name, description, author, version, rating, installs, category, permissions, price }`
  - `InstalledPlugin` — `{ listing: PluginListing, installedAt, enabled, storageConfig: BatchStorage }`
  - `DataSourceInfo` — `{ type: ConnectorType, label, recordCount, sizeBytes, storageTarget: StorageTarget }`
  - `StorageStats` — `{ localUsedBytes, cloudUsedBytes, cloudLimitBytes, sources: DataSourceInfo[] }`
 - [ ] Update `tsconfig.json` paths if needed to include `src/shared/`
- **Files:** `src/shared/api-types.ts`, `tsconfig.json`
+- **Files:** `src/shared/api-types.ts`, `src/shared/batch-types.ts`, `tsconfig.json`
- **Outcome:** Type-safe contracts for all backend communication. Backend repo mirrors these as Pydantic schemas.
+- **Outcome:** Type-safe contracts for all backend communication and the batch/storage subsystem. Backend repo mirrors these as Pydantic schemas.
 ---
@@ -279,55 +295,173 @@
 ## Phase 6 — Renderer UI Updates
-### Step 6.1 — Update Settings page for multi-provider config
+> **Navigation model:** The app has a sidebar with top-level routes matching the pages below. Each page is a full-screen view. Shared hooks live in `src/renderer/hooks/`. All data access goes through tRPC procedures — no direct IPC calls from components.
- [ ] Add provider management UI to Settings:
+
-  - List of configured providers with status (active/inactive/error)
+### Step 6.1 — Restructure app shell and routing
 - [ ] Update `src/renderer/App.tsx`:
  - Define top-level routes: `/chat`, `/batch-builder`, `/plugins`, `/data-manager`, `/settings`, `/activity`
  - Add sidebar navigation with icons and labels for each route
  - Persist last active route in electron-store
 - [ ] Create `src/renderer/hooks/useProvider.ts`:
  - `useProvider()` — returns active provider config, `setProvider()`, `testProvider()`, list of configured providers
  - Backed by tRPC `provider.*` procedures (to be added in Phase 1)
 - [ ] Create `src/renderer/hooks/useStorage.ts`:
  - `useStorage()` — returns `StorageStats`, `setStorageTarget(source, target)`, `migrateData(source, from, to)`
  - Backed by tRPC `storage.*` procedures (to be added in Phase 2)
 - **Files:** `src/renderer/App.tsx`, `src/renderer/hooks/useProvider.ts`, `src/renderer/hooks/useStorage.ts`
 - **Outcome:** App shell with all top-level routes and shared data hooks.
 ### Step 6.2 — ChatPage with context panel
 - [ ] Create `src/renderer/pages/ChatPage.tsx`:
  - Two-column layout: chat area (left/main) + collapsible `ContextPanel` (right)
  - Wraps `ChatWindow` and `ContextPanel` components
  - Online/offline status bar at top
 - [ ] Create `src/renderer/components/chat/ChatWindow.tsx`:
  - Message list rendering `MessageBubble` for each entry
  - Input bar with send button and attachment support
  - Handles streaming tokens from `useChat` hook
  - Plan approval UI inline: expandable plan steps with approve/reject per-step
  - Error states: offline, auth expired, rate limited, server error (distinct UI for each)
 - [ ] Create `src/renderer/components/chat/MessageBubble.tsx`:
  - Renders user / assistant / system messages
  - Supports markdown rendering for assistant messages
  - Shows tool-call indicators when the agent uses a tool
  - Timestamp and copy-to-clipboard action
 - [ ] Create `src/renderer/components/chat/ContextPanel.tsx`:
  - Shows what context the agent used for the last response: matched documents, recent tasks, memory entries
  - Each context item links to its source (note, file, batch result)
  - Collapsible, persists open/closed state
 - [ ] Create `src/renderer/hooks/useChat.ts`:
  - `useChat(sessionId)` — message list, `sendMessage()`, streaming state, connection mode (`'backend'` | `'local'`)
  - Automatically falls back to local orchestrator when offline
  - Exposes `approveStep(stepId)` / `rejectStep(stepId)` for plan execution
 - **Files:** `src/renderer/pages/ChatPage.tsx`, `src/renderer/components/chat/ChatWindow.tsx`, `src/renderer/components/chat/MessageBubble.tsx`, `src/renderer/components/chat/ContextPanel.tsx`, `src/renderer/hooks/useChat.ts`
 - **Outcome:** Full chat UI with context transparency, plan approval, and seamless online/offline fallback.
 ### Step 6.3 — BatchBuilderPage
 - [ ] Create `src/renderer/pages/BatchBuilderPage.tsx`:
  - Two views: **Active Batches** list (default) and **Create New Batch** wizard
  - Active list renders `BatchCard` for each active batch config
  - "Create" button opens the wizard
 - [ ] Create `src/renderer/components/batch-builder/NaturalLanguageInput.tsx`:
  - Textarea where the user describes the batch in plain language
  - "Generate" button calls `useBatchBuilder().generate(description)`
  - Loading skeleton while the LLM generates the config
 - [ ] Create `src/renderer/components/batch-builder/ConfigPreview.tsx`:
  - Shows the generated `BatchConfig` as an editable form (not raw JSON)
  - Sections: Source, Trigger, Analysis, Actions, Storage — each collapsible
  - Inline editing for every field (prompt textarea, cron expression with human-readable label, mapping table)
  - "Edit raw JSON" toggle for power users
 - [ ] Create `src/renderer/components/batch-builder/ConnectorPicker.tsx`:
  - Dropdown of available connector types (IMAP, Filesystem, Gmail, GDrive, Outlook, Calendar, Generic API)
  - When selected, shows connector-specific config fields (e.g. IMAP: host, folder, filter_from; Filesystem: path picker)
  - OAuth connectors show "Connect account" button that opens the OAuth flow
 - [ ] Create `src/renderer/components/batch-builder/StoragePicker.tsx`:
  - Three-way toggle per storage dimension: **Local** / **Cloud** / **Sync** / **None**
  - Dimensions: Records, Vectors, Raw data
  - Shows storage impact estimate per option
  - Disabled options grayed out with tier tooltip if current tier doesn't support cloud
 - [ ] Create `src/renderer/components/batch-builder/SchedulePicker.tsx`:
  - Mode toggle: **Cron** (with human-readable label, e.g. "Every day at 08:00") / **Event** (on new data from connector)
  - Timezone selector (defaults to system timezone)
  - Visual cron builder for non-technical users (with raw cron input fallback)
 - [ ] Create `src/renderer/components/batch-builder/BatchCard.tsx`:
  - Shows batch name, connector icon, last run time, next run time, status badge (`idle` / `running` / `error` / `disabled`)
  - Actions: Run now, Edit, Disable/Enable, Delete
  - Expandable to show last run summary (items processed, errors)
 - [ ] Create `src/renderer/components/batch-builder/BatchTestRunner.tsx`:
  - "Dry Run" panel: picks one real item from the source, runs the full analysis pipeline, shows output without saving
  - Shows LLM output, action mapping preview, what would be stored and where
  - Pass/Fail indicator with detailed error on failure
 - [ ] Create `src/renderer/hooks/useBatchBuilder.ts`:
  - `useBatchBuilder()` — `generate(description): Promise<BatchConfig>`, `validate(config)`, `save(config)`, `activate(id)`, `deactivate(id)`, `runNow(id)`, `dryRun(id)`, `delete(id)`, list of saved configs with live status
  - Backed by tRPC `batch.*` procedures
 - **Files:** `src/renderer/pages/BatchBuilderPage.tsx`, `src/renderer/components/batch-builder/{NaturalLanguageInput,ConfigPreview,ConnectorPicker,StoragePicker,SchedulePicker,BatchCard,BatchTestRunner}.tsx`, `src/renderer/hooks/useBatchBuilder.ts`
 - **Outcome:** Full Batch Builder UI — users can describe a batch in natural language, review/edit the generated config, dry-run it, and activate it with a single flow.
 ### Step 6.4 — PluginStorePage
 - [ ] Create `src/renderer/pages/PluginStorePage.tsx`:
  - Two tabs: **Marketplace** (browse available plugins) and **Installed** (manage installed plugins)
  - Marketplace: search bar, category filter chips, grid of plugin cards sorted by rating/installs
  - Installed: list of `InstalledPlugin` entries with enable/disable toggles and settings links
 - [ ] Create plugin card component (inline or shared `common/`):
  - Shows name, author, description, rating (stars), install count, category badge, price/free badge
  - "Install" button → triggers permission request dialog → installs plugin
  - "Settings" button (installed) → opens plugin-specific config drawer
 - [ ] Plugin install flow:
  - On install click: fetch plugin manifest from backend
  - Show `PermissionDialog` with the permissions the plugin requires
  - On approve: call tRPC `plugins.install(id)`, download and register the plugin worker
  - Show `StoragePicker` for the plugin's data (what goes local/cloud/sync)
 - **Files:** `src/renderer/pages/PluginStorePage.tsx`
 - **Outcome:** Users can discover and install pre-built plugins from the marketplace with full permission visibility.
 ### Step 6.5 — DataManagerPage
 - [ ] Create `src/renderer/pages/DataManagerPage.tsx`:
  - Top section: `StorageOverview` dashboard
  - Below: list of `DataSourceCard` for each active data source (one card per connector/plugin)
  - "Migrate" button opens `MigrationWizard`
 - [ ] Create `src/renderer/components/data-manager/StorageOverview.tsx`:
  - Visual breakdown: local disk used vs. cloud used vs. cloud limit
  - Per-category breakdown (emails, files, notes, calendar, vectors)
  - Tier upgrade CTA if approaching cloud limit
 - [ ] Create `src/renderer/components/data-manager/DataSourceCard.tsx`:
  - Card per data source (e.g. "Gmail Scanner", "Documenti/Fatture watcher")
  - Shows record count, size, last sync time
  - Inline `StoragePicker` toggle for that source (where its data lives)
  - "Clear local cache" / "Delete all data" actions with confirmation
 - [ ] Create `src/renderer/components/data-manager/MigrationWizard.tsx`:
  - Step wizard: select source → select direction (local → cloud or cloud → local) → confirm
  - Shows estimated data size and time
  - Progress indicator during migration
  - Rolls back on error
 - **Files:** `src/renderer/pages/DataManagerPage.tsx`, `src/renderer/components/data-manager/{StorageOverview,DataSourceCard,MigrationWizard}.tsx`
 - **Outcome:** Users have full visibility and control over where every piece of their data lives.
 ### Step 6.6 — ActivityLogPage
 - [ ] Create `src/renderer/pages/ActivityLogPage.tsx`:
  - Full-page filterable table of all batch/plugin activity entries
  - Columns: timestamp, source (batch name / plugin name), action type, data accessed, storage destination, status
  - Filters: source, date range, action type, status (success/error)
  - Row expand: shows full detail — which records were created/updated, which files were read, LLM calls made
  - Export as CSV button
 - **Files:** `src/renderer/pages/ActivityLogPage.tsx`
 - **Outcome:** Complete transparency log so users can audit exactly what each agent did and when.
 ### Step 6.7 — SettingsPage (multi-provider, auth, backup, embeddings)
 - [ ] Create `src/renderer/pages/SettingsPage.tsx` with tabbed sections:
  - **AI Providers** tab:
    - List of configured providers with status badge (active / inactive / error)
    - Add provider form: name dropdown (OpenAI, Anthropic, Google, Mistral, Groq, Ollama), API key input, model selection, endpoint (for Ollama)
-  - Set primary and fallback providers
+    - Set primary provider and fallback chain
    - Test connection button per provider
- [ ] Add auth section to Settings:
+    - Separate "Embeddings provider" section: provider + model for embeddings (OpenAI, Cohere, Voyage, Mistral Embed)
-  - Login/register form
+    - Info callout: "Text sent to the embeddings provider to generate vectors — make sure you trust this provider with your data"
-  - Current tier display with upgrade CTA
+  - **Account & Billing** tab:
    - Login/register form (when not authenticated)
    - Current tier display with feature list and upgrade CTA
    - Usage indicators (batch count, cloud storage used)
    - Logout button
- [ ] Add backup section to Settings:
+  - **Backup & Sync** tab:
-  - Create/view recovery passphrase
+    - Recovery passphrase: generate new / view existing (masked, reveal on click)
-  - Manual backup trigger
+    - Manual backup trigger with last backup timestamp
-  - Backup history with restore points
+    - Auto-backup schedule toggle + interval picker
-  - Auto-backup schedule toggle
+    - Backup history table (timestamp, size, restore button)
- **Files:** `src/renderer/components/settings/` (new), route file
+  - **Permissions** tab:
- **Outcome:** Users can manage AI providers, auth, and backups from Settings.
+    - Table of all active permission grants (plugin/batch, permission type, resource, granted date)
-
+    - Revoke button per grant
-### Step 6.2 — Add Permission Dialog and Activity Log
+    - Links to ActivityLogPage for per-source audit
- [ ] Create `src/renderer/components/permissions/PermissionDialog.tsx`:
+- [ ] Create `src/renderer/components/common/ProviderSelector.tsx`:
-  - Modal shown when a plugin requests new permissions
+  - Reusable dropdown that lists configured LLM providers
-  - Lists requested permissions with reasons
+  - Used in BatchBuilder (model_override field) and Settings
-  - Per-permission approve/deny toggles
+- [ ] Create `src/renderer/components/common/PermissionDialog.tsx`:
-  - Shows plugin manifest info (name, description, version)
+  - Modal triggered when a plugin/batch requests new permissions
- [ ] Create `src/renderer/components/permissions/ActivityLog.tsx`:
+  - Lists each requested permission with its reason and resource path
-  - Filterable table of all plugin activity
+  - Per-permission approve/deny toggles (deny is default)
-  - Columns: timestamp, plugin name, action type, resource, status
+  - Shows plugin/batch manifest info (name, description, version)
-  - Filter by plugin, date range, action type
+  - "Approve selected" confirms; "Deny all" closes without granting
-  - Export as CSV
+- **Files:** `src/renderer/pages/SettingsPage.tsx`, `src/renderer/components/common/PermissionDialog.tsx`, `src/renderer/components/common/ProviderSelector.tsx`
- [ ] Add tRPC procedures for permission management and activity log queries
+- **Outcome:** Centralised settings covering providers, embeddings, auth, backup, and permissions.
 - **Files:** `src/renderer/components/permissions/PermissionDialog.tsx`, `src/renderer/components/permissions/ActivityLog.tsx`, `src/main/router/index.ts`
 - **Outcome:** Transparent permission system with full activity audit trail.
 ### Step 6.3 — Update AIChatPanel for backend-powered chat
 - [ ] Update `src/renderer/hooks/useAIChat.ts`:
  - Support WebSocket streaming from backend (when online)
  - Fall back to IPC streaming (when offline, using local orchestrator)
  - Add connection status indicator (online/offline/reconnecting)
  - Support execution plan responses: show plan preview, allow user to approve/modify before execution
 - [ ] Update `src/renderer/components/ai/AIChatPanel.tsx`:
  - Add connection status badge
  - Add tier indicator (shows current plan limitations)
  - Plan approval UI: expandable plan steps with approve/reject buttons
  - Enhanced error states: differentiate between offline, auth expired, rate limited, server error
 - [ ] Update `src/renderer/components/ai/FloatingChat.tsx`:
  - Same streaming changes as AIChatPanel
  - Compact plan approval for inline context
 - **Files:** `src/renderer/hooks/useAIChat.ts`, `src/renderer/components/ai/AIChatPanel.tsx`, `src/renderer/components/ai/FloatingChat.tsx`
 - **Outcome:** Chat UI seamlessly handles both online (backend) and offline (local) modes.
 ---
@@ -375,8 +509,10 @@
 | `argon2` | Key derivation for E2E backup |
 | `node-cron` | Batch agent scheduling |
 | `chokidar` | File watching (FileWatcher plugin) |
-| `imapflow` | IMAP client (EmailScanner plugin) |
+| `imapflow` | IMAP client (IMAP connector) |
-| `onnxruntime-node` | Local embeddings (optional) |
+| `googleapis` | Gmail + GDrive OAuth connectors |
 | `lancedb` | Local vector store |
 | `onnxruntime-node` | Local embeddings (optional, future) |
 ---
@@ -387,3 +523,4 @@
 - **Phase 3** (backend integration) requires the backend repo to have the `/api/v1/chat` endpoint ready.
 - **Phase 5.2** (SQLCipher) is intentionally late to avoid encryption overhead during active schema changes.
 - **The existing app continues to work** throughout the migration. Local orchestration is preserved until backend is ready (Step 3.2).
 - **One step at a time.** Implement one numbered step per session. When the step is fully done, mark all its checkboxes as `[x]` in this file and commit with message `step N complete: <outcome line>`.
--- a/BACKEND_PLAN.md
+++ b/BACKEND_PLAN.md
@@ -1,358 +0,0 @@
 # Backend Plan — Adiuva Cloud API
 > **Separate repository.** This document defines the FastAPI backend that the Electron app communicates with.
 >
 > The backend owns: orchestration logic, chat agent intelligence, prompt IP, auth, billing, and backup blob storage.
 > The backend NEVER persists user data. It receives context in requests, uses it for orchestration, and discards it.
 ---
 ## Project Structure
 ```
 adiuva-backend/
 ├── app/
 │   ├── __init__.py
 │   ├── main.py                    # FastAPI entry + CORS + lifespan + router includes
 │   ├── core/
 │   │   ├── __init__.py
 │   │   ├── agent_registry.py      # Base classes + singleton registry
 │   │   ├── orchestrator.py        # LLM-based intent router
 │   │   ├── execution_plan.py      # Plan builder + cache
 │   │   └── plugin_loader.py       # Dynamic agent loading
 │   ├── agents/
 │   │   ├── __init__.py            # Auto-registers all agents
 │   │   ├── task_agent.py
 │   │   ├── calendar_agent.py
 │   │   ├── email_agent.py
 │   │   └── analytics_agent.py
 │   ├── api/
 │   │   ├── __init__.py
 │   │   ├── routes/
 │   │   │   ├── __init__.py
 │   │   │   ├── chat.py            # POST /chat + WS /chat/stream
 │   │   │   ├── plans.py           # GET /plans/playbook
 │   │   │   ├── backup.py          # PUT/GET /backup
 │   │   │   ├── auth.py            # Register/login/refresh
 │   │   │   └── billing.py         # Checkout/webhook/subscription
 │   │   └── middleware/
 │   │       ├── __init__.py
 │   │       ├── auth.py            # JWT validation
 │   │       ├── rate_limit.py      # Tier-aware rate limiting
 │   │       └── sanitizer.py       # Strip prompt metadata from responses
 │   ├── billing/
 │   │   ├── __init__.py
 │   │   ├── stripe_service.py      # Stripe checkout + webhooks
 │   │   └── tier_manager.py        # Feature matrix per tier
 │   └── config/
 │       ├── __init__.py
 │       └── settings.py            # Pydantic BaseSettings (env-based)
 ├── tests/
 │   ├── __init__.py
 │   ├── conftest.py                # Fixtures: test client, mock agents, mock LLM
 │   ├── test_orchestrator.py
 │   ├── test_agents.py
 │   ├── test_auth.py
 │   └── test_backup.py
 ├── alembic/                       # DB migrations (auth/billing tables only)
 │   ├── alembic.ini
 │   └── versions/
 ├── requirements.txt
 ├── Dockerfile
 ├── docker-compose.yml             # App + PostgreSQL + Redis (dev)
 ├── .env.example
 └── README.md
 ```
 ---
 ## Step-by-Step Implementation
 ### Step 1 — Project scaffolding
 - [ ] Initialize repo with the directory structure above
 - [ ] Write `requirements.txt`:
  ```
  fastapi>=0.115.0
  uvicorn[standard]>=0.34.0
  langchain>=0.3.0
  langchain-openai>=0.3.0
  pydantic>=2.10.0
  python-jose[cryptography]>=3.3.0
  stripe>=11.0.0
  boto3>=1.35.0
  slowapi>=0.1.9
  sqlalchemy>=2.0.0
  asyncpg>=0.30.0
  alembic>=1.14.0
  bcrypt>=4.2.0
  python-dotenv>=1.0.0
  httpx>=0.28.0
  websockets>=14.0
  pytest>=8.0.0
  pytest-asyncio>=0.24.0
  ```
 - [ ] Write `app/main.py`: FastAPI app with CORS (allow `app://`, `http://localhost:*`), lifespan (init DB pool, init agent registry), include all routers under `/api/v1`
 - [ ] Write `app/config/settings.py`: `Settings(BaseSettings)` with fields: `DATABASE_URL`, `JWT_SECRET`, `JWT_ALGORITHM` (default HS256), `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, `S3_BUCKET`, `S3_REGION`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `OPENAI_API_KEY`, `CORS_ORIGINS`, `ENV` (dev/prod)
 - [ ] Write `Dockerfile`: Python 3.12 slim, multi-stage (builder + runtime), non-root user
 - [ ] Write `docker-compose.yml`: app, postgres:16, optional redis
 - [ ] Write `.env.example`
 - **Outcome:** Runnable FastAPI skeleton (returns 404 on all routes).
 ### Step 2 — Pydantic schemas (API contracts)
 - [ ] Create `app/schemas.py` (mirrors `src/shared/api-types.ts` from Electron repo):
  - `ChatRequest`: `message: str`, `context: ChatContext`, `execution_mode: Literal['direct', 'plan']`
  - `ChatContext`: `user_profile: dict`, `relevant_documents: list[str]`, `recent_tasks: list[dict]`, `conversation_history: list[dict]`
  - `ChatResponse`: `response: str`, `actions: list[PlanAction]`
  - `PlanAction`: `type: Literal['create_record', 'update_record', 'delete_record', 'index_document', 'send_notification']`, `table: str | None`, `data: dict | None`
  - `ExecutionPlan`: `agent: str`, `steps: list[PlanStep]`
  - `PlanStep`: `action: str`, `prompt_template: str | None`, `variables: dict | None`, `data_from_step: int | None`
  - `BackupMetadata`: `version: int`, `timestamp: int`, `checksum: str`, `chunk_count: int`
  - `BillingTier`: `Literal['free', 'pro', 'power', 'team']`
  - `AuthTokens`: `access_token: str`, `refresh_token: str`, `expires_at: int`
  - `UserProfile`: `id: str`, `email: str`, `tier: BillingTier`
 - **Outcome:** All request/response models defined and validated.
 ### Step 3 — Agent Registry + base classes
 - [ ] `app/core/agent_registry.py`:
  - `BaseAgent(ABC)`:
    - `user_id: str`, `shared_memory: dict`, `vector_store_context: list[str]`, `skills: list[str]`
    - Abstract `get_name() -> str`, `get_description() -> str`
  - `ChatAgent(BaseAgent)`:
    - Abstract `async handle(query: str, context: dict) -> str`
    - Abstract `get_tools() -> list` (LangChain tool definitions)
    - Concrete `_tool_loop(llm, messages, tools, max_iter=5) -> str` — shared tool-calling loop
  - `AgentRegistry` (singleton):
    - `_agents: dict[str, ChatAgent]`
    - `register(agent_class)` — decorator pattern
    - `get(name) -> ChatAgent`
    - `list_agents() -> list[dict]` — returns `[{name, description}]` for orchestrator prompt
    - `async call_agent(name, query, context) -> str` — for inter-agent calls
 - [ ] Unit tests: register, get, list, call_agent with mock
 - **Outcome:** Pluggable agent framework.
 ### Step 4 — Orchestrator
 - [ ] `app/core/orchestrator.py`:
  - `async classify_intent(message, context, registry) -> str`:
    - System prompt: "You are an intent classifier. Given the user message and context, decide which agent to route to. Available agents: {registry.list_agents()}. Respond with just the agent name."
    - Uses gpt-4o-mini via LangChain for low latency
    - Falls back to `task_agent` if no clear match
  - `async route_single(agent_name, message, context) -> ChatResponse`:
    - Instantiates agent from registry
    - Calls `agent.handle(message, context)`
    - Returns response + any actions the agent produced
  - `async route_pipeline(agent_names, message, context) -> ChatResponse`:
    - Executes agents in sequence
    - Each agent receives `{...context, previous_results: [...]}`
    - Final synthesis via LLM: "Summarize these agent results into a coherent response"
  - `async orchestrate(request: ChatRequest) -> ChatResponse | ExecutionPlan`:
    - Main entry point
    - Classifies intent
    - If `execution_mode == 'direct'`: route + return response
    - If `execution_mode == 'plan'`: route + return execution plan with template IDs
  - `async orchestrate_stream(request: ChatRequest) -> AsyncGenerator[str, None]`:
    - Same as orchestrate but yields tokens for WebSocket streaming
 - [ ] Integration tests with mocked LLM and mocked agents
 - **Outcome:** Intelligent routing with single-agent and pipeline modes.
 ### Step 5 — Execution Plan generator
 - [ ] `app/core/execution_plan.py`:
  - `PromptTemplateRegistry`: dict of `template_id -> prompt_text`. Templates are server-side only — client receives IDs.
  - `ExecutionPlanBuilder`:
    - `add_step(action, params) -> self`
    - `add_llm_step(template_id, variables) -> self`
    - `add_data_step(action, data_from_step) -> self`
    - `build() -> ExecutionPlan` — validates step references
  - `PlanCache`:
    - In-memory LRU (maxsize=1000)
    - `cache_plan(key, plan)`, `get_plan(key)`, `get_all_playbooks() -> list[ExecutionPlan]`
    - Playbooks are pre-built plans for common operations (e.g., "create task from email", "generate weekly report")
 - **Outcome:** Plans are cacheable as playbooks. Prompt IP never leaves the server.
 ### Step 6 — Chat Agents
 - [ ] `app/agents/task_agent.py` — `@registry.register`:
  - Description: "Manages tasks: create, update, list, suggest"
  - Tools: `create_task(title, description, priority, due_date)`, `update_task(id, updates)`, `list_tasks(filters)`, `suggest_tasks(notes_context)`
  - System prompt: PM-oriented, validates task structure, infers priority from context
  - `handle()`: LLM + tool loop via `_tool_loop()`, returns response text + list of actions performed
 - [ ] `app/agents/calendar_agent.py` — `@registry.register`:
  - Description: "Calendar management: events, conflicts, scheduling"
  - Tools: `list_events(date_range)`, `detect_conflicts(events)`, `suggest_reschedule(conflict)`
  - Works with event metadata passed in context (never raw calendar data stored)
 - [ ] `app/agents/email_agent.py` — `@registry.register`:
  - Description: "Email analysis: classify, extract actions, draft responses"
  - Tools: `classify_email(metadata)`, `extract_action_items(metadata)`, `draft_response(thread_context)`
  - Only processes metadata sent by client — never raw email bodies
 - [ ] `app/agents/analytics_agent.py` — `@registry.register`:
  - Description: "Workspace analytics: metrics, reports, trends"
  - Tools: `calculate_metrics(task_data)`, `generate_report(period, data)`, `trend_analysis(data_points)`
  - Crunches numbers from context, returns structured insights
 - [ ] `app/agents/__init__.py`: imports all agent modules to trigger `@registry.register` decorators
 - [ ] Unit tests per agent with mocked LLM
 - **Outcome:** Four specialized agents, all registered and tested.
 ### Step 7 — API Routes
 #### 7a — Chat endpoint
 - [ ] `app/api/routes/chat.py`:
  - `POST /api/v1/chat`:
    - Request: `ChatRequest`
    - Calls `orchestrate(request)` or `orchestrate()` + `build_plan()`
    - Response: `ChatResponse` or `ExecutionPlan`
  - `WebSocket /api/v1/chat/stream`:
    - Client sends `ChatRequest` as first JSON frame
    - Server yields token strings via `orchestrate_stream()`
    - Final frame: JSON `ChatResponse` with `{"done": true, "response": "...", "actions": [...]}`
    - Heartbeat ping every 30s to keep connection alive
 #### 7b — Plans endpoint
 - [ ] `app/api/routes/plans.py`:
  - `GET /api/v1/plans/playbook`: Returns all playbooks available for the user's tier
  - `GET /api/v1/plans/playbook/{plan_id}`: Returns a specific plan
 #### 7c — Backup endpoint
 - [ ] `app/api/routes/backup.py`:
  - `PUT /api/v1/backup`: Accepts binary blob + metadata headers (`X-Backup-Version`, `X-Backup-Timestamp`, `X-Backup-Checksum`). Stores in S3 keyed by `{user_id}/{timestamp}`. Enforces tier limits:
    - Free: 0 (no backup)
    - Pro: 5 GB
    - Power: 50 GB
    - Team: unlimited
  - `GET /api/v1/backup`: Returns latest blob for authenticated user. Supports `If-Modified-Since`.
  - `GET /api/v1/backup/history`: Returns list of `BackupMetadata` (no blobs).
  - `DELETE /api/v1/backup/{backup_id}`: Delete specific backup.
 #### 7d — Auth endpoint
 - [ ] `app/api/routes/auth.py`:
  - `POST /api/v1/auth/register`: `{email, password}` → bcrypt hash → insert user → return `AuthTokens`
  - `POST /api/v1/auth/login`: Validate credentials → return `AuthTokens`
  - `POST /api/v1/auth/refresh`: Rotate refresh token → return new `AuthTokens`
  - `GET /api/v1/auth/me`: Return `UserProfile` for current JWT
 #### 7e — Billing endpoint
 - [ ] `app/api/routes/billing.py`:
  - `POST /api/v1/billing/checkout`: Creates Stripe checkout session → returns URL
  - `POST /api/v1/billing/webhook`: Handles Stripe webhooks (subscription lifecycle)
  - `GET /api/v1/billing/subscription`: Returns current subscription info
  - `DELETE /api/v1/billing/subscription`: Cancels subscription
 - **Outcome:** Complete REST + WebSocket API.
 ### Step 8 — Middleware
 #### 8a — Auth middleware
 - [ ] `app/api/middleware/auth.py`:
  - FastAPI dependency: `get_current_user(token: str = Depends(oauth2_scheme)) -> UserProfile`
  - Validates JWT signature, expiry, extracts `user_id` and `tier`
  - Raises `401` on invalid/expired token
  - Exempt routes: `/api/v1/auth/register`, `/api/v1/auth/login`, `/api/v1/billing/webhook`
 #### 8b — Rate limiter
 - [ ] `app/api/middleware/rate_limit.py`:
  - Uses `slowapi` with `Limiter(key_func=get_user_id_from_jwt)`
  - Tier-based limits:
    - Free: 20 req/min
    - Pro: 60 req/min
    - Power: 120 req/min
    - Team: 200 req/seat/min
  - Custom 429 response with `Retry-After` header
 #### 8c — Sanitizer
 - [ ] `app/api/middleware/sanitizer.py`:
  - Response middleware that scans response bodies
  - Strips: system prompt fragments, agent internal reasoning, tool schemas, routing metadata
  - Pattern-based detection + exact match against known prompt fingerprints
  - Logs sanitization events for monitoring
 - **Outcome:** Secure, rate-limited API with prompt IP protection.
 ### Step 9 — Billing & Tier management
 - [ ] `app/billing/stripe_service.py`:
  - `create_checkout_session(user_id, tier) -> str`
  - `handle_webhook(payload, sig_header) -> None`: processes `checkout.session.completed`, `customer.subscription.updated`, `customer.subscription.deleted`, `invoice.payment_failed`
  - `get_subscription(user_id) -> dict | None`
  - `cancel_subscription(user_id) -> None`
 - [ ] `app/billing/tier_manager.py`:
  - `TierManager`:
    - Feature matrix:
      ```python
      FEATURES = {
          'free':  {'agents': 3, 'batch': False, 'providers': 1, 'backup_gb': 0},
          'pro':   {'agents': -1, 'batch': True, 'providers': -1, 'backup_gb': 5},
          'power': {'agents': -1, 'batch': True, 'providers': -1, 'backup_gb': 50, 'byok': True},
          'team':  {'agents': -1, 'batch': True, 'providers': -1, 'backup_gb': -1, 'sso': True},
      }
      ```
    - `get_tier(user_id) -> BillingTier`
    - `check_feature(user_id, feature) -> bool`
    - `get_rate_limit(tier) -> int`
 - **Outcome:** Stripe integration with tier-based feature gating.
 ### Step 10 — Database (auth/billing only)
 - [ ] PostgreSQL schema via Alembic:
  - `users`: `id UUID PK`, `email UNIQUE`, `password_hash`, `tier` (default 'free'), `stripe_customer_id`, `created_at`, `updated_at`
  - `refresh_tokens`: `id UUID PK`, `user_id FK`, `token_hash`, `expires_at`, `created_at`
  - `subscriptions`: `id UUID PK`, `user_id FK`, `stripe_subscription_id`, `tier`, `status`, `current_period_end`, `created_at`
  - `backup_metadata`: `id UUID PK`, `user_id FK`, `s3_key`, `version`, `timestamp`, `checksum`, `size_bytes`, `created_at`
 - [ ] Initial Alembic migration
 - [ ] SQLAlchemy models in `app/models.py`
 - **Outcome:** Auth and billing persistence. Zero user data stored.
 ### Step 11 — Testing & deployment
 - [ ] `tests/conftest.py`: TestClient fixture, mock LLM fixture (`AsyncMock` returning canned responses), mock agent fixture, test DB (SQLite in-memory for speed)
 - [ ] `tests/test_orchestrator.py`: classify_intent routing, single agent, pipeline, plan mode
 - [ ] `tests/test_agents.py`: each agent with mocked tools
 - [ ] `tests/test_auth.py`: register → login → access protected → refresh → expired token
 - [ ] `tests/test_backup.py`: upload → download → history → delete, tier limit enforcement
 - [ ] `Dockerfile` optimized for production (gunicorn + uvicorn workers)
 - [ ] GitHub Actions CI: lint (ruff), test (pytest), build Docker image
 - **Outcome:** Fully tested, deployable backend.
 ---
 ## API Contract Summary
 | Method | Endpoint | Auth | Request | Response |
 |--------|----------|------|---------|----------|
 | POST | `/api/v1/auth/register` | No | `{email, password}` | `AuthTokens` |
 | POST | `/api/v1/auth/login` | No | `{email, password}` | `AuthTokens` |
 | POST | `/api/v1/auth/refresh` | No | `{refresh_token}` | `AuthTokens` |
 | GET | `/api/v1/auth/me` | JWT | — | `UserProfile` |
 | POST | `/api/v1/chat` | JWT | `ChatRequest` | `ChatResponse \| ExecutionPlan` |
 | WS | `/api/v1/chat/stream` | JWT | `ChatRequest` (first frame) | Token stream + final JSON |
 | GET | `/api/v1/plans/playbook` | JWT | — | `ExecutionPlan[]` |
 | GET | `/api/v1/plans/playbook/:id` | JWT | — | `ExecutionPlan` |
 | PUT | `/api/v1/backup` | JWT | Binary blob + headers | `{ok: true}` |
 | GET | `/api/v1/backup` | JWT | — | Binary blob |
 | GET | `/api/v1/backup/history` | JWT | — | `BackupMetadata[]` |
 | DELETE | `/api/v1/backup/:id` | JWT | — | `{ok: true}` |
 | POST | `/api/v1/billing/checkout` | JWT | `{tier}` | `{checkout_url}` |
 | POST | `/api/v1/billing/webhook` | Stripe sig | Stripe event | `{ok: true}` |
 | GET | `/api/v1/billing/subscription` | JWT | — | Subscription info |
 | DELETE | `/api/v1/billing/subscription` | JWT | — | `{ok: true}` |
 | GET | `/api/v1/health` | No | — | `{status, version}` |
 ---
 ## Stack
 | Layer | Technology |
 |-------|-----------|
 | Framework | FastAPI + Uvicorn |
 | LLM | LangChain + langchain-openai |
 | Auth | PyJWT + bcrypt + OAuth2 |
 | Billing | stripe-python |
 | Storage | boto3 (S3) |
 | Database | PostgreSQL + SQLAlchemy + Alembic |
 | Rate limiting | slowapi |
 | Testing | pytest + pytest-asyncio + httpx |
 | Deployment | Docker → fly.io / Railway / AWS ECS |
 ---
 ## Development Rules
 1. **NEVER persist user data.** The DB stores only auth, billing, and backup metadata. User context arrives in requests and is discarded after processing.
 2. **NEVER expose prompts.** System prompts are composed server-side from fragments. Responses are sanitized before sending.
 3. **Stateless request handling.** No server-side session state. All context comes from the client + JWT.
 4. **Type hints everywhere.** All functions have full type annotations.
 5. **Test every agent.** Each chat agent has unit tests with mocked LLM responses.
 6. **Structured logging.** JSON logs with request ID correlation.
Author	SHA1	Message	Date
roberto	b77c6d1195	updated plan	2026-03-02 17:57:02 +01:00
roberto	489e8e3bc9	update refactor plan	2026-03-02 14:06:38 +01:00
roberto	1ba9c9eee2	remove unused file	2026-03-02 00:07:32 +01:00