update refactor plan

AI_REFACTOR_PLAN.md

@@ -1,6 +1,6 @@
 # AI Refactor Plan — Adiuva Electron App
 
-> **Objective:** Transform the Electron app from a single-process AI integration into a local-first multi-agent client with plugin-based batch agents, multi-provider LLM support, E2E encrypted backup, granular permissions, and cloud backend integration.
+> **Objective:** Transform the Electron app into a hybrid-first multi-agent client. The user controls where data is stored (local / cloud / sync), which AI provider to use (BYOK multi-provider), and which automations to run — either custom batch agents built with the LLM-powered Batch Builder, or pre-built plugins from the marketplace. All data access is opt-in, transparent, and auditable.
 >
 > **Backend:** Lives in a separate repository. See `BACKEND_PLAN.md` for the API contract and backend implementation guide.
 >
@@ -12,7 +12,7 @@
 
 ### Step 0.1 — Define backend API contract types
 - [ ] Create `src/shared/api-types.ts` with all interfaces the Electron app needs to communicate with the backend:
-  - `ExecutionPlan`, `PlanStep`, `PlanAction` (action types: `create_record`, `update_record`, `delete_record`, `index_document`, `send_notification`)
+  - `ExecutionPlan`, `PlanStep`, `PlanAction` (action types: `create_record`, `update_record`, `delete_record`, `index_document`, `send_notification`, `call_agent`)
   - `ChatRequest` (message, context, execution_mode: `'direct'` | `'plan'`)
   - `ChatResponse` (response, actions)
   - `ChatContext` (user_profile, relevant_documents, recent_tasks, conversation_history)
@@ -22,9 +22,25 @@
   - `BillingTier` enum (`free`, `pro`, `power`, `team`)
   - `AuthTokens` (access_token, refresh_token, expires_at)
   - `UserProfile` (id, email, tier)
+- [ ] Create `src/shared/batch-types.ts` with all types for the batch builder and storage layer:
+  - `StorageTarget` — `'local'` | `'cloud'` | `'sync'` | `'none'`
+  - `ConnectorType` — `'imap'` | `'filesystem'` | `'calendar'` | `'api'` | `'gmail'` | `'gdrive'` | `'outlook'`
+  - `BatchActionType` — `'create_record'` | `'update_record'` | `'delete_record'` | `'index_document'` | `'send_notification'` | `'call_agent'`
+  - `BatchSource` — `{ connector: ConnectorType, config: Record<string, unknown> }`
+  - `BatchTrigger` — `{ type: 'cron' | 'event', schedule?: string, timezone?: string }`
+  - `BatchAnalysis` — `{ prompt: string, model_override?: string, output_schema?: object }`
+  - `BatchAction` — `{ type: BatchActionType, table?: string, mapping?: Record<string, string> }`
+  - `BatchStorage` — `{ records: StorageTarget, vectors: StorageTarget, raw_data: StorageTarget }`
+  - `BatchConfig` — full config object: `id`, `name`, `description`, `enabled`, `source`, `trigger`, `analysis`, `actions`, `storage`, `permissions`
+  - `BatchStatus` — `'idle'` | `'running'` | `'error'` | `'disabled'`
+  - `BatchRunResult` — `{ batchId, runAt, status, itemsProcessed, errors }`
+  - `PluginListing` — `{ id, name, description, author, version, rating, installs, category, permissions, price }`
+  - `InstalledPlugin` — `{ listing: PluginListing, installedAt, enabled, storageConfig: BatchStorage }`
+  - `DataSourceInfo` — `{ type: ConnectorType, label, recordCount, sizeBytes, storageTarget: StorageTarget }`
+  - `StorageStats` — `{ localUsedBytes, cloudUsedBytes, cloudLimitBytes, sources: DataSourceInfo[] }`
 - [ ] Update `tsconfig.json` paths if needed to include `src/shared/`
-- **Files:** `src/shared/api-types.ts`, `tsconfig.json`
+- **Files:** `src/shared/api-types.ts`, `src/shared/batch-types.ts`, `tsconfig.json`
-- **Outcome:** Type-safe contracts for all backend communication. Backend repo mirrors these as Pydantic schemas.
+- **Outcome:** Type-safe contracts for all backend communication and the batch/storage subsystem. Backend repo mirrors these as Pydantic schemas.
 
 ---
 
@@ -279,55 +295,173 @@
 
 ## Phase 6 — Renderer UI Updates
 
-### Step 6.1 — Update Settings page for multi-provider config
+> **Navigation model:** The app has a sidebar with top-level routes matching the pages below. Each page is a full-screen view. Shared hooks live in `src/renderer/hooks/`. All data access goes through tRPC procedures — no direct IPC calls from components.
-- [ ] Add provider management UI to Settings:
-  - List of configured providers with status (active/inactive/error)
-  - Add provider form: name dropdown (OpenAI, Anthropic, Google, Mistral, Groq, Ollama), API key input, model selection, endpoint (for Ollama)
-  - Set primary and fallback providers
-  - Test connection button per provider
-- [ ] Add auth section to Settings:
-  - Login/register form
-  - Current tier display with upgrade CTA
-  - Logout button
-- [ ] Add backup section to Settings:
-  - Create/view recovery passphrase
-  - Manual backup trigger
-  - Backup history with restore points
-  - Auto-backup schedule toggle
-- **Files:** `src/renderer/components/settings/` (new), route file
-- **Outcome:** Users can manage AI providers, auth, and backups from Settings.
 
-### Step 6.2 — Add Permission Dialog and Activity Log
+### Step 6.1 — Restructure app shell and routing
-- [ ] Create `src/renderer/components/permissions/PermissionDialog.tsx`:
+- [ ] Update `src/renderer/App.tsx`:
-  - Modal shown when a plugin requests new permissions
+  - Define top-level routes: `/chat`, `/batch-builder`, `/plugins`, `/data-manager`, `/settings`, `/activity`
-  - Lists requested permissions with reasons
+  - Add sidebar navigation with icons and labels for each route
-  - Per-permission approve/deny toggles
+  - Persist last active route in electron-store
-  - Shows plugin manifest info (name, description, version)
+- [ ] Create `src/renderer/hooks/useProvider.ts`:
-- [ ] Create `src/renderer/components/permissions/ActivityLog.tsx`:
+  - `useProvider()` — returns active provider config, `setProvider()`, `testProvider()`, list of configured providers
-  - Filterable table of all plugin activity
+  - Backed by tRPC `provider.*` procedures (to be added in Phase 1)
-  - Columns: timestamp, plugin name, action type, resource, status
+- [ ] Create `src/renderer/hooks/useStorage.ts`:
-  - Filter by plugin, date range, action type
+  - `useStorage()` — returns `StorageStats`, `setStorageTarget(source, target)`, `migrateData(source, from, to)`
-  - Export as CSV
+  - Backed by tRPC `storage.*` procedures (to be added in Phase 2)
-- [ ] Add tRPC procedures for permission management and activity log queries
+- **Files:** `src/renderer/App.tsx`, `src/renderer/hooks/useProvider.ts`, `src/renderer/hooks/useStorage.ts`
-- **Files:** `src/renderer/components/permissions/PermissionDialog.tsx`, `src/renderer/components/permissions/ActivityLog.tsx`, `src/main/router/index.ts`
+- **Outcome:** App shell with all top-level routes and shared data hooks.
-- **Outcome:** Transparent permission system with full activity audit trail.
 
-### Step 6.3 — Update AIChatPanel for backend-powered chat
+### Step 6.2 — ChatPage with context panel
-- [ ] Update `src/renderer/hooks/useAIChat.ts`:
+- [ ] Create `src/renderer/pages/ChatPage.tsx`:
-  - Support WebSocket streaming from backend (when online)
+  - Two-column layout: chat area (left/main) + collapsible `ContextPanel` (right)
-  - Fall back to IPC streaming (when offline, using local orchestrator)
+  - Wraps `ChatWindow` and `ContextPanel` components
-  - Add connection status indicator (online/offline/reconnecting)
+  - Online/offline status bar at top
-  - Support execution plan responses: show plan preview, allow user to approve/modify before execution
+- [ ] Create `src/renderer/components/chat/ChatWindow.tsx`:
-- [ ] Update `src/renderer/components/ai/AIChatPanel.tsx`:
+  - Message list rendering `MessageBubble` for each entry
-  - Add connection status badge
+  - Input bar with send button and attachment support
-  - Add tier indicator (shows current plan limitations)
+  - Handles streaming tokens from `useChat` hook
-  - Plan approval UI: expandable plan steps with approve/reject buttons
+  - Plan approval UI inline: expandable plan steps with approve/reject per-step
-  - Enhanced error states: differentiate between offline, auth expired, rate limited, server error
+  - Error states: offline, auth expired, rate limited, server error (distinct UI for each)
-- [ ] Update `src/renderer/components/ai/FloatingChat.tsx`:
+- [ ] Create `src/renderer/components/chat/MessageBubble.tsx`:
-  - Same streaming changes as AIChatPanel
+  - Renders user / assistant / system messages
-  - Compact plan approval for inline context
+  - Supports markdown rendering for assistant messages
-- **Files:** `src/renderer/hooks/useAIChat.ts`, `src/renderer/components/ai/AIChatPanel.tsx`, `src/renderer/components/ai/FloatingChat.tsx`
+  - Shows tool-call indicators when the agent uses a tool
-- **Outcome:** Chat UI seamlessly handles both online (backend) and offline (local) modes.
+  - Timestamp and copy-to-clipboard action
+- [ ] Create `src/renderer/components/chat/ContextPanel.tsx`:
+  - Shows what context the agent used for the last response: matched documents, recent tasks, memory entries
+  - Each context item links to its source (note, file, batch result)
+  - Collapsible, persists open/closed state
+- [ ] Create `src/renderer/hooks/useChat.ts`:
+  - `useChat(sessionId)` — message list, `sendMessage()`, streaming state, connection mode (`'backend'` | `'local'`)
+  - Automatically falls back to local orchestrator when offline
+  - Exposes `approveStep(stepId)` / `rejectStep(stepId)` for plan execution
+- **Files:** `src/renderer/pages/ChatPage.tsx`, `src/renderer/components/chat/ChatWindow.tsx`, `src/renderer/components/chat/MessageBubble.tsx`, `src/renderer/components/chat/ContextPanel.tsx`, `src/renderer/hooks/useChat.ts`
+- **Outcome:** Full chat UI with context transparency, plan approval, and seamless online/offline fallback.
+
+### Step 6.3 — BatchBuilderPage
+- [ ] Create `src/renderer/pages/BatchBuilderPage.tsx`:
+  - Two views: **Active Batches** list (default) and **Create New Batch** wizard
+  - Active list renders `BatchCard` for each active batch config
+  - "Create" button opens the wizard
+- [ ] Create `src/renderer/components/batch-builder/NaturalLanguageInput.tsx`:
+  - Textarea where the user describes the batch in plain language
+  - "Generate" button calls `useBatchBuilder().generate(description)`
+  - Loading skeleton while the LLM generates the config
+- [ ] Create `src/renderer/components/batch-builder/ConfigPreview.tsx`:
+  - Shows the generated `BatchConfig` as an editable form (not raw JSON)
+  - Sections: Source, Trigger, Analysis, Actions, Storage — each collapsible
+  - Inline editing for every field (prompt textarea, cron expression with human-readable label, mapping table)
+  - "Edit raw JSON" toggle for power users
+- [ ] Create `src/renderer/components/batch-builder/ConnectorPicker.tsx`:
+  - Dropdown of available connector types (IMAP, Filesystem, Gmail, GDrive, Outlook, Calendar, Generic API)
+  - When selected, shows connector-specific config fields (e.g. IMAP: host, folder, filter_from; Filesystem: path picker)
+  - OAuth connectors show "Connect account" button that opens the OAuth flow
+- [ ] Create `src/renderer/components/batch-builder/StoragePicker.tsx`:
+  - Three-way toggle per storage dimension: **Local** / **Cloud** / **Sync** / **None**
+  - Dimensions: Records, Vectors, Raw data
+  - Shows storage impact estimate per option
+  - Disabled options grayed out with tier tooltip if current tier doesn't support cloud
+- [ ] Create `src/renderer/components/batch-builder/SchedulePicker.tsx`:
+  - Mode toggle: **Cron** (with human-readable label, e.g. "Every day at 08:00") / **Event** (on new data from connector)
+  - Timezone selector (defaults to system timezone)
+  - Visual cron builder for non-technical users (with raw cron input fallback)
+- [ ] Create `src/renderer/components/batch-builder/BatchCard.tsx`:
+  - Shows batch name, connector icon, last run time, next run time, status badge (`idle` / `running` / `error` / `disabled`)
+  - Actions: Run now, Edit, Disable/Enable, Delete
+  - Expandable to show last run summary (items processed, errors)
+- [ ] Create `src/renderer/components/batch-builder/BatchTestRunner.tsx`:
+  - "Dry Run" panel: picks one real item from the source, runs the full analysis pipeline, shows output without saving
+  - Shows LLM output, action mapping preview, what would be stored and where
+  - Pass/Fail indicator with detailed error on failure
+- [ ] Create `src/renderer/hooks/useBatchBuilder.ts`:
+  - `useBatchBuilder()` — `generate(description): Promise<BatchConfig>`, `validate(config)`, `save(config)`, `activate(id)`, `deactivate(id)`, `runNow(id)`, `dryRun(id)`, `delete(id)`, list of saved configs with live status
+  - Backed by tRPC `batch.*` procedures
+- **Files:** `src/renderer/pages/BatchBuilderPage.tsx`, `src/renderer/components/batch-builder/{NaturalLanguageInput,ConfigPreview,ConnectorPicker,StoragePicker,SchedulePicker,BatchCard,BatchTestRunner}.tsx`, `src/renderer/hooks/useBatchBuilder.ts`
+- **Outcome:** Full Batch Builder UI — users can describe a batch in natural language, review/edit the generated config, dry-run it, and activate it with a single flow.
+
+### Step 6.4 — PluginStorePage
+- [ ] Create `src/renderer/pages/PluginStorePage.tsx`:
+  - Two tabs: **Marketplace** (browse available plugins) and **Installed** (manage installed plugins)
+  - Marketplace: search bar, category filter chips, grid of plugin cards sorted by rating/installs
+  - Installed: list of `InstalledPlugin` entries with enable/disable toggles and settings links
+- [ ] Create plugin card component (inline or shared `common/`):
+  - Shows name, author, description, rating (stars), install count, category badge, price/free badge
+  - "Install" button → triggers permission request dialog → installs plugin
+  - "Settings" button (installed) → opens plugin-specific config drawer
+- [ ] Plugin install flow:
+  - On install click: fetch plugin manifest from backend
+  - Show `PermissionDialog` with the permissions the plugin requires
+  - On approve: call tRPC `plugins.install(id)`, download and register the plugin worker
+  - Show `StoragePicker` for the plugin's data (what goes local/cloud/sync)
+- **Files:** `src/renderer/pages/PluginStorePage.tsx`
+- **Outcome:** Users can discover and install pre-built plugins from the marketplace with full permission visibility.
+
+### Step 6.5 — DataManagerPage
+- [ ] Create `src/renderer/pages/DataManagerPage.tsx`:
+  - Top section: `StorageOverview` dashboard
+  - Below: list of `DataSourceCard` for each active data source (one card per connector/plugin)
+  - "Migrate" button opens `MigrationWizard`
+- [ ] Create `src/renderer/components/data-manager/StorageOverview.tsx`:
+  - Visual breakdown: local disk used vs. cloud used vs. cloud limit
+  - Per-category breakdown (emails, files, notes, calendar, vectors)
+  - Tier upgrade CTA if approaching cloud limit
+- [ ] Create `src/renderer/components/data-manager/DataSourceCard.tsx`:
+  - Card per data source (e.g. "Gmail Scanner", "Documenti/Fatture watcher")
+  - Shows record count, size, last sync time
+  - Inline `StoragePicker` toggle for that source (where its data lives)
+  - "Clear local cache" / "Delete all data" actions with confirmation
+- [ ] Create `src/renderer/components/data-manager/MigrationWizard.tsx`:
+  - Step wizard: select source → select direction (local → cloud or cloud → local) → confirm
+  - Shows estimated data size and time
+  - Progress indicator during migration
+  - Rolls back on error
+- **Files:** `src/renderer/pages/DataManagerPage.tsx`, `src/renderer/components/data-manager/{StorageOverview,DataSourceCard,MigrationWizard}.tsx`
+- **Outcome:** Users have full visibility and control over where every piece of their data lives.
+
+### Step 6.6 — ActivityLogPage
+- [ ] Create `src/renderer/pages/ActivityLogPage.tsx`:
+  - Full-page filterable table of all batch/plugin activity entries
+  - Columns: timestamp, source (batch name / plugin name), action type, data accessed, storage destination, status
+  - Filters: source, date range, action type, status (success/error)
+  - Row expand: shows full detail — which records were created/updated, which files were read, LLM calls made
+  - Export as CSV button
+- **Files:** `src/renderer/pages/ActivityLogPage.tsx`
+- **Outcome:** Complete transparency log so users can audit exactly what each agent did and when.
+
+### Step 6.7 — SettingsPage (multi-provider, auth, backup, embeddings)
+- [ ] Create `src/renderer/pages/SettingsPage.tsx` with tabbed sections:
+  - **AI Providers** tab:
+    - List of configured providers with status badge (active / inactive / error)
+    - Add provider form: name dropdown (OpenAI, Anthropic, Google, Mistral, Groq, Ollama), API key input, model selection, endpoint (for Ollama)
+    - Set primary provider and fallback chain
+    - Test connection button per provider
+    - Separate "Embeddings provider" section: provider + model for embeddings (OpenAI, Cohere, Voyage, Mistral Embed)
+    - Info callout: "Text sent to the embeddings provider to generate vectors — make sure you trust this provider with your data"
+  - **Account & Billing** tab:
+    - Login/register form (when not authenticated)
+    - Current tier display with feature list and upgrade CTA
+    - Usage indicators (batch count, cloud storage used)
+    - Logout button
+  - **Backup & Sync** tab:
+    - Recovery passphrase: generate new / view existing (masked, reveal on click)
+    - Manual backup trigger with last backup timestamp
+    - Auto-backup schedule toggle + interval picker
+    - Backup history table (timestamp, size, restore button)
+  - **Permissions** tab:
+    - Table of all active permission grants (plugin/batch, permission type, resource, granted date)
+    - Revoke button per grant
+    - Links to ActivityLogPage for per-source audit
+- [ ] Create `src/renderer/components/common/ProviderSelector.tsx`:
+  - Reusable dropdown that lists configured LLM providers
+  - Used in BatchBuilder (model_override field) and Settings
+- [ ] Create `src/renderer/components/common/PermissionDialog.tsx`:
+  - Modal triggered when a plugin/batch requests new permissions
+  - Lists each requested permission with its reason and resource path
+  - Per-permission approve/deny toggles (deny is default)
+  - Shows plugin/batch manifest info (name, description, version)
+  - "Approve selected" confirms; "Deny all" closes without granting
+- **Files:** `src/renderer/pages/SettingsPage.tsx`, `src/renderer/components/common/PermissionDialog.tsx`, `src/renderer/components/common/ProviderSelector.tsx`
+- **Outcome:** Centralised settings covering providers, embeddings, auth, backup, and permissions.
 
 ---
 
@@ -375,8 +509,10 @@
 | `argon2` | Key derivation for E2E backup |
 | `node-cron` | Batch agent scheduling |
 | `chokidar` | File watching (FileWatcher plugin) |
-| `imapflow` | IMAP client (EmailScanner plugin) |
+| `imapflow` | IMAP client (IMAP connector) |
-| `onnxruntime-node` | Local embeddings (optional) |
+| `googleapis` | Gmail + GDrive OAuth connectors |
+| `lancedb` | Local vector store |
+| `onnxruntime-node` | Local embeddings (optional, future) |
 
 ---