diff --git a/docs/superpowers/plans/2026-05-11-project-folder-integration.md b/docs/superpowers/plans/2026-05-11-project-folder-integration.md new file mode 100644 index 0000000..7809319 --- /dev/null +++ b/docs/superpowers/plans/2026-05-11-project-folder-integration.md @@ -0,0 +1,3035 @@ +# Project Folder Integration Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let users link a local folder to a project; backend agents (Home / Brief / Task-Brief) receive a pre-injected manifest of per-file summaries and can read files on demand via a scoped tool. + +**Architecture:** Lightweight manifest (no embeddings). Electron scans + streams files over the existing `/api/v1/device` WS. Backend generates per-file summaries via gpt-4o-mini (text + vision), records token usage in Postgres, returns summaries to Electron which stores them locally. Agents pre-inject the manifest and lazy-read file content through a new scoped tool. + +**Tech Stack:** +- API: FastAPI, SQLAlchemy 2.0 async, Alembic, LiteLLM, Langfuse, pytest + pytest-asyncio. +- adiuvAI: Electron, TypeScript, Drizzle ORM + better-sqlite3, React 19, TanStack Router, tRPC v11, shadcn/ui new-york. + +**Reference spec:** `docs/superpowers/specs/2026-05-11-project-folder-integration-design.md` + +--- + +## File Inventory + +### API (Python) + +| File | Action | Responsibility | +|---|---|---| +| `api/alembic/versions/d6e3f4a5b6c7_folder_index_tables.py` | Create | Migration: `agent_run_logs.tokens_used`, `monthly_token_usage` | +| `api/app/models.py` | Modify | Add `MonthlyTokenUsage` model, `AgentRunLog.tokens_used` | +| `api/app/billing/tier_manager.py` | Modify | Add `folder_max_files`, `folder_monthly_tokens` features | +| `api/app/billing/quota.py` | Create | `check_folder_quota`, `add_token_usage` helpers | +| `api/app/api/routes/billing.py` | Modify | Add `POST /quota/check` endpoint | +| `api/app/core/folder_indexer.py` | Create | `summarize_text`, `summarize_image` LLM calls | +| `api/app/agents/folder_agent.py` | Create | Scoped `read_project_folder_file` tool | +| `api/app/api/routes/device_ws.py` | Modify | New frame handlers (`index_*`) | +| `api/app/core/deep_agent.py` | Modify | Manifest pre-injection in `run_home`, `run_brief`, `run_task_brief_research_stream` | +| `api/tests/test_folder_indexer.py` | Create | summarize_text/image happy path + token recording | +| `api/tests/test_folder_quota.py` | Create | tier rejection + atomic increment + monthly rollover | +| `api/tests/test_ws_index_session.py` | Create | session lifecycle + cancel + bad payloads | +| `api/tests/test_folder_agent_tool.py` | Create | scoped tool + traversal guard | +| `api/tests/test_manifest_injection.py` | Create | manifest formatting + truncation | + +### adiuvAI (TypeScript) + +| File | Action | Responsibility | +|---|---|---| +| `adiuvAI/src/main/db/schema.ts` | Modify | Add `projects.folder*` columns + `projectFolderFiles` table | +| `adiuvAI/src/main/db/migrations/0008_project_folders.sql` | Create | Drizzle SQL migration | +| `adiuvAI/src/main/db/migrations/meta/_journal.json` | Modify | Register new migration | +| `adiuvAI/src/main/files/constants.ts` | Create | Extension whitelist, size caps | +| `adiuvAI/src/main/files/scanner.ts` | Create | Walk dir + filter + mtime delta | +| `adiuvAI/src/main/files/indexer.ts` | Create | WS index session orchestration | +| `adiuvAI/src/main/files/daily-rescan.ts` | Create | On-ready 24h staleness check | +| `adiuvAI/src/main/router/projectFolders.ts` | Create | tRPC procedures | +| `adiuvAI/src/main/router/index.ts` | Modify | Register `projectFolders` sub-router | +| `adiuvAI/src/main/api/backend-client.ts` | Modify | Add `sendIndexBatch`, `cancelIndexSession` | +| `adiuvAI/src/main/api/drizzle-executor.ts` | Modify | Add `read_project_folder_manifest`, `read_project_folder_file`, `list_projects_with_folder_manifests` | +| `adiuvAI/src/main/index.ts` | Modify | Wire `runDailyRescan()` on `ready` | +| `adiuvAI/src/renderer/components/projects/folder/FolderChip.tsx` | Create | Hero chip | +| `adiuvAI/src/renderer/components/projects/folder/FilesSection.tsx` | Create | Files tab body | +| `adiuvAI/src/renderer/components/projects/folder/FolderLinkCard.tsx` | Create | Path + Rescan + Unlink | +| `adiuvAI/src/renderer/components/projects/folder/FolderFileList.tsx` | Create | Manifest list | +| `adiuvAI/src/renderer/components/projects/folder/FolderUnlinkDialog.tsx` | Create | Confirm unlink | +| `adiuvAI/src/renderer/components/projects/ProjectTabBar.tsx` | Modify | Add `'files'` section | +| `adiuvAI/src/renderer/components/projects/ProjectDetail.tsx` | Modify | Wire chip + Files section | +| `adiuvAI/src/renderer/locales/{en,it,es,fr,de}/translation.json` | Modify | Add `projects.folder.*` keys | + +--- + +## Phase A — Backend Foundation (Quota + Migrations) + +### Task 1: Alembic migration — folder index tables + +**Files:** +- Create: `api/alembic/versions/d6e3f4a5b6c7_folder_index_tables.py` + +- [ ] **Step 1: Write the migration file** + +```python +"""Add token tracking columns for folder integration. + +Revision ID: d6e3f4a5b6c7 +Revises: e04100e88ace +Create Date: 2026-05-11 00:00:00.000000 +""" +from __future__ import annotations + +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op +from sqlalchemy.dialects.postgresql import UUID + +revision: str = "d6e3f4a5b6c7" +down_revision: Union[str, None] = "e04100e88ace" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.add_column( + "agent_run_logs", + sa.Column("tokens_used", sa.Integer(), nullable=False, server_default="0"), + ) + op.create_table( + "monthly_token_usage", + sa.Column("user_id", UUID(as_uuid=False), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False), + sa.Column("year_month", sa.String(7), nullable=False), + sa.Column("feature", sa.String(64), nullable=False), + sa.Column("tokens_used", sa.Integer(), nullable=False, server_default="0"), + sa.PrimaryKeyConstraint("user_id", "year_month", "feature"), + ) + op.create_index( + "ix_monthly_token_usage_user_month", + "monthly_token_usage", + ["user_id", "year_month"], + ) + + +def downgrade() -> None: + op.drop_index("ix_monthly_token_usage_user_month", table_name="monthly_token_usage") + op.drop_table("monthly_token_usage") + op.drop_column("agent_run_logs", "tokens_used") +``` + +- [ ] **Step 2: Run migration locally** + +Run: `cd api && alembic upgrade head` +Expected: `INFO [alembic.runtime.migration] Running upgrade e04100e88ace -> d6e3f4a5b6c7` + +- [ ] **Step 3: Commit** + +```bash +git add api/alembic/versions/d6e3f4a5b6c7_folder_index_tables.py +git commit -m "feat(api): add migration for folder token tracking" +``` + +--- + +### Task 2: ORM model — MonthlyTokenUsage + AgentRunLog.tokens_used + +**Files:** +- Modify: `api/app/models.py` (add column to `AgentRunLog`, new `MonthlyTokenUsage` class) + +- [ ] **Step 1: Add `tokens_used` column to `AgentRunLog`** + +Insert after `items_created` (around line 245): + +```python + tokens_used: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") +``` + +- [ ] **Step 2: Add `MonthlyTokenUsage` model** + +Append below `AgentRunLog` (after line 263): + +```python +class MonthlyTokenUsage(Base): + __tablename__ = "monthly_token_usage" + + user_id: Mapped[str] = mapped_column( + Uuid(as_uuid=False), ForeignKey("users.id", ondelete="CASCADE"), primary_key=True + ) + year_month: Mapped[str] = mapped_column(String(7), primary_key=True) # 'YYYY-MM' + feature: Mapped[str] = mapped_column(String(64), primary_key=True) + tokens_used: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") +``` + +- [ ] **Step 3: Run existing test suite to verify no regression** + +Run: `cd api && pytest tests/ -x --tb=short` +Expected: PASS (existing tests unaffected by added column) + +- [ ] **Step 4: Commit** + +```bash +git add api/app/models.py +git commit -m "feat(api): MonthlyTokenUsage model + AgentRunLog.tokens_used" +``` + +--- + +### Task 3: Tier matrix — folder features + +**Files:** +- Modify: `api/app/billing/tier_manager.py:20-69` + +- [ ] **Step 1: Add features to each tier** + +Add inside each tier dict in `FEATURES`: + +```python + "free": { + # existing keys... + "folder_max_files": 200, + "folder_monthly_tokens": 100_000, + }, + "pro": { + # existing keys... + "folder_max_files": 5000, + "folder_monthly_tokens": 2_000_000, + }, + "power": { + # existing keys... + "folder_max_files": -1, + "folder_monthly_tokens": -1, + }, + "team": { + # existing keys... + "folder_max_files": -1, + "folder_monthly_tokens": -1, + }, +``` + +- [ ] **Step 2: Add `get_feature_value` helper** + +Append a method to `TierManager` class (after `require_feature`): + +```python + def get_feature_value(self, tier: BillingTier, feature: str) -> int: + """Return integer feature value for tier. -1 means unlimited.""" + value = FEATURES.get(tier, FEATURES["free"]).get(feature) + if not isinstance(value, int): + return 0 + return value +``` + +- [ ] **Step 3: Commit** + +```bash +git add api/app/billing/tier_manager.py +git commit -m "feat(api): tier features for folder integration" +``` + +--- + +### Task 4: Quota helpers (TDD) + +**Files:** +- Create: `api/app/billing/quota.py` +- Test: `api/tests/test_folder_quota.py` + +- [ ] **Step 1: Write the failing test** + +```python +# api/tests/test_folder_quota.py +"""Folder quota helpers.""" +from __future__ import annotations + +from datetime import datetime, timezone + +import pytest +from sqlalchemy import select + +from app.billing.quota import ( + check_folder_quota, + add_token_usage, + QuotaExceeded, +) +from app.models import MonthlyTokenUsage + + +pytestmark = pytest.mark.asyncio + + +async def test_check_folder_quota_free_rejects_above_file_cap(db, test_user_free): + with pytest.raises(QuotaExceeded) as exc: + await check_folder_quota( + user_id=test_user_free.id, tier="free", estimated_files=500, db=db + ) + assert exc.value.reason == "max_files" + + +async def test_check_folder_quota_free_passes_under_cap(db, test_user_free): + # No raise + await check_folder_quota( + user_id=test_user_free.id, tier="free", estimated_files=50, db=db + ) + + +async def test_check_folder_quota_rejects_when_monthly_exhausted(db, test_user_free): + ym = datetime.now(timezone.utc).strftime("%Y-%m") + db.add(MonthlyTokenUsage( + user_id=test_user_free.id, year_month=ym, feature="folder_index", tokens_used=100_000 + )) + await db.commit() + with pytest.raises(QuotaExceeded) as exc: + await check_folder_quota( + user_id=test_user_free.id, tier="free", estimated_files=10, db=db + ) + assert exc.value.reason == "monthly_tokens" + + +async def test_check_folder_quota_power_unlimited(db, test_user_power): + await check_folder_quota( + user_id=test_user_power.id, tier="power", estimated_files=999_999, db=db + ) + + +async def test_add_token_usage_atomic_increment(db, test_user_free): + await add_token_usage(user_id=test_user_free.id, feature="folder_index", tokens=1500, db=db) + await add_token_usage(user_id=test_user_free.id, feature="folder_index", tokens=2500, db=db) + ym = datetime.now(timezone.utc).strftime("%Y-%m") + row = (await db.execute( + select(MonthlyTokenUsage).where( + MonthlyTokenUsage.user_id == test_user_free.id, + MonthlyTokenUsage.year_month == ym, + MonthlyTokenUsage.feature == "folder_index", + ) + )).scalar_one() + assert row.tokens_used == 4000 + + +async def test_add_token_usage_returns_exhausted_when_over_cap(db, test_user_free): + result = await add_token_usage( + user_id=test_user_free.id, feature="folder_index", tokens=150_000, db=db, cap=100_000 + ) + assert result.exhausted is True + assert result.tokens_used == 150_000 +``` + +You will also need to add `test_user_free` and `test_user_power` fixtures to `api/tests/conftest.py` (one user with `Subscription.tier='free'`, one with `'power'`) — follow the existing user-creation pattern in that file. + +- [ ] **Step 2: Run test — expect import errors** + +Run: `cd api && pytest tests/test_folder_quota.py -v` +Expected: ImportError (`app.billing.quota` does not exist). + +- [ ] **Step 3: Implement `app/billing/quota.py`** + +```python +"""Quota checks and atomic token-usage accounting for folder integration.""" +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timezone + +from sqlalchemy import select +from sqlalchemy.dialects.postgresql import insert as pg_insert +from sqlalchemy.ext.asyncio import AsyncSession + +from app.billing.tier_manager import TierManager +from app.models import MonthlyTokenUsage +from app.schemas import BillingTier + + +class QuotaExceeded(Exception): + """Raised when a folder operation cannot proceed under the user's tier.""" + + def __init__(self, reason: str, message: str) -> None: + super().__init__(message) + self.reason = reason # "max_files" | "monthly_tokens" + + +@dataclass +class TokenUsageResult: + tokens_used: int + exhausted: bool + + +def _current_year_month() -> str: + return datetime.now(timezone.utc).strftime("%Y-%m") + + +_tier_manager = TierManager() + + +async def check_folder_quota( + *, + user_id: str, + tier: BillingTier, + estimated_files: int, + db: AsyncSession, +) -> None: + """Raise QuotaExceeded if folder_max_files or folder_monthly_tokens + would be violated. -1 in either feature means unlimited.""" + max_files = _tier_manager.get_feature_value(tier, "folder_max_files") + if max_files != -1 and estimated_files > max_files: + raise QuotaExceeded( + "max_files", + f"Folder has {estimated_files} files; tier '{tier}' allows max {max_files}.", + ) + + cap = _tier_manager.get_feature_value(tier, "folder_monthly_tokens") + if cap == -1: + return + ym = _current_year_month() + row = ( + await db.execute( + select(MonthlyTokenUsage).where( + MonthlyTokenUsage.user_id == user_id, + MonthlyTokenUsage.year_month == ym, + MonthlyTokenUsage.feature == "folder_index", + ) + ) + ).scalar_one_or_none() + used = row.tokens_used if row else 0 + if used >= cap: + raise QuotaExceeded( + "monthly_tokens", + f"Monthly token budget exhausted ({used}/{cap}); resets next month.", + ) + + +async def add_token_usage( + *, + user_id: str, + feature: str, + tokens: int, + db: AsyncSession, + cap: int | None = None, +) -> TokenUsageResult: + """Atomically add `tokens` to MonthlyTokenUsage row for (user, current month, feature). + Returns post-update total + whether cap is exhausted.""" + ym = _current_year_month() + stmt = pg_insert(MonthlyTokenUsage).values( + user_id=user_id, year_month=ym, feature=feature, tokens_used=tokens, + ).on_conflict_do_update( + index_elements=["user_id", "year_month", "feature"], + set_={"tokens_used": MonthlyTokenUsage.tokens_used + tokens}, + ).returning(MonthlyTokenUsage.tokens_used) + used = (await db.execute(stmt)).scalar_one() + await db.commit() + exhausted = cap is not None and cap != -1 and used >= cap + return TokenUsageResult(tokens_used=used, exhausted=exhausted) +``` + +- [ ] **Step 4: Run tests, verify PASS** + +Run: `cd api && pytest tests/test_folder_quota.py -v` +Expected: 6 passed + +- [ ] **Step 5: Commit** + +```bash +git add api/app/billing/quota.py api/tests/test_folder_quota.py api/tests/conftest.py +git commit -m "feat(api): folder quota helpers with atomic token usage" +``` + +--- + +### Task 5: `POST /api/v1/billing/quota/check` endpoint + +**Files:** +- Modify: `api/app/api/routes/billing.py` +- Test: extend `api/tests/test_folder_quota.py` + +- [ ] **Step 1: Write failing endpoint test** + +Append to `api/tests/test_folder_quota.py`: + +```python +async def test_quota_check_endpoint_rejects(client, auth_headers_free): + res = await client.post( + "/api/v1/billing/quota/check", + json={"feature": "folder_index", "estimated_files": 500}, + headers=auth_headers_free, + ) + assert res.status_code == 402 + body = res.json() + assert body["detail"]["reason"] == "max_files" + + +async def test_quota_check_endpoint_passes(client, auth_headers_free): + res = await client.post( + "/api/v1/billing/quota/check", + json={"feature": "folder_index", "estimated_files": 50}, + headers=auth_headers_free, + ) + assert res.status_code == 200 + assert res.json() == {"ok": True} +``` + +(Reuse the `client` and `auth_headers_*` fixtures from `conftest.py`.) + +- [ ] **Step 2: Run — expect 404 (route missing)** + +Run: `cd api && pytest tests/test_folder_quota.py::test_quota_check_endpoint_rejects -v` +Expected: assertion 404 != 402 + +- [ ] **Step 3: Add endpoint to `billing.py`** + +Add at the bottom of `api/app/api/routes/billing.py`: + +```python +from pydantic import BaseModel +from app.billing.quota import check_folder_quota, QuotaExceeded + + +class QuotaCheckRequest(BaseModel): + feature: str + estimated_files: int + + +@router.post("/quota/check") +async def quota_check( + payload: QuotaCheckRequest, + current_user=Depends(get_current_user), + db: AsyncSession = Depends(get_db), +) -> dict: + """Pre-flight folder quota check. 402 if tier limits would be exceeded.""" + if payload.feature != "folder_index": + raise HTTPException(status_code=400, detail="Unknown feature") + try: + await check_folder_quota( + user_id=current_user.id, + tier=current_user.tier, + estimated_files=payload.estimated_files, + db=db, + ) + except QuotaExceeded as exc: + raise HTTPException( + status_code=402, + detail={"reason": exc.reason, "message": str(exc)}, + ) + return {"ok": True} +``` + +If `HTTPException`, `Depends`, `get_current_user`, `get_db`, `AsyncSession`, or `router` are not yet imported in `billing.py`, add them at the top (match existing imports in that file). + +- [ ] **Step 4: Run tests, verify PASS** + +Run: `cd api && pytest tests/test_folder_quota.py -v` +Expected: 8 passed + +- [ ] **Step 5: Commit** + +```bash +git add api/app/api/routes/billing.py api/tests/test_folder_quota.py +git commit -m "feat(api): POST /billing/quota/check endpoint" +``` + +--- + +## Phase B — Backend Indexer + +### Task 6: Langfuse prompts (manual) + +**Files:** (none — Langfuse UI) + +- [ ] **Step 1: In the Langfuse UI, create two text prompts** + +Prompt 1: `folder_file_summary_text` +``` +You are summarising a file for an AI assistant that helps the user manage a project. +Produce a single sentence (≤30 words, ≤200 chars) that captures the file's purpose and most important detail. +File extension: {{ext}} +File name: {{name}} +Content (truncated if long): +{{content}} +``` + +Prompt 2: `folder_file_summary_image` +``` +You are summarising an image attached to a project folder. +Produce a single sentence (≤30 words, ≤200 chars) describing what the image shows and any obvious purpose (logo, screenshot, diagram, photo of a whiteboard, etc.). +``` + +Tag both as `production`. Note: backend will use these via `get_prompt_or_fallback` so hardcoded fallbacks are mandatory in the next task. + +- [ ] **Step 2: No commit (Langfuse-only).** Proceed to Task 7. + +--- + +### Task 7: `core/folder_indexer.py` — summarize_text (TDD) + +**Files:** +- Create: `api/app/core/folder_indexer.py` +- Test: `api/tests/test_folder_indexer.py` + +- [ ] **Step 1: Write failing test** + +```python +# api/tests/test_folder_indexer.py +"""Folder indexer LLM helpers.""" +from __future__ import annotations + +from unittest.mock import AsyncMock, patch + +import pytest + +from app.core.folder_indexer import summarize_text, summarize_image, IndexResult + +pytestmark = pytest.mark.asyncio + + +async def test_summarize_text_returns_summary_and_tokens(): + mock_resp = AsyncMock() + mock_resp.content = "Kickoff notes covering scope and deadlines." + mock_resp.usage_metadata = {"input_tokens": 320, "output_tokens": 18, "total_tokens": 338} + with patch("app.core.folder_indexer._llm_text", new=AsyncMock(return_value=mock_resp)): + result = await summarize_text(content="hello world", ext=".md", name="kickoff.md") + assert isinstance(result, IndexResult) + assert result.summary == "Kickoff notes covering scope and deadlines." + assert result.tokens_used == 338 + + +async def test_summarize_text_truncates_summary_at_500_chars(): + mock_resp = AsyncMock() + mock_resp.content = "x" * 1000 + mock_resp.usage_metadata = {"total_tokens": 100} + with patch("app.core.folder_indexer._llm_text", new=AsyncMock(return_value=mock_resp)): + result = await summarize_text(content="x", ext=".md", name="x.md") + assert len(result.summary) <= 500 +``` + +- [ ] **Step 2: Run — expect ImportError** + +Run: `cd api && pytest tests/test_folder_indexer.py -v` +Expected: ImportError + +- [ ] **Step 3: Implement `app/core/folder_indexer.py` (text only)** + +```python +"""Per-file summarisation for project folder integration.""" +from __future__ import annotations + +from dataclasses import dataclass + +from langchain_core.messages import HumanMessage, SystemMessage + +from app.core.langfuse_client import ( + compile_prompt, + extract_usage, + get_prompt_or_fallback, +) +from app.core.llm import get_llm + +_TEXT_FALLBACK = ( + "You are summarising a file for an AI assistant that helps the user manage a project.\n" + "Produce a single sentence (≤30 words, ≤200 chars) that captures the file's purpose " + "and most important detail.\nFile extension: {{ext}}\nFile name: {{name}}\nContent (truncated if long):\n{{content}}" +) +_IMAGE_FALLBACK = ( + "You are summarising an image attached to a project folder.\n" + "Produce a single sentence (≤30 words, ≤200 chars) describing what the image shows " + "and any obvious purpose (logo, screenshot, diagram, photo of a whiteboard, etc.)." +) +_MAX_INPUT_CHARS = 6000 + + +@dataclass +class IndexResult: + summary: str + tokens_used: int + + +async def _llm_text(): + """Indirection so tests can patch the LLM call.""" + return get_llm(model="gpt-4o-mini", temperature=0.2) + + +async def summarize_text(*, content: str, ext: str, name: str) -> IndexResult: + template, prompt_obj = get_prompt_or_fallback("folder_file_summary_text", _TEXT_FALLBACK) + truncated = content[:_MAX_INPUT_CHARS] + compiled = compile_prompt(template, ext=ext, name=name, content=truncated) + llm = await _llm_text() + response = await llm.ainvoke([ + SystemMessage(content=compiled), + HumanMessage(content="Summarise this file."), + ]) + usage = extract_usage(response) + summary = (response.content or "").strip()[:500] + return IndexResult(summary=summary, tokens_used=usage.get("total_tokens", 0)) +``` + +Note: leave `summarize_image` undefined for now — the test for it will fail in Task 8. + +- [ ] **Step 4: Run text tests, verify PASS** + +Run: `cd api && pytest tests/test_folder_indexer.py::test_summarize_text_returns_summary_and_tokens tests/test_folder_indexer.py::test_summarize_text_truncates_summary_at_500_chars -v` +Expected: 2 passed + +- [ ] **Step 5: Commit** + +```bash +git add api/app/core/folder_indexer.py api/tests/test_folder_indexer.py +git commit -m "feat(api): folder_indexer.summarize_text via gpt-4o-mini" +``` + +--- + +### Task 8: `summarize_image` (vision) + +**Files:** +- Modify: `api/app/core/folder_indexer.py` +- Modify: `api/tests/test_folder_indexer.py` + +- [ ] **Step 1: Append failing test** + +```python +async def test_summarize_image_uses_vision_content_blocks(): + mock_resp = AsyncMock() + mock_resp.content = "Final logo on white background." + mock_resp.usage_metadata = {"total_tokens": 500} + captured = {} + async def fake_invoke(messages): + captured["messages"] = messages + return mock_resp + fake_llm = AsyncMock() + fake_llm.ainvoke = fake_invoke + with patch("app.core.folder_indexer._llm_vision", new=AsyncMock(return_value=fake_llm)): + result = await summarize_image(image_b64="iVBORw0KG", mime="image/png") + assert "Final logo" in result.summary + assert result.tokens_used == 500 + # last message contains an image content block + last = captured["messages"][-1] + assert any( + isinstance(p, dict) and p.get("type") == "image_url" + for p in (last.content if isinstance(last.content, list) else []) + ) +``` + +- [ ] **Step 2: Run — expect failure** + +Run: `cd api && pytest tests/test_folder_indexer.py::test_summarize_image_uses_vision_content_blocks -v` +Expected: FAIL (`summarize_image` not defined) + +- [ ] **Step 3: Implement `summarize_image`** + +Add at the bottom of `app/core/folder_indexer.py`: + +```python +async def _llm_vision(): + return get_llm(model="gpt-4o-mini", temperature=0.2) + + +async def summarize_image(*, image_b64: str, mime: str) -> IndexResult: + template, prompt_obj = get_prompt_or_fallback("folder_file_summary_image", _IMAGE_FALLBACK) + llm = await _llm_vision() + response = await llm.ainvoke([ + SystemMessage(content=template), + HumanMessage(content=[ + {"type": "text", "text": "Summarise this image."}, + {"type": "image_url", "image_url": {"url": f"data:{mime};base64,{image_b64}"}}, + ]), + ]) + usage = extract_usage(response) + summary = (response.content or "").strip()[:500] + return IndexResult(summary=summary, tokens_used=usage.get("total_tokens", 0)) +``` + +- [ ] **Step 4: Run test, verify PASS** + +Run: `cd api && pytest tests/test_folder_indexer.py -v` +Expected: 3 passed + +- [ ] **Step 5: Commit** + +```bash +git add api/app/core/folder_indexer.py api/tests/test_folder_indexer.py +git commit -m "feat(api): folder_indexer.summarize_image via gpt-4o-mini vision" +``` + +--- + +### Task 8b: PDF & DOCX text extraction in indexer + +**Files:** +- Modify: `api/app/core/folder_indexer.py` +- Modify: `api/tests/test_folder_indexer.py` +- Modify: `api/pyproject.toml` (add `pypdf`, `python-docx` deps) + +- [ ] **Step 1: Append failing tests** + +```python +async def test_summarize_pdf_extracts_then_summarizes(monkeypatch): + # pypdf.PdfReader returns text from pages + from app.core import folder_indexer + class FakePage: + def extract_text(self): return "PDF page content with project info." + class FakeReader: + pages = [FakePage(), FakePage()] + monkeypatch.setattr(folder_indexer, "PdfReader", lambda buf: FakeReader()) + mock_resp = AsyncMock(); mock_resp.content = "Project info doc."; mock_resp.usage_metadata = {"total_tokens": 50} + with patch("app.core.folder_indexer._llm_text", new=AsyncMock(return_value=mock_resp)): + result = await folder_indexer.summarize_pdf(pdf_b64="SGVsbG8=", name="doc.pdf") + assert "Project info" in result.summary + assert result.tokens_used == 50 + + +async def test_summarize_docx_extracts_then_summarizes(monkeypatch): + from app.core import folder_indexer + class FakePara: + def __init__(self, t): self.text = t + class FakeDoc: + paragraphs = [FakePara("Heading"), FakePara("Body paragraph one.")] + monkeypatch.setattr(folder_indexer, "DocxDocument", lambda buf: FakeDoc()) + mock_resp = AsyncMock(); mock_resp.content = "Heading and body."; mock_resp.usage_metadata = {"total_tokens": 30} + with patch("app.core.folder_indexer._llm_text", new=AsyncMock(return_value=mock_resp)): + result = await folder_indexer.summarize_docx(docx_b64="UEsDBBQ=", name="doc.docx") + assert result.summary == "Heading and body." +``` + +- [ ] **Step 2: Add dependencies** + +Edit `api/pyproject.toml` `[project.dependencies]` block — append: + +```toml +"pypdf>=4.0", +"python-docx>=1.1", +``` + +Run: `cd api && pip install -e .` +Expected: dependencies installed. + +- [ ] **Step 3: Add extractor helpers + summarize wrappers** + +Append to `app/core/folder_indexer.py`: + +```python +import base64 +import io + +from pypdf import PdfReader +from docx import Document as DocxDocument + + +def _extract_pdf_text(pdf_b64: str) -> str: + buf = io.BytesIO(base64.b64decode(pdf_b64)) + reader = PdfReader(buf) + parts: list[str] = [] + for page in reader.pages: + try: + parts.append(page.extract_text() or "") + except Exception: + continue + return "\n".join(parts).strip() + + +def _extract_docx_text(docx_b64: str) -> str: + buf = io.BytesIO(base64.b64decode(docx_b64)) + doc = DocxDocument(buf) + return "\n".join(p.text for p in doc.paragraphs if p.text).strip() + + +async def summarize_pdf(*, pdf_b64: str, name: str) -> IndexResult: + text = _extract_pdf_text(pdf_b64) + if not text: + return IndexResult(summary="Could not extract text", tokens_used=0) + return await summarize_text(content=text, ext=".pdf", name=name) + + +async def summarize_docx(*, docx_b64: str, name: str) -> IndexResult: + text = _extract_docx_text(docx_b64) + if not text: + return IndexResult(summary="Could not extract text", tokens_used=0) + return await summarize_text(content=text, ext=".docx", name=name) +``` + +- [ ] **Step 4: Run tests, verify PASS** + +Run: `cd api && pytest tests/test_folder_indexer.py -v` +Expected: 5 passed + +- [ ] **Step 5: Commit** + +```bash +git add api/app/core/folder_indexer.py api/tests/test_folder_indexer.py api/pyproject.toml +git commit -m "feat(api): PDF + DOCX extraction in folder indexer" +``` + +--- + +### Task 9: WS index session — frames & state machine + +**Files:** +- Modify: `api/app/api/routes/device_ws.py` +- Test: `api/tests/test_ws_index_session.py` + +- [ ] **Step 1: Read existing `device_ws.py` to find the frame-dispatch switch** + +Run: `grep -n "frame.*type\|type ==" api/app/api/routes/device_ws.py | head -20` + +Note the function that dispatches on `frame["type"]`. New cases go there. + +- [ ] **Step 2: Write failing tests** + +```python +# api/tests/test_ws_index_session.py +"""WS index session lifecycle.""" +from __future__ import annotations + +from unittest.mock import AsyncMock, patch + +import pytest + +pytestmark = pytest.mark.asyncio + + +async def test_index_session_happy_path(ws_client, test_user_power): + # Open WS via fixture + await ws_client.send_json({ + "type": "index_session_start", + "sessionId": "s1", + "projectId": "p1", + "totalFiles": 2, + }) + # Patch indexer so we don't actually call OpenAI + with patch("app.api.routes.device_ws.summarize_text", new=AsyncMock( + return_value=type("R", (), {"summary": "ok", "tokens_used": 10})() + )): + await ws_client.send_json({ + "type": "index_file_batch", + "sessionId": "s1", + "files": [ + {"relPath": "a.md", "kind": "text", "content": "hello", "sizeBytes": 5, "mtimeMs": 1}, + {"relPath": "b.md", "kind": "text", "content": "world", "sizeBytes": 5, "mtimeMs": 2}, + ], + }) + results = [] + for _ in range(3): # 2 file results + 1 progress + msg = await ws_client.receive_json() + results.append(msg) + types = [m["type"] for m in results] + assert types.count("index_file_result") == 2 + assert "index_session_progress" in types + + +async def test_index_session_cancel(ws_client): + await ws_client.send_json({ + "type": "index_session_start", "sessionId": "s2", "projectId": "p1", "totalFiles": 100, + }) + await ws_client.send_json({"type": "index_session_cancel", "sessionId": "s2"}) + msg = await ws_client.receive_json() + assert msg["type"] == "index_session_done" + assert msg["status"] == "cancelled" + + +async def test_index_session_quota_exceeded(ws_client, test_user_free): + # Pre-fill monthly_token_usage to exhaust the cap (see conftest helper) + await exhaust_folder_tokens(test_user_free, ws_client) # helper in conftest + await ws_client.send_json({ + "type": "index_session_start", "sessionId": "s3", "projectId": "p1", "totalFiles": 1, + }) + await ws_client.send_json({ + "type": "index_file_batch", "sessionId": "s3", + "files": [{"relPath": "x.md", "kind": "text", "content": "x", "sizeBytes": 1, "mtimeMs": 1}], + }) + msgs = [await ws_client.receive_json() for _ in range(2)] + assert any(m.get("type") == "index_session_done" and m.get("status") == "quota_exceeded" for m in msgs) +``` + +The `ws_client` fixture and `exhaust_folder_tokens` helper need to be added to `tests/conftest.py` if absent; pattern in existing `tests/test_ws_unified.py`. + +- [ ] **Step 3: Run — expect failure** + +Run: `cd api && pytest tests/test_ws_index_session.py -v` +Expected: assertion failures (frame types unknown — handler doesn't exist). + +- [ ] **Step 4: Implement frame handlers in `device_ws.py`** + +Add a new module-level dict to hold per-session state: + +```python +# Top of device_ws.py +_index_sessions: dict[str, dict] = {} # sessionId -> { user_id, project_id, processed, total, cancelled } +``` + +Add three new cases in the dispatch: + +```python +elif frame_type == "index_session_start": + _index_sessions[frame["sessionId"]] = { + "user_id": ws_ctx.user_id, + "project_id": frame["projectId"], + "processed": 0, + "total": int(frame["totalFiles"]), + "cancelled": False, + } + +elif frame_type == "index_session_cancel": + sess = _index_sessions.get(frame["sessionId"]) + if sess: + sess["cancelled"] = True + await ws.send_json({ + "type": "index_session_done", + "sessionId": frame["sessionId"], + "status": "cancelled", + }) + _index_sessions.pop(frame["sessionId"], None) + +elif frame_type == "index_file_batch": + from app.core.folder_indexer import summarize_text, summarize_image, summarize_pdf, summarize_docx + from app.billing.quota import add_token_usage + from app.billing.tier_manager import TierManager + sess = _index_sessions.get(frame["sessionId"]) + if not sess or sess["cancelled"]: + return + cap = TierManager().get_feature_value(ws_ctx.user.tier, "folder_monthly_tokens") + for f in frame["files"]: + try: + if f["kind"] == "image": + res = await summarize_image(image_b64=f["content"], mime=f.get("mime", "image/png")) + elif f["kind"] == "pdf": + res = await summarize_pdf(pdf_b64=f["content"], name=f["relPath"]) + elif f["kind"] == "docx": + res = await summarize_docx(docx_b64=f["content"], name=f["relPath"]) + else: + res = await summarize_text(content=f["content"], ext=f.get("ext", ""), name=f["relPath"]) + usage = await add_token_usage( + user_id=sess["user_id"], + feature="folder_index", + tokens=res.tokens_used, + db=db_session, + cap=cap, + ) + except Exception as exc: + await ws.send_json({ + "type": "index_file_result", + "sessionId": frame["sessionId"], + "relPath": f["relPath"], + "summary": None, + "tokensUsed": 0, + "error": str(exc), + }) + continue + await ws.send_json({ + "type": "index_file_result", + "sessionId": frame["sessionId"], + "relPath": f["relPath"], + "summary": res.summary, + "tokensUsed": res.tokens_used, + }) + sess["processed"] += 1 + if usage.exhausted: + await ws.send_json({ + "type": "index_session_done", + "sessionId": frame["sessionId"], + "status": "quota_exceeded", + }) + _index_sessions.pop(frame["sessionId"], None) + return + await ws.send_json({ + "type": "index_session_progress", + "sessionId": frame["sessionId"], + "processed": sess["processed"], + "total": sess["total"], + }) + if sess["processed"] >= sess["total"]: + await ws.send_json({ + "type": "index_session_done", + "sessionId": frame["sessionId"], + "status": "completed", + }) + _index_sessions.pop(frame["sessionId"], None) +``` + +If `ws_ctx`, `db_session`, or the dispatch structure use different identifiers in the existing file, mirror those — read 30 lines around the dispatch site before pasting. + +- [ ] **Step 5: Run tests, verify PASS** + +Run: `cd api && pytest tests/test_ws_index_session.py -v` +Expected: 3 passed + +- [ ] **Step 6: Commit** + +```bash +git add api/app/api/routes/device_ws.py api/tests/test_ws_index_session.py api/tests/conftest.py +git commit -m "feat(api): WS index_session frames + handlers" +``` + +--- + +## Phase C — Backend Agent Wiring + +### Task 10: `folder_agent.py` — scoped read tool (TDD) + +**Files:** +- Create: `api/app/agents/folder_agent.py` +- Test: `api/tests/test_folder_agent_tool.py` + +- [ ] **Step 1: Write failing test** + +```python +# api/tests/test_folder_agent_tool.py +from __future__ import annotations + +from unittest.mock import AsyncMock, patch + +import pytest + +from app.agents.folder_agent import read_project_folder_file + +pytestmark = pytest.mark.asyncio + + +async def test_happy_path(): + with patch( + "app.agents.folder_agent.execute_on_client", + new=AsyncMock(return_value={"content": "file body"}), + ): + out = await read_project_folder_file.ainvoke({"project_id": "p1", "relative_path": "docs/x.md"}) + assert out == "file body" + + +async def test_traversal_rejected(): + out = await read_project_folder_file.ainvoke({"project_id": "p1", "relative_path": "../../etc/passwd"}) + assert out == "Access denied" + + +async def test_absolute_path_rejected(): + out = await read_project_folder_file.ainvoke({"project_id": "p1", "relative_path": "C:\\Windows\\foo"}) + assert out == "Access denied" + + +async def test_missing_file(): + with patch( + "app.agents.folder_agent.execute_on_client", + new=AsyncMock(return_value={"content": ""}), + ): + out = await read_project_folder_file.ainvoke({"project_id": "p1", "relative_path": "ghost.md"}) + assert "not found" in out.lower() +``` + +- [ ] **Step 2: Run — ImportError** + +Run: `cd api && pytest tests/test_folder_agent_tool.py -v` +Expected: ImportError + +- [ ] **Step 3: Implement `app/agents/folder_agent.py`** + +```python +"""Scoped file-read tool for the project folder feature.""" +from __future__ import annotations + +from langchain_core.tools import tool + +from app.core.ws_context import execute_on_client + + +def _is_unsafe_path(rel: str) -> bool: + if not rel: + return True + norm = rel.replace("\\", "/") + if norm.startswith("/"): + return True + # Windows drive letter + if len(rel) >= 2 and rel[1] == ":": + return True + parts = norm.split("/") + return ".." in parts + + +@tool +async def read_project_folder_file(project_id: str, relative_path: str) -> str: + """Read full content of a file inside the project's linked folder.""" + if _is_unsafe_path(relative_path): + return "Access denied" + result = await execute_on_client( + action="read_project_folder_file", + data={"projectId": project_id, "relativePath": relative_path}, + ) + content = result.get("content", "") + if not content: + return f"File not found: {relative_path}" + return content + + +FOLDER_TOOLS = [read_project_folder_file] +``` + +- [ ] **Step 4: Run tests, verify PASS** + +Run: `cd api && pytest tests/test_folder_agent_tool.py -v` +Expected: 4 passed + +- [ ] **Step 5: Commit** + +```bash +git add api/app/agents/folder_agent.py api/tests/test_folder_agent_tool.py +git commit -m "feat(api): scoped read_project_folder_file tool with traversal guard" +``` + +--- + +### Task 11: Manifest injection — formatter + truncation (TDD) + +**Files:** +- Modify: `api/app/core/deep_agent.py` +- Test: `api/tests/test_manifest_injection.py` + +- [ ] **Step 1: Write failing test** + +```python +# api/tests/test_manifest_injection.py +from __future__ import annotations + +from app.core.deep_agent import format_folder_manifest, MANIFEST_TOKEN_BUDGET + + +def test_format_folder_manifest_basic(): + manifest = { + "folderPath": "D:\\Acme", + "lastScannedAt": "2h ago", + "files": [ + {"relPath": "briefs/kickoff.md", "kind": "text", "summary": "Kickoff notes; scope and deadlines."}, + {"relPath": "logos/logo-v3.png", "kind": "image", "summary": "Final logo on white."}, + ], + } + out = format_folder_manifest(manifest) + assert "" in out + assert "/briefs/kickoff.md" in out or "briefs/kickoff.md" in out + assert "[text]" in out + assert "[image]" in out + + +def test_format_folder_manifest_truncates_past_budget(): + files = [ + {"relPath": f"f{i}.md", "kind": "text", "summary": "x" * 100, "mtimeMs": i} + for i in range(2000) + ] + out = format_folder_manifest({"folderPath": "p", "lastScannedAt": "now", "files": files}) + assert "more files omitted" in out + # Rough token check + assert len(out) // 4 < MANIFEST_TOKEN_BUDGET + 200 + + +def test_format_folder_manifest_null_returns_empty(): + assert format_folder_manifest(None) == "" + assert format_folder_manifest({"files": []}) == "" +``` + +- [ ] **Step 2: Run — expect failure** + +Run: `cd api && pytest tests/test_manifest_injection.py -v` +Expected: ImportError + +- [ ] **Step 3: Add formatter to `deep_agent.py`** + +Add near the top-level helpers (e.g. after the existing language-instruction helper): + +```python +MANIFEST_TOKEN_BUDGET = 3000 # rough budget for block + + +def format_folder_manifest(manifest: dict | None) -> str: + """Format a folder manifest into the block. + + Truncates by mtime DESC if estimated tokens exceed MANIFEST_TOKEN_BUDGET. + Returns empty string if manifest is None or has no files. + """ + if not manifest or not manifest.get("files"): + return "" + files = list(manifest["files"]) + files.sort(key=lambda f: f.get("mtimeMs", 0), reverse=True) + + header = ( + f"\npath: {manifest.get('folderPath', '?')} " + f"({len(files)} files, scanned {manifest.get('lastScannedAt', '?')})\nfiles:\n" + ) + footer_template = "… {} more files omitted, use read_project_folder_file to access by path\n" + + char_budget = MANIFEST_TOKEN_BUDGET * 4 # ~4 chars/token + body = "" + included = 0 + for f in files: + line = f"- /{f['relPath']} [{f.get('kind','text')}] {f.get('summary','')}\n" + if len(header) + len(body) + len(line) + len(footer_template.format(0)) > char_budget: + break + body += line + included += 1 + omitted = len(files) - included + if omitted > 0: + return header + body + footer_template.format(omitted) + return header + body + "" +``` + +- [ ] **Step 4: Run tests, verify PASS** + +Run: `cd api && pytest tests/test_manifest_injection.py -v` +Expected: 3 passed + +- [ ] **Step 5: Commit** + +```bash +git add api/app/core/deep_agent.py api/tests/test_manifest_injection.py +git commit -m "feat(api): manifest formatter with token-budget truncation" +``` + +--- + +### Task 12: Wire manifest into `run_task_brief_research_stream` + +**Files:** +- Modify: `api/app/core/deep_agent.py` + +- [ ] **Step 1: Read existing function signature** + +Run: `grep -n "async def run_task_brief_research_stream" api/app/core/deep_agent.py` + +Read 60 lines starting at that line. Identify (a) where the system prompt is assembled, (b) how `task` / `projectId` are available, (c) where tools are bound. + +- [ ] **Step 2: Add helper that fetches manifest** + +Add to `deep_agent.py`: + +```python +async def _fetch_project_manifest(project_id: str) -> dict | None: + """Fetch manifest from Electron via execute_on_client. Returns None if unlinked or error.""" + from app.core.ws_context import execute_on_client + try: + result = await execute_on_client( + action="read_project_folder_manifest", + data={"projectId": project_id}, + ) + if not result or not result.get("folderPath"): + return None + return result + except Exception: + return None +``` + +- [ ] **Step 3: Inject manifest in task-brief** + +Inside `run_task_brief_research_stream`, after `projectId` is known and before the system prompt is finalised, add: + +```python +manifest_block = "" +if project_id: + manifest = await _fetch_project_manifest(project_id) + manifest_block = format_folder_manifest(manifest) +system_prompt = system_prompt + ("\n\n" + manifest_block if manifest_block else "") +``` + +Also append the new tool to the agent's tool list: + +```python +from app.agents.folder_agent import FOLDER_TOOLS +tools = [...existing..., *FOLDER_TOOLS] +``` + +(Use exact variable names found in the function — do not guess.) + +- [ ] **Step 4: Smoke-test by running existing brief tests** + +Run: `cd api && pytest tests/test_brief_agent.py tests/test_deep_agent.py -v --tb=short` +Expected: PASS (manifest is `None` in tests since no WS, no behaviour change for existing flows). + +- [ ] **Step 5: Commit** + +```bash +git add api/app/core/deep_agent.py +git commit -m "feat(api): inject folder manifest into task brief agent" +``` + +--- + +### Task 13: Wire manifest into `run_home` + +**Files:** +- Modify: `api/app/core/deep_agent.py` + +- [ ] **Step 1: Locate `run_home`** + +Run: `grep -n "async def run_home" api/app/core/deep_agent.py` + +Identify how the caller can pass an `active_project_id` (search for existing `@project` mention parsing or `projectId` in the request payload — look at `device_ws.py` to see what fields the home frame contains). + +- [ ] **Step 2: Inject manifest when project context is set** + +Same pattern as Task 12 — inside `run_home`, if `project_id` (or `active_project_id`) is set, fetch + inject the manifest. Bind `FOLDER_TOOLS`. + +- [ ] **Step 3: Run home-agent tests** + +Run: `cd api && pytest tests/test_deep_agent.py -v --tb=short` +Expected: PASS + +- [ ] **Step 4: Commit** + +```bash +git add api/app/core/deep_agent.py +git commit -m "feat(api): inject folder manifest into home agent when project context active" +``` + +--- + +### Task 14: Brief agent — multi-project manifest (TDD) + +**Files:** +- Modify: `api/app/core/deep_agent.py` +- Modify: `api/tests/test_manifest_injection.py` + +- [ ] **Step 1: Write failing test** + +Append to `test_manifest_injection.py`: + +```python +from unittest.mock import AsyncMock, patch +import pytest + +from app.core.deep_agent import build_brief_multi_project_manifest + +pytestmark = pytest.mark.asyncio + + +async def test_brief_multi_project_manifest_top_5_per_project(): + fake_response = [ + { + "projectId": "p1", "projectName": "Acme", "folderPath": "/a", + "lastScannedAt": "now", + "files": [ + {"relPath": f"f{i}.md", "kind": "text", "summary": "s", "mtimeMs": i} + for i in range(10) + ], + }, + { + "projectId": "p2", "projectName": "Beta", "folderPath": "/b", + "lastScannedAt": "now", + "files": [{"relPath": "x.md", "kind": "text", "summary": "s", "mtimeMs": 1}], + }, + ] + with patch( + "app.core.deep_agent.execute_on_client", + new=AsyncMock(return_value={"projects": fake_response}), + ): + out = await build_brief_multi_project_manifest() + # Project 1 has 10 files, only top 5 by mtimeMs should appear + assert out.count("[p1]") <= 5 + # Project 2 has 1 file, must appear + assert "[p2]" in out or "Beta" in out +``` + +- [ ] **Step 2: Run — ImportError** + +Run: `cd api && pytest tests/test_manifest_injection.py::test_brief_multi_project_manifest_top_5_per_project -v` +Expected: ImportError + +- [ ] **Step 3: Implement builder** + +Add to `deep_agent.py`: + +```python +async def build_brief_multi_project_manifest() -> str: + """Build a compact multi-project manifest for the daily brief agent. + + Calls execute_on_client('list_projects_with_folder_manifests') and keeps + the top 5 most-recently-modified files per project. + """ + from app.core.ws_context import execute_on_client + try: + result = await execute_on_client( + action="list_projects_with_folder_manifests", + data={}, + ) + except Exception: + return "" + projects = (result or {}).get("projects") or [] + if not projects: + return "" + blocks: list[str] = [""] + for p in projects: + files = sorted(p.get("files", []), key=lambda f: f.get("mtimeMs", 0), reverse=True)[:5] + if not files: + continue + blocks.append(f"project: {p.get('projectName','?')} [{p.get('projectId','?')}]") + blocks.append(f" path: {p.get('folderPath','?')} (scanned {p.get('lastScannedAt','?')})") + for f in files: + blocks.append(f" - /{f['relPath']} [{f.get('kind','text')}] {f.get('summary','')}") + blocks.append("") + return "\n".join(blocks) +``` + +- [ ] **Step 4: Wire into `run_brief`** + +Find `run_brief` in `deep_agent.py`. Before the system prompt is finalised: + +```python +brief_manifest = await build_brief_multi_project_manifest() +system_prompt = system_prompt + ("\n\n" + brief_manifest if brief_manifest else "") + +from app.agents.folder_agent import FOLDER_TOOLS +tools = [...existing..., *FOLDER_TOOLS] +``` + +- [ ] **Step 5: Run tests, verify PASS** + +Run: `cd api && pytest tests/test_manifest_injection.py tests/test_brief_agent.py -v --tb=short` +Expected: all pass + +- [ ] **Step 6: Commit** + +```bash +git add api/app/core/deep_agent.py api/tests/test_manifest_injection.py +git commit -m "feat(api): multi-project folder manifest for daily brief" +``` + +--- + +## Phase D — Electron Schema, Scanner & tRPC + +### Task 15: Drizzle schema additions + +**Files:** +- Modify: `adiuvAI/src/main/db/schema.ts` + +- [ ] **Step 1: Extend `projects`** + +In `adiuvAI/src/main/db/schema.ts:12-19`, add columns: + +```typescript +export const projects = sqliteTable('projects', { + id: text('id').primaryKey(), + clientId: text('client_id'), + name: text('name').notNull(), + status: text('status', { enum: ['active', 'archived'] }).notNull().default('active'), + aiSummary: text('ai_summary'), + createdAt: integer('created_at', { mode: 'number' }).notNull(), + folderPath: text('folder_path'), + folderLastScannedAt: integer('folder_last_scanned_at', { mode: 'number' }), + folderLastScanStatus: text('folder_last_scan_status', { + enum: ['idle', 'scanning', 'error'], + }).default('idle'), + folderTotalFiles: integer('folder_total_files', { mode: 'number' }).notNull().default(0), +}); +``` + +- [ ] **Step 2: Add `projectFolderFiles` table** + +After the `notes` table definition: + +```typescript +export const projectFolderFiles = sqliteTable('project_folder_files', { + id: text('id').primaryKey(), + projectId: text('project_id').notNull(), + relativePath: text('relative_path').notNull(), + ext: text('ext').notNull(), + kind: text('kind', { enum: ['text', 'image', 'pdf', 'docx', 'csv', 'skipped', 'error'] }).notNull(), + sizeBytes: integer('size_bytes', { mode: 'number' }).notNull(), + mtimeMs: integer('mtime_ms', { mode: 'number' }).notNull(), + summary: text('summary'), + summaryUpdatedAt: integer('summary_updated_at', { mode: 'number' }), +}); + +export type ProjectFolderFile = InferSelectModel; +export type NewProjectFolderFile = InferInsertModel; +``` + +- [ ] **Step 3: Generate migration** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npx drizzle-kit generate` +Expected: new file under `src/main/db/migrations/` named `0008_*.sql`. + +- [ ] **Step 4: Inspect generated migration** + +Open the generated `0008_*.sql` and verify it contains: +- `ALTER TABLE projects ADD COLUMN folder_path TEXT;` +- `ALTER TABLE projects ADD COLUMN folder_last_scanned_at INTEGER;` +- `ALTER TABLE projects ADD COLUMN folder_last_scan_status TEXT DEFAULT 'idle';` +- `ALTER TABLE projects ADD COLUMN folder_total_files INTEGER DEFAULT 0 NOT NULL;` +- `CREATE TABLE project_folder_files (...);` + +If anything is wrong, edit the SQL by hand and re-save. Also verify the journal in `migrations/meta/_journal.json` includes the new entry. + +- [ ] **Step 5: Apply migration** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npx drizzle-kit push` +Expected: `Changes applied`. + +- [ ] **Step 6: Smoke check** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npm start` +- App boots. +- Open DevTools → run `await window.electronTRPC.sendMessage(...)` or just verify no migration error in the terminal. +- Quit. + +- [ ] **Step 7: Commit** + +```bash +git add adiuvAI/src/main/db/schema.ts adiuvAI/src/main/db/migrations/ +git commit -m "feat(adiuvAI): schema for project folder integration" +``` + +--- + +### Task 16: `files/constants.ts` + +**Files:** +- Create: `adiuvAI/src/main/files/constants.ts` + +- [ ] **Step 1: Write the file** + +```typescript +/** File-type whitelists & size caps for project folder indexing. */ + +export const TEXT_EXTS = new Set([ + '.md', '.txt', '.rst', '.adoc', + '.json', '.yaml', '.yml', '.toml', '.ini', '.csv', '.tsv', + '.html', '.htm', '.xml', + '.js', '.jsx', '.ts', '.tsx', '.mjs', '.cjs', + '.py', '.rb', '.go', '.rs', '.java', '.kt', '.swift', + '.c', '.h', '.cpp', '.hpp', '.cs', '.php', '.sh', '.ps1', + '.css', '.scss', '.sass', + '.sql', +]); + +export const IMAGE_EXTS = new Set(['.png', '.jpg', '.jpeg', '.webp']); + +export const PDF_EXTS = new Set(['.pdf']); +export const DOCX_EXTS = new Set(['.docx']); + +export const MAX_TEXT_FILE_BYTES = 1 * 1024 * 1024; // 1 MB +export const MAX_IMAGE_FILE_BYTES = 5 * 1024 * 1024; // 5 MB +export const INDEX_BATCH_SIZE = 5; +``` + +- [ ] **Step 2: Commit** + +```bash +git add adiuvAI/src/main/files/constants.ts +git commit -m "feat(adiuvAI): folder index constants" +``` + +--- + +### Task 17: `files/scanner.ts` — walk + filter + delta + +**Files:** +- Create: `adiuvAI/src/main/files/scanner.ts` + +- [ ] **Step 1: Write the file** + +```typescript +/** Filesystem scanner — walks a directory, filters by whitelist, computes delta vs DB manifest. */ + +import { readdir, stat } from 'node:fs/promises'; +import path from 'node:path'; +import { getDb } from '../db'; +import { projectFolderFiles } from '../db/schema'; +import { eq } from 'drizzle-orm'; +import { + TEXT_EXTS, IMAGE_EXTS, PDF_EXTS, DOCX_EXTS, + MAX_TEXT_FILE_BYTES, MAX_IMAGE_FILE_BYTES, +} from './constants'; + +export type FileKind = 'text' | 'image' | 'pdf' | 'docx' | 'skipped'; + +export interface ScannedFile { + relativePath: string; + ext: string; + kind: FileKind; + sizeBytes: number; + mtimeMs: number; +} + +export interface ScanDelta { + newFiles: ScannedFile[]; + changedFiles: ScannedFile[]; + unchangedCount: number; + deletedRelPaths: string[]; +} + +function classify(ext: string, sizeBytes: number): FileKind | null { + const e = ext.toLowerCase(); + if (TEXT_EXTS.has(e)) return sizeBytes <= MAX_TEXT_FILE_BYTES ? 'text' : 'skipped'; + if (IMAGE_EXTS.has(e)) return sizeBytes <= MAX_IMAGE_FILE_BYTES ? 'image' : 'skipped'; + if (PDF_EXTS.has(e)) return sizeBytes <= MAX_TEXT_FILE_BYTES ? 'pdf' : 'skipped'; + if (DOCX_EXTS.has(e)) return sizeBytes <= MAX_TEXT_FILE_BYTES ? 'docx' : 'skipped'; + return null; // not indexable +} + +async function walk(root: string): Promise { + const out: ScannedFile[] = []; + async function recurse(dir: string) { + let entries; + try { + entries = await readdir(dir, { withFileTypes: true }); + } catch { + return; // permission denied — skip silently + } + for (const e of entries) { + if (e.name.startsWith('.')) continue; // skip dot dirs / files + if (e.name === 'node_modules') continue; // common noise + const full = path.join(dir, e.name); + if (e.isDirectory()) { + await recurse(full); + } else if (e.isFile()) { + let s; + try { s = await stat(full); } catch { continue; } + const ext = path.extname(e.name); + const kind = classify(ext, s.size); + if (kind === null) continue; + out.push({ + relativePath: path.relative(root, full), + ext, + kind, + sizeBytes: s.size, + mtimeMs: Math.floor(s.mtimeMs), + }); + } + } + } + await recurse(root); + return out; +} + +export async function scanFolder(projectId: string, folderPath: string): Promise { + const scanned = await walk(folderPath); + const existing = getDb() + .select() + .from(projectFolderFiles) + .where(eq(projectFolderFiles.projectId, projectId)) + .all(); + + const existingMap = new Map(existing.map(r => [r.relativePath, r])); + const newFiles: ScannedFile[] = []; + const changedFiles: ScannedFile[] = []; + let unchanged = 0; + for (const f of scanned) { + const prev = existingMap.get(f.relativePath); + if (!prev) newFiles.push(f); + else if (prev.mtimeMs !== f.mtimeMs || prev.sizeBytes !== f.sizeBytes) changedFiles.push(f); + else unchanged++; + existingMap.delete(f.relativePath); + } + const deletedRelPaths = Array.from(existingMap.keys()); + return { newFiles, changedFiles, unchangedCount: unchanged, deletedRelPaths }; +} +``` + +- [ ] **Step 2: Verify it compiles** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npm run lint` +Expected: PASS (no lint errors in `scanner.ts`). + +- [ ] **Step 3: Commit** + +```bash +git add adiuvAI/src/main/files/scanner.ts +git commit -m "feat(adiuvAI): folder scanner with mtime delta" +``` + +--- + +### Task 18: `backend-client.ts` — index frame senders + +**Files:** +- Modify: `adiuvAI/src/main/api/backend-client.ts` + +- [ ] **Step 1: Read existing client** + +Run: `grep -n "sendHomeRequest\|sendFloatingRequest\|send\|onFrame\|class.*Client" adiuvAI/src/main/api/backend-client.ts | head -20` + +Note the WS send/receive pattern (likely a method `sendFrame(payload)` and an event-listener dispatch keyed off `frame.type`). + +- [ ] **Step 2: Add senders & receivers** + +In the `BackendClient` class, add public methods: + +```typescript +sendIndexSessionStart(sessionId: string, projectId: string, totalFiles: number): void { + this.sendFrame({ type: 'index_session_start', sessionId, projectId, totalFiles }); +} + +sendIndexFileBatch(sessionId: string, files: Array<{ + relPath: string; kind: string; content: string; ext?: string; mime?: string; + sizeBytes: number; mtimeMs: number; +}>): void { + this.sendFrame({ type: 'index_file_batch', sessionId, files }); +} + +sendIndexSessionCancel(sessionId: string): void { + this.sendFrame({ type: 'index_session_cancel', sessionId }); +} +``` + +Then extend the inbound dispatch (the place where it switches on `frame.type`) to forward `index_file_result`, `index_session_progress`, and `index_session_done` to listeners. + +```typescript +case 'index_file_result': +case 'index_session_progress': +case 'index_session_done': + this.emit('index', frame); + break; +``` + +(Use the existing event-emitter idiom — if `this.emit` doesn't exist, follow the existing pattern for dispatching v3 stream frames.) + +- [ ] **Step 3: Lint** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npm run lint` +Expected: PASS + +- [ ] **Step 4: Commit** + +```bash +git add adiuvAI/src/main/api/backend-client.ts +git commit -m "feat(adiuvAI): WS index session frame senders + dispatcher" +``` + +--- + +### Task 19: `files/indexer.ts` — session orchestration + +**Files:** +- Create: `adiuvAI/src/main/files/indexer.ts` + +- [ ] **Step 1: Write the file** + +```typescript +/** + * Folder index session orchestrator. + * + * Walks a folder via scanner.ts, sends batches over WS to the backend, applies + * returned summaries to projectFolderFiles, drives progress callbacks. + */ +import { readFile } from 'node:fs/promises'; +import path from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { getDb } from '../db'; +import { projects, projectFolderFiles } from '../db/schema'; +import { eq, and } from 'drizzle-orm'; +import { scanFolder, type ScannedFile } from './scanner'; +import { INDEX_BATCH_SIZE } from './constants'; +import { getBackendClient } from '../api/backend-client'; + +export interface IndexProgress { + sessionId: string; + processed: number; + total: number; + status: 'starting' | 'scanning' | 'cancelled' | 'completed' | 'quota_exceeded' | 'error'; + error?: string; +} + +export type ProgressListener = (p: IndexProgress) => void; + +async function readForIndex(folderPath: string, f: ScannedFile): Promise<{ content: string; mime?: string }> { + const abs = path.join(folderPath, f.relativePath); + if (f.kind === 'image') { + const buf = await readFile(abs); + const ext = f.ext.toLowerCase(); + const mime = ext === '.png' ? 'image/png' : ext === '.webp' ? 'image/webp' : 'image/jpeg'; + return { content: buf.toString('base64'), mime }; + } + if (f.kind === 'text') { + return { content: await readFile(abs, 'utf-8') }; + } + // pdf / docx: read as binary, base64. Server is responsible for extraction. + const buf = await readFile(abs); + return { content: buf.toString('base64') }; +} + +export async function startIndexSession( + projectId: string, + onProgress: ProgressListener, +): Promise<{ sessionId: string; cancel: () => void }> { + const sessionId = randomUUID(); + const db = getDb(); + + // Read folderPath from DB + const proj = db.select().from(projects).where(eq(projects.id, projectId)).get(); + if (!proj || !proj.folderPath) { + onProgress({ sessionId, processed: 0, total: 0, status: 'error', error: 'No folder linked' }); + return { sessionId, cancel: () => {} }; + } + + db.update(projects).set({ folderLastScanStatus: 'scanning' }).where(eq(projects.id, projectId)).run(); + onProgress({ sessionId, processed: 0, total: 0, status: 'scanning' }); + + const delta = await scanFolder(projectId, proj.folderPath); + const toIndex = [...delta.newFiles, ...delta.changedFiles]; + const total = toIndex.length; + + // Delete rows for deleted files + for (const rel of delta.deletedRelPaths) { + db.delete(projectFolderFiles).where( + and(eq(projectFolderFiles.projectId, projectId), eq(projectFolderFiles.relativePath, rel)) + ).run(); + } + + if (total === 0) { + db.update(projects).set({ + folderLastScanStatus: 'idle', + folderLastScannedAt: Date.now(), + folderTotalFiles: delta.unchangedCount, + }).where(eq(projects.id, projectId)).run(); + onProgress({ sessionId, processed: 0, total: 0, status: 'completed' }); + return { sessionId, cancel: () => {} }; + } + + const backend = getBackendClient(); + + // Listener + let processed = 0; + let cancelled = false; + const finalize = (status: IndexProgress['status'], error?: string) => { + db.update(projects).set({ + folderLastScanStatus: status === 'completed' ? 'idle' : status === 'cancelled' ? 'idle' : 'error', + folderLastScannedAt: Date.now(), + folderTotalFiles: delta.unchangedCount + processed, + }).where(eq(projects.id, projectId)).run(); + onProgress({ sessionId, processed, total, status, error }); + }; + + const handler = (frame: any) => { + if (frame.sessionId !== sessionId) return; + if (frame.type === 'index_file_result') { + const f = toIndex.find(x => x.relativePath === frame.relPath); + if (f && !frame.error) { + const now = Date.now(); + db.insert(projectFolderFiles).values({ + id: randomUUID(), + projectId, + relativePath: f.relativePath, + ext: f.ext, + kind: f.kind, + sizeBytes: f.sizeBytes, + mtimeMs: f.mtimeMs, + summary: frame.summary ?? null, + summaryUpdatedAt: now, + }).onConflictDoUpdate({ + target: [projectFolderFiles.projectId, projectFolderFiles.relativePath], + set: { + mtimeMs: f.mtimeMs, + sizeBytes: f.sizeBytes, + kind: f.kind, + summary: frame.summary ?? null, + summaryUpdatedAt: now, + }, + }).run(); + } + } else if (frame.type === 'index_session_progress') { + processed = frame.processed; + onProgress({ sessionId, processed, total, status: 'scanning' }); + } else if (frame.type === 'index_session_done') { + backend.off('index', handler); + finalize(frame.status === 'completed' ? 'completed' + : frame.status === 'cancelled' ? 'cancelled' + : frame.status === 'quota_exceeded' ? 'quota_exceeded' + : 'error'); + } + }; + backend.on('index', handler); + + backend.sendIndexSessionStart(sessionId, projectId, total); + + // Send batches + for (let i = 0; i < toIndex.length; i += INDEX_BATCH_SIZE) { + if (cancelled) break; + const batch = toIndex.slice(i, i + INDEX_BATCH_SIZE); + const payload = await Promise.all(batch.map(async f => { + const { content, mime } = await readForIndex(proj.folderPath!, f); + return { + relPath: f.relativePath, kind: f.kind, content, + ext: f.ext, mime, + sizeBytes: f.sizeBytes, mtimeMs: f.mtimeMs, + }; + })); + backend.sendIndexFileBatch(sessionId, payload); + } + + const cancel = () => { + cancelled = true; + backend.sendIndexSessionCancel(sessionId); + }; + return { sessionId, cancel }; +} +``` + +- [ ] **Step 2: Lint** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npm run lint` +Expected: PASS + +- [ ] **Step 3: Commit** + +```bash +git add adiuvAI/src/main/files/indexer.ts +git commit -m "feat(adiuvAI): WS index session orchestrator" +``` + +--- + +### Task 20: `router/projectFolders.ts` + register + +**Files:** +- Create: `adiuvAI/src/main/router/projectFolders.ts` +- Modify: `adiuvAI/src/main/router/index.ts` + +- [ ] **Step 1: Create router** + +```typescript +// adiuvAI/src/main/router/projectFolders.ts +import { initTRPC } from '@trpc/server'; +import { z } from 'zod'; +import { dialog } from 'electron'; +import { eq } from 'drizzle-orm'; +import { getDb } from '../db'; +import { projects, projectFolderFiles } from '../db/schema'; +import { startIndexSession, type IndexProgress } from '../files/indexer'; +import { getBackendClient } from '../api/backend-client'; +import type { TRPCContext } from '../ipc'; + +const t = initTRPC.context().create(); +const router = t.router; +const publicProcedure = t.procedure; + +// In-memory map of active sessions per projectId so we can cancel +const _active = new Map void; lastProgress: IndexProgress }>(); + +export const projectFoldersRouter = router({ + chooseFolder: publicProcedure.mutation(async () => { + const result = await dialog.showOpenDialog({ properties: ['openDirectory'] }); + if (result.canceled || result.filePaths.length === 0) return null; + return result.filePaths[0]; + }), + + link: publicProcedure + .input(z.object({ projectId: z.string(), folderPath: z.string() })) + .mutation(async ({ input }) => { + const db = getDb(); + db.update(projects) + .set({ folderPath: input.folderPath, folderLastScanStatus: 'idle', folderTotalFiles: 0 }) + .where(eq(projects.id, input.projectId)).run(); + return { ok: true }; + }), + + unlink: publicProcedure + .input(z.object({ projectId: z.string() })) + .mutation(async ({ input }) => { + const db = getDb(); + db.delete(projectFolderFiles).where(eq(projectFolderFiles.projectId, input.projectId)).run(); + db.update(projects).set({ + folderPath: null, folderLastScannedAt: null, + folderLastScanStatus: 'idle', folderTotalFiles: 0, + }).where(eq(projects.id, input.projectId)).run(); + return { ok: true }; + }), + + startScan: publicProcedure + .input(z.object({ projectId: z.string() })) + .mutation(async ({ input }) => { + const db = getDb(); + const proj = db.select().from(projects).where(eq(projects.id, input.projectId)).get(); + if (!proj?.folderPath) throw new Error('No folder linked'); + if (proj.folderLastScanStatus === 'scanning') throw new Error('Scan already in progress'); + + // Pre-flight: count indexable files (cheap walk) then call quota/check + // For simplicity here, send 0 (backend ignores when -1) and rely on + // mid-stream quota_exceeded. (Full pre-flight added in Task 24.) + + const session = await startIndexSession(input.projectId, (p) => { + const entry = _active.get(input.projectId); + if (entry) entry.lastProgress = p; + if (p.status === 'completed' || p.status === 'cancelled' || + p.status === 'quota_exceeded' || p.status === 'error') { + _active.delete(input.projectId); + } + }); + _active.set(input.projectId, { + cancel: session.cancel, + lastProgress: { sessionId: session.sessionId, processed: 0, total: 0, status: 'starting' }, + }); + return { sessionId: session.sessionId }; + }), + + cancelScan: publicProcedure + .input(z.object({ projectId: z.string() })) + .mutation(({ input }) => { + const entry = _active.get(input.projectId); + if (entry) entry.cancel(); + return { ok: true }; + }), + + getStatus: publicProcedure + .input(z.object({ projectId: z.string() })) + .query(({ input }) => { + const entry = _active.get(input.projectId); + return entry?.lastProgress ?? null; + }), + + listFiles: publicProcedure + .input(z.object({ projectId: z.string() })) + .query(({ input }) => { + return getDb() + .select() + .from(projectFolderFiles) + .where(eq(projectFolderFiles.projectId, input.projectId)) + .orderBy(projectFolderFiles.relativePath) + .all(); + }), +}); +``` + +- [ ] **Step 2: Register in `router/index.ts`** + +Edit `adiuvAI/src/main/router/index.ts` — add import and merge into `appRouter`: + +```typescript +import { projectFoldersRouter } from './projectFolders'; + +export const appRouter = router({ + // existing routers... + projectFolders: projectFoldersRouter, +}); +``` + +- [ ] **Step 3: Lint + type check by starting dev** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npm run lint` +Expected: PASS + +- [ ] **Step 4: Commit** + +```bash +git add adiuvAI/src/main/router/projectFolders.ts adiuvAI/src/main/router/index.ts +git commit -m "feat(adiuvAI): projectFolders tRPC router (link, unlink, scan, list)" +``` + +--- + +### Task 21: drizzle-executor actions + +**Files:** +- Modify: `adiuvAI/src/main/api/drizzle-executor.ts` + +- [ ] **Step 1: Add three new action handlers** + +Locate the `switch (call.action)` (or equivalent dispatch) in `drizzle-executor.ts`. Add three new cases: + +```typescript +case 'read_project_folder_manifest': { + const { projectId } = call.data as { projectId: string }; + const proj = getDb().select().from(projects).where(eq(projects.id, projectId)).get(); + if (!proj?.folderPath) return { folderPath: null, lastScannedAt: null, files: [] }; + const files = getDb() + .select({ + relPath: projectFolderFiles.relativePath, + kind: projectFolderFiles.kind, + summary: projectFolderFiles.summary, + mtimeMs: projectFolderFiles.mtimeMs, + }) + .from(projectFolderFiles) + .where(eq(projectFolderFiles.projectId, projectId)) + .all(); + // On-demand mtime check: if any tracked file's mtime is stale, fire-and-forget rescan. + // Returns the current (possibly stale) manifest immediately; the rescan updates rows + // for the next call. + if (proj.folderLastScanStatus !== 'scanning') { + void import('../files/scanner').then(async ({ scanFolder }) => { + const delta = await scanFolder(projectId, proj.folderPath!); + if (delta.newFiles.length > 0 || delta.changedFiles.length > 0 || delta.deletedRelPaths.length > 0) { + const { startIndexSession } = await import('../files/indexer'); + void startIndexSession(projectId, () => {}); + } + }).catch(() => {}); + } + return { + folderPath: proj.folderPath, + lastScannedAt: proj.folderLastScannedAt, + files, + }; +} + +case 'read_project_folder_file': { + const { projectId, relativePath } = call.data as { projectId: string; relativePath: string }; + // Re-check guards even though backend tool also guards + if (!relativePath || relativePath.includes('..') || path.isAbsolute(relativePath)) { + throw new ExecutorError('Access denied'); + } + const proj = getDb().select().from(projects).where(eq(projects.id, projectId)).get(); + if (!proj?.folderPath) return { content: '' }; + const abs = path.join(proj.folderPath, relativePath); + // Confine to folderPath + if (!path.resolve(abs).startsWith(path.resolve(proj.folderPath))) { + throw new ExecutorError('Access denied'); + } + try { + const stat = await import('node:fs/promises').then(m => m.stat(abs)); + if (stat.size > MAX_READ_SIZE_BYTES) { + return { content: (await fs.promises.readFile(abs, 'utf-8')).slice(0, MAX_READ_SIZE_BYTES) }; + } + // Image → base64; else utf-8 + const ext = path.extname(relativePath).toLowerCase(); + if (['.png', '.jpg', '.jpeg', '.webp'].includes(ext)) { + return { content: (await fs.promises.readFile(abs)).toString('base64') }; + } + return { content: await fs.promises.readFile(abs, 'utf-8') }; + } catch { + return { content: '' }; + } +} + +case 'list_projects_with_folder_manifests': { + const projs = getDb() + .select() + .from(projects) + .where(/* folderPath not null — drizzle: */ sql`${projects.folderPath} IS NOT NULL`) + .all(); + const out = [] as Array; + for (const p of projs) { + const files = getDb() + .select({ + relPath: projectFolderFiles.relativePath, + kind: projectFolderFiles.kind, + summary: projectFolderFiles.summary, + mtimeMs: projectFolderFiles.mtimeMs, + }) + .from(projectFolderFiles) + .where(eq(projectFolderFiles.projectId, p.id)) + .all(); + out.push({ + projectId: p.id, + projectName: p.name, + folderPath: p.folderPath, + lastScannedAt: p.folderLastScannedAt, + files, + }); + } + return { projects: out }; +} +``` + +Also add to `TABLE_REGISTRY` if the existing dispatch routes table-by-name operations: + +```typescript +const TABLE_REGISTRY = { + // existing... + projectFolderFiles, +}; +``` + +And import `projectFolderFiles` and `sql` at top of file. + +- [ ] **Step 2: Lint** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npm run lint` +Expected: PASS + +- [ ] **Step 3: Commit** + +```bash +git add adiuvAI/src/main/api/drizzle-executor.ts +git commit -m "feat(adiuvAI): drizzle-executor folder manifest + scoped read actions" +``` + +--- + +### Task 22: `files/daily-rescan.ts` + wire on app `ready` + +**Files:** +- Create: `adiuvAI/src/main/files/daily-rescan.ts` +- Modify: `adiuvAI/src/main/index.ts` + +- [ ] **Step 1: Create the rescan module** + +```typescript +// adiuvAI/src/main/files/daily-rescan.ts +import { getDb } from '../db'; +import { projects } from '../db/schema'; +import { sql, and, isNotNull, lt } from 'drizzle-orm'; +import { startIndexSession } from './indexer'; + +const ONE_DAY_MS = 24 * 60 * 60 * 1000; + +export async function runDailyRescan(): Promise { + const cutoff = Date.now() - ONE_DAY_MS; + const stale = getDb() + .select() + .from(projects) + .where(and( + isNotNull(projects.folderPath), + sql`(${projects.folderLastScannedAt} IS NULL OR ${projects.folderLastScannedAt} < ${cutoff})`, + )) + .all(); + for (const p of stale) { + if (p.folderLastScanStatus === 'scanning') continue; + // Fire-and-forget; no UI listener. + void startIndexSession(p.id, () => {}); + } +} +``` + +- [ ] **Step 2: Wire in `index.ts`** + +In `adiuvAI/src/main/index.ts`, find the `app.whenReady().then(...)` block. Append: + +```typescript +import { runDailyRescan } from './files/daily-rescan'; + +// ...inside whenReady, after window creation and auth init: +setTimeout(() => { void runDailyRescan(); }, 10_000); // delay so WS is up +``` + +- [ ] **Step 3: Lint** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npm run lint` +Expected: PASS + +- [ ] **Step 4: Commit** + +```bash +git add adiuvAI/src/main/files/daily-rescan.ts adiuvAI/src/main/index.ts +git commit -m "feat(adiuvAI): daily auto-rescan of stale folder links" +``` + +--- + +## Phase E — Renderer UI + +### Task 23: i18n keys + +**Files:** +- Modify: `adiuvAI/src/renderer/locales/en/translation.json` +- Modify: `adiuvAI/src/renderer/locales/it/translation.json` +- Modify: `adiuvAI/src/renderer/locales/es/translation.json` +- Modify: `adiuvAI/src/renderer/locales/fr/translation.json` +- Modify: `adiuvAI/src/renderer/locales/de/translation.json` + +- [ ] **Step 1: Add the same keyset to all 5 locales** + +For each `translation.json`, add under `projects`: + +```json +"folder": { + "title": "Files", + "linkCta": "Link folder", + "browse": "Browse", + "rescan": "Rescan", + "unlink": "Unlink", + "filesCount_one": "{{count}} file", + "filesCount_other": "{{count}} files", + "lastScanned": "last scanned {{relative}}", + "scanning": "indexing {{processed}}/{{total}}", + "scanFailed": "Scan failed", + "empty": { + "title": "Link a project folder", + "description": "Connect a local folder so AI agents can read its files when answering questions about this project.", + "cta": "Choose folder…" + }, + "errors": { + "tooBig": "Folder too big for {{tier}} plan — max {{count}} files", + "monthlyExhausted": "Monthly token budget exhausted (resets {{date}})", + "notFound": "Folder not found: {{path}}" + }, + "webOnlyTooltip": "Folder linking available in desktop app" +}, +``` + +Translate the values into each locale's language. Keys stay identical across files. + +- [ ] **Step 2: Commit** + +```bash +git add adiuvAI/src/renderer/locales/ +git commit -m "i18n: projects.folder keys in all 5 locales" +``` + +--- + +### Task 24: `FolderChip.tsx` + +**Files:** +- Create: `adiuvAI/src/renderer/components/projects/folder/FolderChip.tsx` + +- [ ] **Step 1: Write component** + +```tsx +import { useTranslation } from 'react-i18next'; +import { Folder, Sparkles } from 'lucide-react'; +import { cn } from '@/lib/utils'; +import { formatDistanceToNow } from 'date-fns'; + +interface FolderChipProps { + projectId: string; + folderPath: string | null; + totalFiles: number; + lastScannedAt: number | null; + scanStatus: 'idle' | 'scanning' | 'error' | null; + scanProgress?: { processed: number; total: number } | null; + onClick: () => void; +} + +export function FolderChip({ + folderPath, totalFiles, lastScannedAt, scanStatus, scanProgress, onClick, +}: FolderChipProps) { + const { t } = useTranslation(); + + if (!folderPath) { + return ( + + ); + } + + if (scanStatus === 'scanning' && scanProgress) { + return ( + + ); + } + + if (scanStatus === 'error') { + return ( + + ); + } + + const relative = lastScannedAt + ? formatDistanceToNow(new Date(lastScannedAt), { addSuffix: true }) + : '—'; + return ( + + ); +} +``` + +- [ ] **Step 2: Commit** + +```bash +git add adiuvAI/src/renderer/components/projects/folder/FolderChip.tsx +git commit -m "feat(adiuvAI): FolderChip component" +``` + +--- + +### Task 25: `FolderLinkCard.tsx` + +**Files:** +- Create: `adiuvAI/src/renderer/components/projects/folder/FolderLinkCard.tsx` + +- [ ] **Step 1: Write component** + +```tsx +import { useTranslation } from 'react-i18next'; +import { Folder } from 'lucide-react'; +import { Button } from '@/components/ui/button'; +import { trpc } from '@/lib/trpc'; +import { useNotify } from '@/hooks/useNotify'; +import { formatDistanceToNow } from 'date-fns'; + +interface FolderLinkCardProps { + projectId: string; + folderPath: string; + totalFiles: number; + lastScannedAt: number | null; + scanStatus: 'idle' | 'scanning' | 'error' | null; + onUnlinkRequested: () => void; +} + +export function FolderLinkCard({ + projectId, folderPath, totalFiles, lastScannedAt, scanStatus, onUnlinkRequested, +}: FolderLinkCardProps) { + const { t } = useTranslation(); + const notify = useNotify(); + const utils = trpc.useUtils(); + const startScan = trpc.projectFolders.startScan.useMutation({ + onSuccess: () => utils.projects.get.invalidate({ id: projectId }), + onError: (err) => notify.error(err.message), + }); + + return ( +
+
+ +
+
+
{t('projects.folder.title')}
+
{folderPath}
+
+ {t('projects.folder.filesCount', { count: totalFiles })} + {lastScannedAt && ( + <> · {t('projects.folder.lastScanned', { relative: formatDistanceToNow(new Date(lastScannedAt), { addSuffix: true }) })} + )} +
+
+
+ + +
+
+ ); +} +``` + +- [ ] **Step 2: Commit** + +```bash +git add adiuvAI/src/renderer/components/projects/folder/FolderLinkCard.tsx +git commit -m "feat(adiuvAI): FolderLinkCard component" +``` + +--- + +### Task 26: `FolderFileList.tsx` + +**Files:** +- Create: `adiuvAI/src/renderer/components/projects/folder/FolderFileList.tsx` + +- [ ] **Step 1: Write component** + +```tsx +import { useState, useMemo } from 'react'; +import { trpc } from '@/lib/trpc'; +import { Skeleton } from '@/components/ui/skeleton'; +import { cn } from '@/lib/utils'; + +interface FolderFileListProps { + projectId: string; +} + +type Filter = 'all' | 'text' | 'image' | 'pdf' | 'docx'; + +const FILTERS: Filter[] = ['all', 'text', 'image', 'pdf', 'docx']; + +export function FolderFileList({ projectId }: FolderFileListProps) { + const [filter, setFilter] = useState('all'); + const { data, isLoading } = trpc.projectFolders.listFiles.useQuery({ projectId }); + const items = useMemo(() => { + if (!data) return []; + if (filter === 'all') return data; + return data.filter(f => f.kind === filter); + }, [data, filter]); + + if (isLoading) { + return
{[0,1,2].map(i => )}
; + } + + return ( +
+
+ {FILTERS.map(f => ( + + ))} +
+
    + {items.map(f => ( +
  • +
    {f.relativePath}
    + {f.summary && ( +
    {f.summary}
    + )} +
  • + ))} +
+
+ ); +} +``` + +- [ ] **Step 2: Commit** + +```bash +git add adiuvAI/src/renderer/components/projects/folder/FolderFileList.tsx +git commit -m "feat(adiuvAI): FolderFileList with kind filter" +``` + +--- + +### Task 27: `FolderUnlinkDialog.tsx` + +**Files:** +- Create: `adiuvAI/src/renderer/components/projects/folder/FolderUnlinkDialog.tsx` + +- [ ] **Step 1: Write component** + +```tsx +import { useTranslation } from 'react-i18next'; +import { + Dialog, DialogContent, DialogHeader, DialogTitle, DialogDescription, DialogFooter, +} from '@/components/ui/dialog'; +import { Button } from '@/components/ui/button'; +import { trpc } from '@/lib/trpc'; +import { useNotify } from '@/hooks/useNotify'; + +interface FolderUnlinkDialogProps { + projectId: string; + open: boolean; + onOpenChange: (open: boolean) => void; +} + +export function FolderUnlinkDialog({ projectId, open, onOpenChange }: FolderUnlinkDialogProps) { + const { t } = useTranslation(); + const notify = useNotify(); + const utils = trpc.useUtils(); + const unlink = trpc.projectFolders.unlink.useMutation({ + onSuccess: () => { + utils.projects.get.invalidate({ id: projectId }); + utils.projectFolders.listFiles.invalidate({ projectId }); + onOpenChange(false); + }, + onError: (err) => notify.error(err.message), + }); + + return ( + + + + {t('projects.folder.unlink')} + + {t('common.deleteTitle')} + + + + + + + + + ); +} +``` + +- [ ] **Step 2: Commit** + +```bash +git add adiuvAI/src/renderer/components/projects/folder/FolderUnlinkDialog.tsx +git commit -m "feat(adiuvAI): FolderUnlinkDialog" +``` + +--- + +### Task 28: `FilesSection.tsx` + +**Files:** +- Create: `adiuvAI/src/renderer/components/projects/folder/FilesSection.tsx` + +- [ ] **Step 1: Write component** + +```tsx +import { useState } from 'react'; +import { useTranslation } from 'react-i18next'; +import { Sparkles } from 'lucide-react'; +import { trpc } from '@/lib/trpc'; +import { Button } from '@/components/ui/button'; +import { + Empty, EmptyHeader, EmptyMedia, EmptyTitle, EmptyDescription, +} from '@/components/ui/empty'; +import { useNotify } from '@/hooks/useNotify'; +import { usePlatform } from '@/lib/platform'; +import { FolderLinkCard } from './FolderLinkCard'; +import { FolderFileList } from './FolderFileList'; +import { FolderUnlinkDialog } from './FolderUnlinkDialog'; + +interface FilesSectionProps { + projectId: string; + folderPath: string | null; + totalFiles: number; + lastScannedAt: number | null; + scanStatus: 'idle' | 'scanning' | 'error' | null; +} + +export function FilesSection({ + projectId, folderPath, totalFiles, lastScannedAt, scanStatus, +}: FilesSectionProps) { + const { t } = useTranslation(); + const notify = useNotify(); + const platform = usePlatform(); + const [unlinkOpen, setUnlinkOpen] = useState(false); + const utils = trpc.useUtils(); + const chooseFolder = trpc.projectFolders.chooseFolder.useMutation(); + const link = trpc.projectFolders.link.useMutation({ + onSuccess: () => utils.projects.get.invalidate({ id: projectId }), + onError: (err) => notify.error(err.message), + }); + + const handleChoose = async () => { + const folderPath = await chooseFolder.mutateAsync(); + if (folderPath) { + await link.mutateAsync({ projectId, folderPath }); + // Kick first scan + await utils.client.projectFolders.startScan.mutate({ projectId }); + } + }; + + if (!platform.isElectron) { + return ( +
+ {t('projects.folder.webOnlyTooltip')} +
+ ); + } + + if (!folderPath) { + return ( + + + + {t('projects.folder.empty.title')} + {t('projects.folder.empty.description')} + + + + ); + } + + return ( +
+ setUnlinkOpen(true)} + /> + + +
+ ); +} +``` + +- [ ] **Step 2: Commit** + +```bash +git add adiuvAI/src/renderer/components/projects/folder/FilesSection.tsx +git commit -m "feat(adiuvAI): FilesSection orchestrator" +``` + +--- + +### Task 29: Wire `'files'` section into ProjectTabBar + +**Files:** +- Modify: `adiuvAI/src/renderer/components/projects/ProjectTabBar.tsx:6-7` + +- [ ] **Step 1: Extend SECTIONS** + +Change: + +```typescript +export const SECTIONS = ['overview', 'timeline', 'tasks', 'notes'] as const; +``` + +to: + +```typescript +export const SECTIONS = ['overview', 'timeline', 'tasks', 'notes', 'files'] as const; +``` + +- [ ] **Step 2: Add label** + +Inside `TAB_LABELS` (around line 81): + +```typescript +files: t('projects.folder.title'), +``` + +- [ ] **Step 3: Commit** + +```bash +git add adiuvAI/src/renderer/components/projects/ProjectTabBar.tsx +git commit -m "feat(adiuvAI): add 'files' tab to ProjectTabBar" +``` + +--- + +### Task 30: Wire chip + section into `ProjectDetail.tsx` + +**Files:** +- Modify: `adiuvAI/src/renderer/components/projects/ProjectDetail.tsx` + +- [ ] **Step 1: Import additions** + +At the top of the file: + +```typescript +import { FolderChip } from './folder/FolderChip'; +import { FilesSection } from './folder/FilesSection'; +``` + +- [ ] **Step 2: Add `filesRef` and register it in `sectionRefs`** + +Inside `ProjectDetail`: + +```typescript +const filesRef = useRef(null); +const sectionRefs: Record> = useMemo(() => ({ + overview: summaryRef, + timeline: timelineRef, + tasks: tasksRef, + notes: notesRef, + files: filesRef, +}), []); +``` + +- [ ] **Step 3: Subscribe to live scan status** + +Inside `ProjectDetail`, add: + +```typescript +const { data: scanStatus } = trpc.projectFolders.getStatus.useQuery( + { projectId }, + { refetchInterval: (data) => data?.status === 'scanning' ? 1000 : false }, +); +``` + +- [ ] **Step 4: Render the FolderChip in the hero region** + +Inside the hero JSX (`
...`), near the title block, add a row that includes ``: + +```tsx + {/* scroll to files; reuse pattern from other tabs */}} +/> +``` + +For the click handler, copy the pattern from existing `` `scrollToSection` call sites — typically by setting a query param or scrolling to `filesRef.current`. + +- [ ] **Step 5: Render the section body** + +Below the existing `
` block, add: + +```tsx +
+ +
+``` + +- [ ] **Step 6: Manual smoke test** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npm start` + +- App boots. +- Open any project. Verify "Files" tab appears in tab bar; chip appears in hero (dashed CTA when unlinked). +- Click "Files" tab → empty state renders. +- Click "Choose folder…" → OS dialog opens. Pick a small folder with mixed file types. +- Folder is linked; chip switches to "indexing N/M"; progress polls every second; finishes; chip shows "N files · just now". +- Tab body shows FolderLinkCard + filterable file list. +- Click Rescan; status flips back and resolves. +- Click Unlink → dialog confirms; chip returns to "Link folder". + +- [ ] **Step 7: Commit** + +```bash +git add adiuvAI/src/renderer/components/projects/ProjectDetail.tsx +git commit -m "feat(adiuvAI): wire FolderChip + FilesSection into ProjectDetail" +``` + +--- + +### Task 31: Pre-flight quota check + error toasts + +**Files:** +- Modify: `adiuvAI/src/main/router/projectFolders.ts` +- Modify: `adiuvAI/src/renderer/components/projects/folder/FilesSection.tsx` + +- [ ] **Step 1: Add pre-flight in router** + +In `startScan` mutation, before calling `startIndexSession`, count files via `scanFolder` and call backend `/api/v1/billing/quota/check`: + +```typescript +import { scanFolder } from '../files/scanner'; +import { getAuthManager } from '../auth/auth-manager'; + +// inside startScan: +const delta = await scanFolder(input.projectId, proj.folderPath); +const estimated = delta.newFiles.length + delta.changedFiles.length + delta.unchangedCount; + +const auth = getAuthManager(); +const token = await auth.getAccessToken(); +const apiBase = process.env.API_BASE_URL ?? 'http://localhost:8000'; // pull from existing config +const res = await fetch(`${apiBase}/api/v1/billing/quota/check`, { + method: 'POST', + headers: { + 'content-type': 'application/json', + authorization: `Bearer ${token}`, + }, + body: JSON.stringify({ feature: 'folder_index', estimated_files: estimated }), +}); +if (res.status === 402) { + const body = await res.json(); + throw new Error(`QUOTA:${body.detail.reason}:${body.detail.message}`); +} +``` + +(Reuse whatever existing pattern the codebase already has for authenticated fetches from the main process — `BackendClient` may expose an HTTP helper; if so, use that instead of raw fetch.) + +- [ ] **Step 2: Surface quota errors in `FilesSection.tsx`** + +Wrap the `handleChoose` call: + +```typescript +try { + await utils.client.projectFolders.startScan.mutate({ projectId }); +} catch (err: any) { + const msg = err?.message ?? ''; + if (msg.startsWith('QUOTA:max_files')) { + notify.error(t('projects.folder.errors.tooBig', { tier: 'free', count: 200 })); + } else if (msg.startsWith('QUOTA:monthly_tokens')) { + notify.error(t('projects.folder.errors.monthlyExhausted', { date: 'next month' })); + } else { + notify.error(msg); + } +} +``` + +- [ ] **Step 3: Lint + manual smoke** + +Run: `cd adiuvAI && source ~/.nvm/nvm.sh && npm run lint && npm start` + +Smoke-test with a Free-tier account: link a folder with >200 files → toast "Folder too big for free plan — max 200 files". + +- [ ] **Step 4: Commit** + +```bash +git add adiuvAI/src/main/router/projectFolders.ts adiuvAI/src/renderer/components/projects/folder/FilesSection.tsx +git commit -m "feat(adiuvAI): pre-flight quota check + error toasts for folder integration" +``` + +--- + +## Phase F — End-to-End Verification + +### Task 32: Manual smoke checklist + +**Files:** none + +- [ ] **Step 1: Walk the full happy path on a real folder** + +With dev server running: + +1. Create a new project. Open it. +2. Hero shows "📁 Link folder" chip. Click → folder dialog. Choose a folder with ~20 mixed files (some `.md`, `.png`, `.pdf`, one `node_modules` dir). +3. Verify dotfiles + `node_modules` are skipped (look at `projectFolderFiles` rows via DevTools tRPC query). +4. Chip shows "indexing N/M" with spinner. File list rows appear as summaries come in. +5. Final chip: "N files · just now". +6. Open the task brief for any task in this project — its system prompt now contains a `` block with file summaries (verify in Langfuse trace). +7. Ask the agent "what is in the kickoff doc?" — agent should call `read_project_folder_file` and answer with content. + +- [ ] **Step 2: Delta + rescan** + +Modify one `.md` in the linked folder, add one `.png`, delete one file. +Click Rescan. Verify: +- Only the modified `.md` and the new `.png` show up in `index_file_result` frames. +- The deleted file's row is removed from `projectFolderFiles`. +- Chip ticks the new count. + +- [ ] **Step 3: Cancel** + +Link a folder with many files. Mid-scan, navigate away or click Unlink. Confirm: +- Scan stops. +- Status row in `projects` flips to `'idle'` (after cancel). +- No background errors logged. + +- [ ] **Step 4: Quota** + +Switch the test user to `free` tier in the DB. Try to link a folder with >200 files → toast appears, link rejected. + +- [ ] **Step 5: Traversal guard** + +In a manual agent run, prompt the agent to call `read_project_folder_file` with `relative_path="../../etc/passwd"`. Tool must return `"Access denied"`. + +- [ ] **Step 6: Document any issues found** + +If any step fails, file an issue with reproduction steps and add a follow-up task to the plan. + +- [ ] **Step 7: Commit the verification log (optional)** + +If you keep a verification log file, commit it now. Otherwise skip. + +--- + +## Done Criteria + +- All API tests pass (`pytest tests/test_folder_*.py tests/test_ws_index_session.py tests/test_manifest_injection.py`). +- Electron app boots without migration errors. +- Manual smoke checklist (Task 32) passes end to end. +- No lint regressions (`npm run lint`). +- No new console errors on app startup. + +--- + +## Out of Scope + +- Token-usage display UI (recorded backend-side; Settings page deferred). +- Multi-folder per project. +- Live file watcher (chokidar). +- Phase-2 wiki tier. +- Web SPA support (Files tab disabled message only). +- File editing from adiuvAI.