Keep only 4.1 (first reply contains question) as automated eval.
Multi-turn cases (4.2–4.5) are non-deterministic and tested manually
with results tracked in Langfuse.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Langfuse V3 does not accept user_id/session_id on observation-level calls.
Moved to metadata dict in agent_runner, deep_agent, and agent_setup.
refactor(tests): fixture-based pattern for agent_runner_v2 eval tests
- cases.yaml + data/ fixtures under tests/fixtures/agent_runner_v2/
- pytest_generate_tests parametrizes test_eval_runner from YAML
- _resolve_projects() handles symbolic names and inline dicts
- _evaluate_case() centralizes all assertion logic
- --runner-dir CLI option for custom fixture folders
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lf.trace() and lf.score(trace_id=...) are V2 API removed in V3.
V3 pattern:
lf.start_as_current_observation(name=...) as context manager → obs
obs.score(name=..., value=...)
contextlib.nullcontext() when lf is None so structure stays the same
Updated tests 2.1–2.7 in test_agent_runner_v2.py accordingly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
I test del preprocessor sono deterministici — nessun LLM coinvolto,
nessuno score da tracciare.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
file: serve sia come path da leggere che come nome passato a detect_content_type.
Non c'è motivo di averli separati.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- conftest.py: registra --preprocess-dir via pytest_addoption
- test_preprocessors.py: usa pytest_generate_tests per leggere i casi
a collection time con accesso a config; _content e _fixtures_dir
accettano path dinamico
Usage: pytest tests/test_preprocessors.py --preprocess-dir /my/folder
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
YAML: rimosse op/description/score_name/assertions block — ora detect/process
come chiave diretta, assertions piatte sullo stesso livello del caso.
Runner: eliminato _run_assertions engine, assertions inline in test_preprocess.
Riduzione da ~170 a ~75 righe totali tra YAML + test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scoring is only meaningful for LLM-backed steps. Preprocess tests are
deterministic Python, so scores add no value. Kept only for detect tests.
- test_preprocess: drop _lf_score call, simplify _run_assertions return type
- cases.yaml: remove score_name from all op=preprocess entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root causes fixed:
1. PROJECT_TOOLS removed from Step 2 tool set — project assignment is now
exclusively handled by the runner in code, never by the LLM.
2. When Step 1 returns "new", runner calls execute_on_client insert/projects
directly (before Step 2), gets the created id, and passes it as context.
3. Newly created projects are appended to the local `projects` list so that
subsequent files in the same run can match to them via Step 1 — prevents
one project per file when multiple files share the same topic.
Also add tests/test_classify_file.py with pytest cases for _classify_file
and a CLI runner: python -m tests.test_classify_file <file> [project...]
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Code bugs fixed:
- checkpoint_agent.py, project_agent.py, note_agent.py: add missing
'import json' (used in handle() for context serialization)
Test fixes:
- test_agents.py: add autouse ws_executor fixture that sets a fake
execute_on_client so tools can run in unit tests without a WS session
- Rewrite all TestXxxAgentTools tests: patch execute_on_client per-test,
assert on call_args (what payload was sent to the client) and on the
formatted string return value — matching actual tool behavior
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- device_ws.py: dispatch home_request/popup_request to HomeFormatter/PopupFormatter
via async tasks; each request gets a UUID request_id for frame correlation
- chat.py: remove chat_stream WS endpoint (superseded by unified device WS);
keep POST /chat REST fallback unchanged
- 5 new integration tests pass; all 22 existing device_ws tests still pass
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- orchestrate_v3(user_id, message, context): classifies intent, returns
(agent_name, agent_instance) — caller drives execution
- orchestrate_v3_stream(user_id, message, context): yields (agent_name, token)
pairs; first yield is always (agent_name, "") as a domain-detection signal
- ChatAgent.handle_stream(): default implementation yields handle() result as
one chunk; subclasses override for true token-level streaming
- Fix stale test_orchestrator.py assertions that expected a JSON final frame
that orchestrate_stream never emitted
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ChatAgent.__init__: adds tool_results: list[dict] = []
- _tool_loop: wraps execution in a result collector; populates
self.tool_results with raw execute_on_client dicts after each run
- _tool_loop_stream: streaming variant — uses ainvoke for tool-call
iterations, llm.astream() for the final answer; same result capture
- ws_context.py: adds _tool_result_collector ContextVar +
set/clear helpers; execute_on_client appends to collector when set
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replaced direct instantiation of ChatOpenAI with a centralized get_llm function in CheckpointAgent, NoteAgent, ProjectAgent, and TaskAgent.
- Introduced a new llm.py module to handle LLM model instantiation and API key management.
- Updated settings.py to include LLM_MODEL and LLM_ROUTER_MODEL configurations.
- Modified orchestrator.py to use get_router_llm for intent classification.
- Updated requirements.txt to include litellm for LLM management.
- Adjusted tests to mock get_llm instead of ChatOpenAI directly.
- Updated `TestModuleSingletons` in `test_execution_plan.py` to reflect new agent templates and playbook names.
- Changed assertions in playbook tests to match updated templates and agents.
- Introduced `test_storage.py` to cover the storage layer, including encryption, BlobStore, and VectorStore functionalities.
- Added tests for S3 interactions, ensuring upload, download, delete, and list operations work as expected.
- Implemented mock tests for Pinecone and Qdrant vector stores to validate upsert, search, and delete operations.
