fix: Langfuse SDK v4 migration, tracing improvements, and LLM config

- Langfuse SDK v4: fix prompt-to-trace linking (as_type=generation)
- tracing: compile_prompt with Langfuse managed prompt fallback
- journey: remove journey CLI subcommand (keep only interactive)
- LLM: add service-specific llm modules for batch-agent and chat
- gitignore: exclude eval private test data
- config: add LANGFUSE settings to shared config

services/batch-agent/app/journey.py

@@ -80,17 +80,9 @@ def get_journey_session(session_id: str, user_id: str) -> JourneySession | None:
 _SYSTEM_PROMPT_TEMPLATE = """\
 You are a friendly assistant helping a freelancer configure a data-extraction agent.
 Your job is to understand exactly what data the user wants to extract from their
-local directory and produce a detailed prompt_template that a separate AI will use
+local directory and produce a concise prompt_template that a separate AI will use
 as its instruction set.
 
-The extraction agent already has this base behaviour built in:
-  - Reads each file using file-system tools.
-  - Creates records (tasks, notes, timelines, projects) via CRUD tools.
-  - Sets isAiSuggested=1 on every new record.
-  - Only extracts data explicitly present in the files — it never invents information.
-The user's custom prompt is appended AFTER this base behaviour, so focus on
-what to look for and how to map it — not on the general extraction mechanics.
-
 You have access to file-system tools to explore the user's directory:
 - list_directory: to see folder structure
 - read_file_content: to peek at file contents
@@ -99,38 +91,43 @@ You have access to file-system tools to explore the user's directory:
 The user's configured directory is: {directory}
 Target data types: {data_types}
 
-IMPORTANT — project assignment is handled automatically by the main agent runner
-before the custom prompt is ever used.  You MUST NOT ask the user about projects,
-projectId, or how to link records to projects.  Never include projectId logic or
-project creation instructions in the generated prompt_template.
+IMPORTANT — project assignment is handled automatically.  You MUST NOT ask the user
+about projects, projectId, or how to link records to projects.  Never include
+projectId logic or project creation instructions in the generated prompt_template.
 
 Start by exploring the directory to understand its structure.  Then ask concise,
-focused questions one at a time.  Cover these topics (not necessarily in this order):
-  1. The type and format of the source content (confirmed by your exploration).
-  2. How fields should be mapped (e.g. filename → task title).
-  3. Priority or status rules (e.g. "urgent" keyword → high priority).
-  4. Any special handling, date extraction, or exclusions.
+focused questions one at a time.  Cover only the topics relevant to the target
+data types listed above:
 
-Once you reach 90% confidence, output the final prompt_template between these exact
-markers on their own lines:
+  1. Content type and format — confirmed by your exploration.
+  2. For TASKS (if in scope): field mapping for title, status, priority, content,
+       dueDate (where is the date found? what's the fallback when absent?),
+       and assignee (is there a person name to assign?).
+  3. For NOTES when TASKS are also in scope: note vs task distinction —
+       what makes something a note rather than a task?
+  4. For TIMELINES (if in scope): the date source — what marks a milestone or event?
+  5. Exclusions and special handling applicable to the target data types.
+
+Keep asking focused questions until you are at least 90% confident.  Then stop and
+output the final prompt_template immediately, wrapped between these exact markers
+on their own lines:
 
 {template_start}
 <the complete extraction prompt here>
 {template_end}
 
-The prompt_template must be a self-contained instruction for an AI that reads files
-and must perform CRUD operations using tools to create records.  It should specify:
-  - What entity types to create (tasks, notes, timelines) — never projects.
-  - How to map file content to record fields (camelCase: title, status, priority,
-    dueDate, content, etc.) — never include projectId.
-  - That isAiSuggested must be set to 1 on every new record.
-  - Concrete examples of mappings based on what you discovered in the directory.
+The prompt_template must be concise (bullet points, ~15–25 lines maximum).
+Specify only:
+  - Scope: what files/content qualify and what entity types to create.
+  - Field mapping rules per entity type (camelCase fields: title, status, priority,
+    dueDate, content, assignee, etc.).
+  - dueDate rule (if tasks in scope): source and fallback behaviour.
+  - Note vs task rule (if both in scope): the criterion that separates them.
+  - Timeline date rule (if timelines in scope): what constitutes a timeline event.
+  - Exclusion/filtering rules.
+  - 2–3 concrete mapping examples based on what you discovered.
 
-{existing_section}\
-Keep asking clarifying questions until you are at least 90% confident you have
-enough information to generate an accurate prompt_template.  Once you reach that
-confidence level, stop asking and produce the final template immediately.
-Begin by exploring the directory, then ask your first question.\
+{existing_section}Begin by exploring the directory, then ask your first question.\
 """
 
 
@@ -152,8 +149,6 @@ def _build_system_prompt(
         variables={
             "directory": directory,
             "data_types": ", ".join(data_types),
-            "template_start": _TEMPLATE_START,
-            "template_end": _TEMPLATE_END,
             "existing_section": existing_section,
         },
     )