bruno/luzia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor cockpit to use DockerTmuxController pattern

Browse Source 
Based on claude-code-tools TmuxCLIController, this refactor:

- Added DockerTmuxController class for robust tmux session management
- Implements send_keys() with configurable delay_enter
- Implements capture_pane() for output retrieval
- Implements wait_for_prompt() for pattern-based completion detection
- Implements wait_for_idle() for content-hash-based idle detection
- Implements wait_for_shell_prompt() for shell prompt detection

Also includes workflow improvements:
- Pre-task git snapshot before agent execution
- Post-task commit protocol in agent guidelines

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
admin
2026-01-14 10:42:16 -03:00
commit ec33ac1936
265 changed files with 92011 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

398
docs/CLAUDE-DISPATCH-ANALYSIS.md Normal file
View File

@@ -0,0 +1,398 @@
# Claude Dispatch and Monitor Flow Analysis
**Date:** 2026-01-11
**Author:** Luzia Research Agent
**Status:** Complete
---
## Executive Summary
Luzia dispatches Claude tasks using a **fully autonomous, non-blocking pattern**. The current architecture intentionally **does not support human-in-the-loop interaction** for background agents. This analysis documents the current flow, identifies pain points, and researches potential improvements for scenarios where user input is needed.
---
## Part 1: Current Dispatch Flow
### 1.1 Task Dispatch Mechanism
**Entry Point:** `spawn_claude_agent()` in `/opt/server-agents/orchestrator/bin/luzia:1102-1401`
**Flow:**
```
User runs: luzia <project> <task>
↓
1. Permission check (project access validation)
↓
2. QA Preflight checks (optional, validates task)
↓
3. Job directory created: /var/log/luz-orchestrator/jobs/{job_id}/
├── prompt.txt (task + context)
├── run.sh (shell script with env setup)
├── meta.json (job metadata)
└── output.log (will capture output)
↓
4. Shell script generated with:
- User-specific TMPDIR to avoid /tmp collisions
- HOME set to target user
- stdbuf for unbuffered output
- tee to capture to output.log
↓
5. Launched via: os.system(f'nohup "{script_file}" >/dev/null 2>&1 &')
↓
6. Control returns IMMEDIATELY to CLI (job_id returned)
↓
7. Agent runs in background, detached from parent process
```
### 1.2 Claude CLI Invocation
**Command Line Built:**
```bash
claude --dangerously-skip-permissions \
--permission-mode bypassPermissions \
--add-dir "{project_path}" \
--add-dir /opt/server-agents \
--print \
--verbose \
-p # Reads prompt from stdin
```
**Critical Flags:**
| Flag | Purpose |
|------|---------|
| `--dangerously-skip-permissions` | Required to use bypassPermissions mode |
| `--permission-mode bypassPermissions` | Skip ALL interactive prompts |
| `--print` | Non-interactive output mode |
| `--verbose` | Progress visibility in logs |
| `-p` | Read prompt from stdin (piped from prompt.txt) |
### 1.3 Output Capture
**Run script template:**
```bash
#!/bin/bash
echo $$ > "{pid_file}"
# Environment setup
export TMPDIR="{user_tmp_dir}"
export HOME="{user_home}"
# Execute with unbuffered output capture
sudo -u {user} bash -c '... cd "{project_path}" && cat "{prompt_file}" | stdbuf -oL -eL {claude_cmd}' 2>&1 | tee "{output_file}"
exit_code=${PIPESTATUS[0]}
echo "" >> "{output_file}"
echo "exit:$exit_code" >> "{output_file}"
```
---
## Part 2: Output Monitoring Flow
### 2.1 Status Checking
**Function:** `get_job_status()` at line 1404
Status is determined by:
1. Reading `output.log` for `exit:` line at end
2. Checking if process is still running (via PID)
3. Updating `meta.json` with completion time metrics
**Status Values:**
- `running` - No exit code yet, PID may still be active
- `completed` - `exit:0` found
- `failed` - `exit:non-zero` found
- `killed` - `exit:-9` or manual kill detected
### 2.2 User Monitoring Commands
```bash
# List all jobs
luzia jobs
# Show specific job status
luzia jobs {job_id}
# View job output
luzia logs {job_id}
# Show with timing details
luzia jobs --timing
```
### 2.3 Notification Flow
On completion, the run script appends to notification log:
```bash
echo "[$(date +%H:%M:%S)] Agent {job_id} finished (exit $exit_code)" >> /var/log/luz-orchestrator/notifications.log
```
This allows external monitoring via:
```bash
tail -f /var/log/luz-orchestrator/notifications.log
```
---
## Part 3: Current User Interaction Handling
### 3.1 The Problem: No Interaction Supported
**Current Design:** Background agents **cannot** receive user input.
**Why:**
1. `nohup` detaches from terminal - stdin unavailable
2. `--permission-mode bypassPermissions` skips prompts
3. No mechanism exists to pause agent and wait for input
4. Output is captured to file, not interactive terminal
### 3.2 When Claude Would Ask Questions
Claude's `AskUserQuestion` tool would block waiting for stdin, which isn't available. Current mitigations:
1. **Context-First Design** - Prompts include all necessary context
2. **Pre-Authorization** - Permissions granted upfront
3. **Structured Tasks** - Clear success criteria reduce ambiguity
4. **Exit Code Signaling** - Agent exits with code 1 if unable to proceed
### 3.3 Current Pain Points
| Pain Point | Impact | Current Workaround |
|------------|--------|-------------------|
| Agent can't ask clarifying questions | May proceed with wrong assumptions | Write detailed prompts |
| User can't provide mid-task guidance | Task might fail when adjustments needed | Retry with modified task |
| No approval workflow for risky actions | Security relies on upfront authorization | Careful permission scoping |
| Long tasks give no progress updates | User doesn't know if task is stuck | Check output.log manually |
| AskUserQuestion blocks indefinitely | Agent hangs, appears as "running" forever | Must kill and retry |
---
## Part 4: Research on Interaction Improvements
### 4.1 Pattern: File-Based Clarification Queue
**Concept:** Agent writes questions to file, waits for answer file.
```
/var/log/luz-orchestrator/jobs/{job_id}/
├── clarification.json # Agent writes question
├── response.json # User writes answer
└── output.log # Agent logs waiting status
```
**Agent Behavior:**
```python
# Agent encounters ambiguity
question = {
"type": "choice",
"question": "Which database: production or staging?",
"options": ["production", "staging"],
"timeout_minutes": 30,
"default_if_timeout": "staging"
}
Path("clarification.json").write_text(json.dumps(question))
# Wait for response (polling)
for _ in range(timeout * 60):
if Path("response.json").exists():
response = json.loads(Path("response.json").read_text())
return response["choice"]
time.sleep(1)
# Timeout - use default
return question["default_if_timeout"]
```
**User Side:**
```bash
# List pending questions
luzia questions
# Answer a question
luzia answer {job_id} staging
```
### 4.2 Pattern: WebSocket Status Bridge
**Concept:** Real-time bidirectional communication via WebSocket.
```
User Browser ←→ Luzia Status Server ←→ Agent Process
↑
/var/lib/luzia/status.sock
```
**Implementation in Existing Code:**
`lib/luzia_status_integration.py` already has a status publisher framework that could be extended.
**Flow:**
1. Agent publishes status updates to socket
2. Status server broadcasts to connected clients
3. When question arises, server notifies all clients
4. User responds via web UI or CLI
5. Response routed back to agent
### 4.3 Pattern: Telegram/Chat Integration
**Existing:** `/opt/server-agents/mcp-servers/assistant-channel/` provides Telegram integration.
**Extended for Agent Questions:**
```python
# Agent needs input
channel_query(
sender=f"agent-{job_id}",
question="Should I update production database?",
context="Running migration task for musica project"
)
# Bruno responds via Telegram
# Response delivered to agent via file or status channel
```
### 4.4 Pattern: Approval Gates
**Concept:** Pre-define checkpoints where agent must wait for approval.
```python
# In task prompt
"""
## Approval Gates
- Before running migrations: await approval
- Before deleting files: await approval
- Before modifying production config: await approval
Write to approval.json when reaching a gate. Wait for approved.json.
"""
```
**Gate File:**
```json
{
"gate": "database_migration",
"description": "About to run 3 migrations on staging DB",
"awaiting_since": "2026-01-11T14:30:00Z",
"auto_approve_after_minutes": null
}
```
### 4.5 Pattern: Interactive Mode Flag
**Concept:** Allow foreground execution when user is present.
```bash
# Background (current default)
luzia musica "run tests"
# Foreground/Interactive
luzia musica "run tests" --fg
# Interactive session (already exists)
luzia work on musica
```
The `--fg` flag already exists but doesn't fully support interactive Q&A. Enhancement needed:
- Don't detach process
- Keep stdin connected
- Allow Claude's AskUserQuestion to work normally
---
## Part 5: Recommendations
### 5.1 Short-Term (Quick Wins)
1. **Better Exit Code Semantics**
- Exit 100 = "needs clarification" (new code)
- Capture the question in `clarification.json`
- `luzia questions` command to list pending
2. **Enhanced `--fg` Mode**
- Don't background the process
- Keep stdin/stdout connected
- Allow normal interactive Claude session
3. **Progress Streaming**
- Add `luzia watch {job_id}` for `tail -f` on output.log
- Color-coded output for better readability
### 5.2 Medium-Term (New Features)
4. **File-Based Clarification System**
- Agent writes to `clarification.json`
- Luzia CLI watches for pending questions
- `luzia answer {job_id} <response>` writes `response.json`
- Agent polls and continues
5. **Telegram/Chat Bridge for Questions**
- Extend assistant-channel for agent questions
- Push notification when agent needs input
- Reply via chat, response routed to agent
6. **Status Dashboard**
- Web UI showing all running agents
- Real-time output streaming
- Question/response interface
### 5.3 Long-Term (Architecture Evolution)
7. **Approval Workflows**
- Define approval gates in task specification
- Configurable auto-approve timeouts
- Audit log of approvals
8. **Agent Orchestration Layer**
- Queue of pending questions across agents
- Priority handling for urgent questions
- SLA tracking for response times
9. **Hybrid Execution Mode**
- Start background, attach to foreground if question arises
- Agent sends signal when needing input
- CLI can "attach" to running agent
---
## Part 6: Implementation Priority
| Priority | Feature | Effort | Impact |
|----------|---------|--------|--------|
| **P0** | Better `--fg` mode | Low | High - enables immediate interactive use |
| **P0** | Exit code 100 for clarification | Low | Medium - better failure understanding |
| **P1** | `luzia watch {job_id}` | Low | Medium - easier monitoring |
| **P1** | File-based clarification | Medium | High - enables async Q&A |
| **P2** | Telegram question bridge | Medium | Medium - mobile notification |
| **P2** | Status dashboard | High | High - visual monitoring |
| **P3** | Approval workflows | High | Medium - enterprise feature |
---
## Conclusion
The current Luzia dispatch architecture is optimized for **fully autonomous** agent execution. This is the right default for background tasks. However, there's a gap for scenarios where:
- Tasks are inherently ambiguous
- User guidance is needed mid-task
- High-stakes actions require approval
The recommended path forward is:
1. **Improve `--fg` mode** for true interactive sessions
2. **Add file-based clarification** for async Q&A on background tasks
3. **Integrate with Telegram** for push notifications on questions
4. **Build status dashboard** for visual monitoring and interaction
These improvements maintain the autonomous-by-default philosophy while enabling human-in-the-loop interaction when needed.
---
## Appendix: Key File Locations
| File | Purpose |
|------|---------|
| `/opt/server-agents/orchestrator/bin/luzia:1102-1401` | `spawn_claude_agent()` - main dispatch |
| `/opt/server-agents/orchestrator/bin/luzia:1404-1449` | `get_job_status()` - status checking |
| `/opt/server-agents/orchestrator/bin/luzia:4000-4042` | `route_logs()` - log viewing |
| `/opt/server-agents/orchestrator/lib/responsive_dispatcher.py` | Async dispatch patterns |
| `/opt/server-agents/orchestrator/lib/cli_feedback.py` | CLI output formatting |
| `/opt/server-agents/orchestrator/AGENT-AUTONOMY-RESEARCH.md` | Prior research on autonomy |
| `/var/log/luz-orchestrator/jobs/` | Job directories |
| `/var/log/luz-orchestrator/notifications.log` | Completion notifications |

167
docs/COCKPIT.md Normal file
View File

@@ -0,0 +1,167 @@
# Luzia Cockpit - Human-in-the-Loop Claude Sessions
## Overview
Cockpit provides **pausable Claude agent sessions** using Docker containers with tmux.
The key innovation is that `docker stop/start` freezes/resumes the entire session state,
and Claude sessions persist via `--session-id` and `--resume` flags.
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ luzia cockpit │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Docker Container │ │
│ │ ┌──────────────────────────────────────────────────┐ │ │
│ │ │ tmux session │ │ │
│ │ │ ┌──────────────────────────────────────────┐ │ │ │
│ │ │ │ Claude CLI │ │ │ │
│ │ │ │ --session-id / --resume │ │ │ │
│ │ │ └──────────────────────────────────────────┘ │ │ │
│ │ └──────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ Mounts: │ │
│ │ - /workspace → project home │ │
│ │ - ~/.claude → credentials + sessions │ │
│ │ - /var/cockpit → state files │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ docker stop → FREEZE (all state preserved) │
│ docker start → RESUME (continue conversation) │
│ │
└─────────────────────────────────────────────────────────────────┘
```
## Commands
### Start a Cockpit
```bash
luzia cockpit start <project>
```
Starts (or resumes) a cockpit container for a project.
### Stop (Freeze) a Cockpit
```bash
luzia cockpit stop <project>
```
Stops the container, freezing all state. Can be resumed later.
### Remove a Cockpit
```bash
luzia cockpit remove <project>
```
Permanently removes the container and state.
### Send a Message
```bash
luzia cockpit send <project> <message>
```
Sends a message to Claude. First message creates the session,
subsequent messages continue it.
### Respond to a Question
```bash
luzia cockpit respond <project> <answer>
```
Alias for send - used when Claude is waiting for input.
### Get Output
```bash
luzia cockpit output <project>
```
Shows recent output from the tmux session.
### Check Status
```bash
luzia cockpit status [project]
```
Shows all cockpits or a specific one, including session ID and
whether Claude is waiting for a response.
### Attach Interactively
```bash
luzia cockpit attach <project>
```
Shows the command to attach to the tmux session for interactive work.
## Session Persistence
Claude sessions are stored in the mounted `~/.claude/` directory:
```
~/.claude/projects/{workspace-path}/{session-id}.jsonl
```
The cockpit tracks:
- `session_id` - UUID for the Claude conversation
- `session_started` - Whether first message has been sent
- `awaiting_response` - If Claude asked a question (detected by "?" at end)
- `last_question` - The question Claude asked
## Example Workflow
```bash
# Start a cockpit for musica project
luzia cockpit start musica
# → Started cockpit, Session: abc-123-def
# Send a task
luzia cockpit send musica "Fix the track component loading bug"
# → Claude analyzes and responds
# Claude asks a question - FREEZE the session
luzia cockpit stop musica
# → Container paused, queue can continue with other projects
# Later, human comes back with answer - RESUME
luzia cockpit start musica
luzia cockpit respond musica "Use lazy loading, target is 200ms"
# → Claude continues with the answer
```
## Integration with Queue
When Claude is waiting for human input:
1. Set project queue to `awaiting_human` status
2. Other projects continue processing
3. On human response, resume project queue
## Docker Image
Built from `/opt/server-agents/orchestrator/docker/cockpit/Dockerfile`:
- Base: `debian:bookworm-slim`
- Node.js 20 LTS
- Claude CLI (`@anthropic-ai/claude-code`)
- tmux with 50000 line history
- Mouse support for human attach
## State Files
```
/var/lib/luz-orchestrator/cockpits/
├── admin.json
├── musica.json
└── overbits.json
```
Each JSON file contains:
```json
{
"project": "musica",
"session_id": "abc-123-def",
"status": "running",
"session_started": true,
"awaiting_response": false,
"last_question": null
}
```
## Benefits
1. **True Pause/Resume** - `docker stop/start` freezes everything
2. **Conversation Memory** - Claude remembers via session persistence
3. **Non-blocking Queue** - Projects don't block each other
4. **Human Attachment** - Can attach tmux for direct interaction
5. **Credential Isolation** - Each project uses shared credentials safely

369
docs/DISPATCHER-INTEGRATION-GUIDE.md Normal file
View File

@@ -0,0 +1,369 @@
# Dispatcher Integration Guide - Luzia CLI Enhancement
## Summary of Changes
The Responsive Dispatcher improves Luzia CLI responsiveness by:
1. **Eliminating blocking** during task dispatch - CLI returns immediately with job_id
2. **Adding background monitoring** - Jobs progress tracked asynchronously
3. **Implementing status feedback** - Live progress updates without blocking
4. **Enabling concurrent management** - Multiple tasks tracked independently
5. **Providing responsive CLI** - Always responsive after dispatch
## Performance Improvements
### Before (Blocking Dispatch)
```
User: luzia overbits "task"
↓ [BLOCKS HERE - CLI waits for agent startup]
(3-5 seconds of blocking)
↓
Output: job_id
Result: CLI frozen during dispatch
```
### After (Non-Blocking Dispatch)
```
User: luzia overbits "task"
↓ [RETURNS IMMEDIATELY]
(<100ms)
↓
Output: job_id
Result: CLI responsive, task runs in background
```
### Metrics
- **Dispatch latency**: <100ms (vs 3-5s before)
- **Throughput**: 434 tasks/second
- **Status retrieval**: <1ms (cached) or <50µs (fresh)
- **Memory per job**: ~2KB
## New Modules
### 1. `lib/responsive_dispatcher.py`
Core non-blocking dispatcher engine.
**Key Classes:**
- `ResponseiveDispatcher` - Main dispatcher with:
- `dispatch_task()` - Returns immediately with job_id
- `get_status()` - Poll job status with caching
- `update_status()` - Update job progress (used by monitor)
- `list_jobs()` - Get job history
- `wait_for_job()` - Block until completion (optional)
- `start_background_monitor()` - Start monitor thread
**Features:**
- Atomic status file operations
- Intelligent caching (1-second TTL)
- Background monitoring queue
- Job history persistence
### 2. `lib/cli_feedback.py`
Pretty-printed CLI feedback and status display.
**Key Classes:**
- `CLIFeedback` - Responsive output formatting:
- `job_dispatched()` - Show dispatch confirmation
- `show_status()` - Display job status with progress
- `show_jobs_list()` - List all jobs
- `show_concurrent_jobs()` - Summary view
- `Colors` - ANSI color codes
- `ProgressBar` - ASCII progress bar renderer
- `ResponseiveOutput` - Context manager for operations
### 3. `lib/dispatcher_enhancements.py`
Integration layer connecting dispatcher to existing Luzia code.
**Key Classes:**
- `EnhancedDispatcher` - Wrapper combining responsive dispatcher + feedback
- `dispatch_and_report()` - Dispatch with automatic feedback
- `get_status_and_display()` - Get and display status
- `show_jobs_summary()` - Show jobs for a project
- `show_concurrent_summary()` - Show all jobs
**Integration Functions:**
- `enhanced_spawn_claude_agent()` - Replacement for existing spawn
- `track_existing_job()` - Retroactive tracking
- `show_job_status_interactive()` - Interactive monitoring
- `start_background_monitoring()` - Start monitor thread
## Integration Steps
### Step 1: Import New Modules
In `bin/luzia`, add at the top:
```python
from lib.responsive_dispatcher import ResponseiveDispatcher
from lib.cli_feedback import CLIFeedback
from lib.dispatcher_enhancements import EnhancedDispatcher, get_enhanced_dispatcher
```
### Step 2: Enhanced Project Task Handler
Replace the existing `route_project_task` handler:
```python
def route_project_task(config: dict, args: list, kwargs: dict) -> int:
"""Handler: luzia <project> <task> (with responsive dispatch)"""
# ... existing validation code ...
project = args[0]
task = " ".join(args[1:])
# ... existing setup code ...
# Use enhanced dispatcher for responsive dispatch
enhanced = get_enhanced_dispatcher()
# Dispatch and show feedback
job_id, status = enhanced.dispatch_and_report(
project=project,
task=task,
show_details=not is_command, # Show details for natural language only
show_feedback=VERBOSE
)
# Output job_id for tracking
print(f"agent:{project}:{job_id}")
return 0
```
### Step 3: Add Job Status Commands
Add new route for `luzia jobs`:
```python
def route_jobs(config: dict, args: list, kwargs: dict) -> int:
"""Handler: luzia jobs [job_id]"""
enhanced = get_enhanced_dispatcher()
if not args:
# Show all jobs
enhanced.show_jobs_summary()
return 0
job_id = args[0]
if "--watch" in args:
# Interactive monitoring
from lib.dispatcher_enhancements import show_job_status_interactive
show_job_status_interactive(job_id)
else:
# Show status
enhanced.get_status_and_display(job_id, show_full=True)
return 0
```
### Step 4: Start Background Monitor
Add to main startup:
```python
def main():
# ... existing code ...
# Start background monitoring
enhanced = get_enhanced_dispatcher()
enhanced.dispatcher.start_background_monitor()
# ... rest of main ...
```
## File Structure
New files created:
```
/opt/server-agents/orchestrator/
├── lib/
│ ├── responsive_dispatcher.py # Core dispatcher
│ ├── cli_feedback.py # CLI feedback system
│ └── dispatcher_enhancements.py # Integration layer
├── tests/
│ └── test_responsive_dispatcher.py # Test suite (11 tests)
├── examples/
│ └── demo_concurrent_tasks.py # Live demonstration
└── docs/
├── RESPONSIVE-DISPATCHER.md # User guide
└── DISPATCHER-INTEGRATION-GUIDE.md (this file)
```
## Usage Examples
### Basic Dispatch (Non-blocking)
```bash
$ luzia overbits "fix the login button"
✓ Dispatched
Job ID: 113754-a2f5
Project: overbits
Use: luzia jobs to view status
luzia jobs 113754-a2f5 for details
$ # CLI is responsive immediately!
$ luzia jobs # Check status without waiting
```
### Monitor Multiple Jobs
```bash
$ luzia overbits "task 1" & luzia musica "task 2" & luzia dss "task 3" &
agent:overbits:113754-a2f5
agent:musica:113754-8e4b
agent:dss:113754-9f3c
$ # All 3 running concurrently
$ luzia jobs
Task Summary:
Running: 3
Pending: 0
```
### Watch Job Progress
```bash
$ luzia jobs 113754-a2f5 --watch
Monitoring job: 113754-a2f5
starting [░░░░░░░░░░░░░░░░░░░░] 5%
running [██████░░░░░░░░░░░░░░] 30%
running [████████████░░░░░░░░] 65%
completed [██████████████████████] 100%
```
## Testing
Run the test suite:
```bash
python3 tests/test_responsive_dispatcher.py
```
All 11 tests should pass:
- ✓ Immediate dispatch
- ✓ Status retrieval
- ✓ Status updates
- ✓ Concurrent jobs
- ✓ Cache behavior
- ✓ CLI feedback
- ✓ Progress bar
- ✓ Background monitoring
- ✓ Enhanced dispatcher dispatch
- ✓ Enhanced dispatcher display
- ✓ Enhanced dispatcher summary
## Demo
Run the live demo:
```bash
python3 examples/demo_concurrent_tasks.py
```
Demonstrates:
1. Concurrent dispatch (5 tasks in <50ms)
2. Non-blocking status polling
3. Independent job monitoring
4. Job listing and summaries
5. Performance metrics (434 tasks/sec, <1ms status retrieval)
## Backward Compatibility
The implementation maintains full backward compatibility:
- Existing `spawn_claude_agent()` still works
- Existing route handlers can continue to work
- New functionality is opt-in through `EnhancedDispatcher`
- Status files stored separately in `/var/lib/luzia/jobs/`
- No changes to job output or agent execution
## Migration Checklist
To fully integrate responsive dispatcher into Luzia:
- [ ] Import new modules in bin/luzia
- [ ] Update route_project_task to use EnhancedDispatcher
- [ ] Add route_jobs handler for `luzia jobs`
- [ ] Start background monitor in main()
- [ ] Add `--watch` flag support to jobs command
- [ ] Test with existing workflows
- [ ] Run full test suite
- [ ] Update CLI help text
- [ ] Document new `luzia jobs` command
- [ ] Document `--watch` flag usage
## Configuration
Optional environment variables:
```bash
# Cache TTL in seconds (default: 1)
export LUZIA_CACHE_TTL=2
# Monitor poll interval (default: 1)
export LUZIA_MONITOR_INTERVAL=0.5
# Max job history (default: 1000)
export LUZIA_MAX_JOBS=500
# Job directory (default: /var/lib/luzia/jobs)
export LUZIA_JOBS_DIR=/custom/path
```
## Troubleshooting
### Monitor not running
Check if background thread started:
```bash
ps aux | grep python | grep luzia
```
Start manually if needed:
```python
from lib.dispatcher_enhancements import start_background_monitoring
start_background_monitoring()
```
### Jobs not updating
Ensure job directory is writable:
```bash
ls -la /var/lib/luzia/jobs/
chmod 755 /var/lib/luzia/jobs
```
### Status cache stale
Force fresh read:
```python
status = dispatcher.get_status(job_id, use_cache=False)
```
## Future Enhancements
Planned additions:
- [ ] Web dashboard for job monitoring
- [ ] WebSocket support for real-time updates
- [ ] Job retry with exponential backoff
- [ ] Job cancellation with graceful shutdown
- [ ] Resource-aware scheduling
- [ ] Job dependencies and DAG execution
- [ ] Slack/email notifications
- [ ] Database persistence (SQLite)
- [ ] Job timeout management
## Support
For issues or questions:
1. Check test suite: `python3 tests/test_responsive_dispatcher.py`
2. Run demo: `python3 examples/demo_concurrent_tasks.py`
3. Review documentation: `docs/RESPONSIVE-DISPATCHER.md`
4. Check logs: `/var/log/luz-orchestrator/`

189
docs/HELP_UPDATE_SUMMARY.md Normal file
View File

@@ -0,0 +1,189 @@
# Luzia Help Reference Update - Summary
**Date:** January 9, 2026
**Status:** ✅ Complete
## What Was Updated
### 1. Main Help Docstring (bin/luzia)
Updated the Python docstring at the top of the `luzia` script with:
- Clear "QUICK START" section
- Organized command categories:
- Core Project Commands
- Maintenance & System
- Failure Management
- Knowledge Graph & QA
- Research (3-Phase Flow)
- Code Analysis
- Advanced Reasoning
- Queue Management
- Low-Level Operations
- Global flags section
- Practical examples
- Reference to full documentation
**File:** `/opt/server-agents/orchestrator/bin/luzia`
**Lines:** 1-92 (docstring)
### 2. Comprehensive Command Reference (NEW)
Created detailed markdown documentation with:
- Overview and quick start
- All 30+ commands with descriptions
- Usage patterns and examples
- Configuration details
- Troubleshooting guide
- Exit codes
**File:** `/opt/server-agents/orchestrator/docs/LUZIA_COMMAND_REFERENCE.md`
### 3. Quick Reference Cheat Sheet (NEW)
Created concise cheat sheet with:
- Essential commands (4 lines)
- Troubleshooting patterns
- System maintenance
- Project work commands
- Knowledge base queries
- Research patterns
- Code analysis
- Advanced features
- Common patterns
**File:** `/opt/server-agents/orchestrator/docs/LUZIA_CHEAT_SHEET.md`
## Testing
✅ Tested help output:
```bash
python3 bin/luzia --help
```
Output is clean, well-organized, and 91 lines of comprehensive documentation.
## Features Documented
### Core Features
- ✅ Project execution (`<project> <task>`)
- ✅ Interactive sessions (`work on <project>`)
- ✅ List/status management
- ✅ Container management (stop, cleanup)
### Maintenance
- ✅ Full cleanup (jobs, containers, logs)
- ✅ Dry-run preview
- ✅ Job listing and management
- ✅ Maintenance recommendations
### Failure Management
- ✅ List failures with exit codes
- ✅ Show failure details
- ✅ Summary by exit code
- ✅ Smart retry (all fixable)
- ✅ Individual retry
- ✅ Kill stuck jobs
### Knowledge Graph & QA
- ✅ QA validation
- ✅ Code sync to KG
- ✅ Multi-domain doc search
- ✅ Entity details
- ✅ KG statistics
- ✅ Markdown sync
### Research (3-Phase Flow)
- ✅ Research initiation (context → search → synthesize)
- ✅ Research listing
- ✅ Session details
- ✅ Knowledge graph display
- ✅ Phase updates (internal)
- ✅ KG entity addition (internal)
### Code Analysis
- ✅ Structure analysis
- ✅ Project-specific analysis
- ✅ Subdirectory analysis
- ✅ JSON output
- ✅ KG integration control
### Advanced Features
- ✅ Deep reasoning (`think deep`)
- ✅ Troubleshooting (`fix`)
- ✅ Queue management
- ✅ Notifications
### Low-Level Operations
- ✅ Raw command execution
- ✅ File read/write
- ✅ Context retrieval
- ✅ JSON output format
### Global Flags
- ✅ Help (`--help`, `-h`, `help`)
- ✅ Verbose mode
- ✅ Foreground execution
## Organization
The help system now has three levels:
1. **Quick Help** (in-command): `luzia --help` (91 lines, well-organized)
2. **Cheat Sheet**: `/docs/LUZIA_CHEAT_SHEET.md` (practical patterns)
3. **Full Reference**: `/docs/LUZIA_COMMAND_REFERENCE.md` (complete details)
## Key Improvements
1. **Better Organization**: Commands grouped by category
2. **Clearer Examples**: Real-world usage patterns
3. **Exit Codes**: Now documented
4. **Quick Start**: Easy entry for new users
5. **Complete Coverage**: All 27 command handlers documented
6. **Accessibility**: Cheat sheet for quick lookups
## Files Modified/Created
| File | Type | Status |
|------|------|--------|
| `bin/luzia` | Modified | ✅ Updated docstring (lines 1-92) |
| `docs/LUZIA_COMMAND_REFERENCE.md` | New | ✅ Created |
| `docs/LUZIA_CHEAT_SHEET.md` | New | ✅ Created |
| `docs/HELP_UPDATE_SUMMARY.md` | New | ✅ Created (this file) |
## Usage
### View Help
```bash
luzia --help
python3 bin/luzia --help
./bin/luzia --help
```
### Quick Reference
```bash
cat docs/LUZIA_CHEAT_SHEET.md
```
### Full Documentation
```bash
cat docs/LUZIA_COMMAND_REFERENCE.md
```
## Next Steps
- Commands are now fully documented
- Users can discover features via `luzia --help`
- Cheat sheet available for quick lookups
- Full reference for detailed exploration
- Help system is discoverable and comprehensive
## Stats
- **Command handlers documented:** 27
- **Help docstring lines:** 91
- **Documentation files created:** 2
- **Total documentation lines:** 500+
- **Command categories:** 9
---
**Status:** Ready for use
**Last Updated:** 2026-01-09
**Next Review:** When new commands are added

206
docs/LUZIA_CHEAT_SHEET.md Normal file
View File

@@ -0,0 +1,206 @@
# Luzia Cheat Sheet
A quick reference for the most commonly used Luzia commands.
## Essential Commands
```bash
luzia --help # Show help
luzia list # List projects
luzia status # Check status
luzia <project> <task> # Run task
```
## Troubleshooting
```bash
# See what failed
luzia failures
# Show error breakdown
luzia failures --summary
# Retry failed job
luzia retry <job_id>
# Auto-retry all fixable
luzia failures --auto-retry
# Kill stuck job
luzia kill <job_id>
```
## System Maintenance
```bash
# Preview cleanup
luzia cleanup --dry-run
# Clean old jobs
luzia cleanup jobs
# Stop stale containers
luzia cleanup containers
# Full cleanup
luzia cleanup
```
## Project Work
```bash
# Interactive session
luzia work on <project>
# View logs
luzia logs <project>
# View history
luzia history <project>
# Stop container
luzia stop <project>
```
## Knowledge Base
```bash
# Search docs
luzia docs "topic"
# Search sysadmin docs
luzia docs sysadmin "nginx"
# Show entity
luzia docs --show entity_name
# Show stats
luzia docs --stats
# Sync docs
luzia docs --sync
```
## Research
```bash
# Start research
luzia research dss "topic"
# List research
luzia research-list dss
# Show research
luzia research-show session_id
# Show knowledge
luzia research-knowledge dss
```
## Code Analysis
```bash
# Analyze project
luzia structure dss
# Output JSON
luzia structure dss --json
# Analyze subdirectory
luzia structure . lib/
```
## Advanced
```bash
# Deep reasoning
luzia think deep "question"
# Run QA checks
luzia qa
# Sync code to KG
luzia qa --sync
# Raw command
luzia --exec project "command"
# Read file
luzia --read project /path/to/file
# Write file
luzia --write project /path/to/file "content"
```
## Flags
```bash
--help # Help
--verbose # Detailed output
--fg # Run in foreground
```
## Quick Patterns
**Check Everything:**
```bash
luzia list
luzia status
luzia maintenance
```
**Fix Problems:**
```bash
luzia failures --summary
luzia retry <job_id>
# or
luzia failures --auto-retry
```
**Work on Project:**
```bash
luzia work on <project>
# or
luzia <project> <task> [args]
```
**Research Topic:**
```bash
luzia research <project> "your question"
luzia research-show <session_id>
```
**Analyze Code:**
```bash
luzia structure <project> --json
luzia docs "search term"
```
**Clean Up:**
```bash
luzia cleanup --dry-run # Preview
luzia cleanup # Execute
```
## Exit Codes
- **0** - Success
- **1** - General error
- **2** - Invalid arguments
- **3** - Project not found
- **4** - Container error
## Project List
View with: `luzia list`
Common projects:
- `musica` - Music processing
- `overbits` - Data systems
- `dss` - Development
- `librechat` - Chat interface
- `admin` - System administration
---
**Full Documentation:** See `LUZIA_COMMAND_REFERENCE.md`

365
docs/LUZIA_COMMAND_REFERENCE.md Normal file
View File

@@ -0,0 +1,365 @@
# Luzia Command Reference
**Luzia** is the unified access point for managing all tasks and projects in the server agents infrastructure.
## Quick Start
```bash
luzia --help # Show all commands
luzia list # List all available projects
luzia status # Show current system status
luzia <project> <task> # Run a task in a project
```
---
## Core Commands
### Project Execution
| Command | Description |
|---------|-------------|
| `luzia <project> <task>` | Execute a task in a project's Docker container |
| `luzia work on <project>` | Start interactive session for a project (delegates to subagent) |
| `luzia list` | List all available projects with their status |
| `luzia status [project]` | Show overall status or specific project status |
| `luzia stop <project>` | Stop a running container |
| `luzia history <project>` | View recent changes in a project |
**Examples:**
```bash
luzia musica analyze logs
luzia work on overbits
luzia list
luzia status dss
```
---
## Maintenance & System Commands
### Cleanup Operations
| Command | Description |
|---------|-------------|
| `luzia cleanup` | Full maintenance (jobs + containers + logs) |
| `luzia cleanup jobs` | Clean old job directories only |
| `luzia cleanup containers` | Stop stale containers only |
| `luzia cleanup --dry-run` | Preview cleanup without deleting |
**Examples:**
```bash
luzia cleanup --dry-run
luzia cleanup containers
luzia cleanup jobs
```
### System Status
| Command | Description |
|---------|-------------|
| `luzia maintenance` | Show maintenance status and recommendations |
| `luzia jobs [job_id]` | List all jobs or show details for a specific job |
| `luzia logs [project]` | View project execution logs |
**Examples:**
```bash
luzia maintenance
luzia jobs
luzia jobs abc123def
luzia logs dss
```
---
## Job Management
### Failure Management (Smart Retry)
| Command | Description |
|---------|-------------|
| `luzia failures` | List recent failures with exit codes |
| `luzia failures <job_id>` | Show detailed failure information |
| `luzia failures --summary` | Summary breakdown by exit code |
| `luzia failures --auto-retry` | Auto-retry all fixable failures |
| `luzia retry <job_id>` | Retry a specific failed job |
| `luzia kill <job_id>` | Kill a running agent job |
**Examples:**
```bash
luzia failures
luzia failures abc123def
luzia failures --summary
luzia failures --auto-retry
luzia retry abc123def
luzia kill abc123def
```
---
## Knowledge Graph & Documentation
### QA & Validation
| Command | Description |
|---------|-------------|
| `luzia qa` | Run QA validation checks |
| `luzia qa --sync` | Sync code to knowledge graph |
**Examples:**
```bash
luzia qa
luzia qa --sync
```
### Documentation Search
| Command | Description |
|---------|-------------|
| `luzia docs <query>` | Search all knowledge graphs |
| `luzia docs sysadmin <query>` | Search sysadmin domain |
| `luzia docs --show <entity>` | Show entity details from KG |
| `luzia docs --stats` | Show knowledge graph statistics |
| `luzia docs --sync` | Sync .md files to knowledge graph |
**Examples:**
```bash
luzia docs docker setup
luzia docs sysadmin nginx
luzia docs --show nginx
luzia docs --stats
luzia docs --sync
```
---
## Research & Analysis
### Research Commands (3-Phase Flow)
| Command | Description |
|---------|-------------|
| `luzia research [project] <topic>` | Start research (context → search → synthesize) |
| `luzia deep research [project] <topic>` | Same as research (alias) |
| `luzia web research [project] <topic>` | Same as research (alias) |
| `luzia research-list [project]` | List research sessions |
| `luzia research-show <session_id>` | Show research session details |
| `luzia research-knowledge [project]` | Show project knowledge graph |
**Examples:**
```bash
luzia research musica database optimization
luzia deep research dss performance tuning
luzia web research overbits authentication
luzia research-list dss
luzia research-show sess_abc123
luzia research-knowledge musica
```
### Internal Research Operations (Called During Flow)
| Command | Description |
|---------|-------------|
| `luzia research-update <id> <phase> <json>` | Update research phase (internal) |
| `luzia research-graph <id> <json>` | Add entities to knowledge graph (internal) |
---
## Code Analysis & Intelligence
### Structural Analysis
| Command | Description |
|---------|-------------|
| `luzia structure` | Analyze current orchestrator structure |
| `luzia structure <project>` | Analyze a specific project |
| `luzia structure . path/src` | Analyze specific subdirectory |
| `luzia structure --json` | Output analysis as JSON |
| `luzia structure --no-kg` | Don't save to knowledge graph |
**Examples:**
```bash
luzia structure
luzia structure dss
luzia structure . lib/docker_bridge.py
luzia structure --json > analysis.json
luzia structure --no-kg
```
---
## Advanced Features
### Deep Reasoning
| Command | Description |
|---------|-------------|
| `luzia think deep <topic>` | Deep reasoning via Zen + Gemini 3 |
**Examples:**
```bash
luzia think deep "how to optimize docker image size"
```
### Troubleshooting
| Command | Description |
|---------|-------------|
| `luzia fix <issue>` | Troubleshooting assistant |
**Examples:**
```bash
luzia fix "container not starting"
```
### Notifications
| Command | Description |
|---------|-------------|
| `luzia notify` | View notifications |
| `luzia notifications` | Alias for notify |
---
## Queue Management (Advanced)
| Command | Description |
|---------|-------------|
| `luzia queue` | Show queue status |
| `luzia dispatch <job>` | Dispatch a job to the queue |
**Examples:**
```bash
luzia queue
luzia dispatch research_agent
```
---
## Low-Level Operations
These are primarily for internal use:
| Command | Description |
|---------|-------------|
| `luzia --exec <project> <command>` | Execute raw command (JSON output) |
| `luzia --read <project> <path>` | Read file contents (JSON output) |
| `luzia --write <project> <path> <content>` | Write to file (JSON output) |
| `luzia --context <project>` | Get project context (JSON output) |
**Examples:**
```bash
luzia --exec musica ls -la
luzia --read dss /workspace/config.json
luzia --write overbits /workspace/test.txt "content here"
luzia --context librechat
```
---
## Global Flags
| Flag | Description |
|------|-------------|
| `--help`, `-h`, `help` | Show this help message |
| `--verbose` | Enable verbose output |
| `--fg` | Run in foreground (don't background) |
**Examples:**
```bash
luzia --help
luzia --verbose status
luzia --fg musica analyze data
```
---
## Exit Codes
| Code | Meaning |
|------|---------|
| 0 | Success |
| 1 | General error |
| 2 | Invalid arguments |
| 3 | Project not found |
| 4 | Container error |
---
## Common Patterns
### Check System Health
```bash
luzia list # See all projects
luzia status # Overall status
luzia maintenance # System recommendations
```
### Run a Task
```bash
luzia <project> <task> <args>
```
### Manage Failures
```bash
luzia failures # See what failed
luzia failures --summary # Breakdown by code
luzia retry <job_id> # Retry one
luzia failures --auto-retry # Retry all fixable
```
### Research a Topic
```bash
luzia research musica "how to optimize queries"
luzia research-show <session_id>
luzia research-knowledge musica
```
### Analyze Code
```bash
luzia structure dss --json
luzia docs dss "query"
luzia qa --sync
```
---
## Configuration
Configuration is loaded from `/opt/server-agents/orchestrator/config.json`:
```json
{
"projects": {
"musica": {
"image": "musica:latest",
"port": 3000
},
...
}
}
```
---
## Troubleshooting
### "Unknown: <command>"
The command wasn't recognized. Use `luzia --help` to see valid commands.
### "Permission denied"
You may not have permission to run commands in that project. Check your user permissions.
### Container errors
Run `luzia cleanup containers` to stop stale containers, then try again.
### Job failures
Use `luzia failures` to see what went wrong, then `luzia retry <job_id>`.
---
## See Also
- `/opt/server-agents/orchestrator/docs/` - Full documentation
- `/opt/server-agents/CLAUDE.md` - Project instructions
- `/etc/claude/GLOBAL.md` - Global server rules

449
docs/PLUGIN-MARKETPLACE-INTEGRATION.md Normal file
View File

@@ -0,0 +1,449 @@
# Claude Plugin Marketplace Integration for Luzia
## Overview
Luzia now integrates with the Claude official plugin marketplace as a trusted source for AI skills and capabilities. This enables intelligent plugin skill detection, matching, and dispatch for tasks.
## Architecture
```
┌─────────────────────────────────────┐
│ Claude Marketplace │
│ (Official Plugins) │
└──────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ PluginMarketplaceRegistry │
│ (Load & Index Plugins) │
└──────────────┬──────────────────────┘
│
┌──────────┴──────────┐
│ │
▼ ▼
┌──────────────────┐ ┌──────────────────────┐
│ PluginSkillLoader│ │ PluginCapabilityMatcher
│ (Skills from Caps) │ (Find Plugins for Tasks)
└──────────────────┘ └──────────────────────┘
│ │
└──────────────┬──────┘
▼
┌────────────────────────┐
│DispatcherPluginBridge │
│(Integrate with Dispatch)
└────────────────────────┘
│
▼
┌──────────────────┐
│Shared Knowledge │
│Graph (Plugins) │
└──────────────────┘
```
## Components
### 1. **PluginMarketplaceRegistry** (`plugin_marketplace.py`)
Central registry of official Claude plugins with built-in support for:
- **Code Simplifier**: Code refactoring and optimization
- **Code Reviewer**: Security, performance, and quality reviews
- **API Integration Helper**: API client generation and validation
**Key Features:**
- Plugin metadata (name, description, version, trust level)
- Capability indexing (fast lookup by category/keyword)
- Plugin matching for task descriptions
- Export to knowledge graph format
**Usage:**
```python
from lib.plugin_marketplace import get_marketplace_registry
registry = get_marketplace_registry()
plugins = registry.list_plugins(category='security')
matched = registry.find_plugins_for_task(
'Find vulnerabilities in code',
['security', 'code', 'review']
)
```
### 2. **PluginSkillLoader** (`plugin_skill_loader.py`)
Converts plugin capabilities into Luzia skills for task dispatch.
**Features:**
- Generate skills from plugin capabilities
- Index skills by keywords and categories
- Cache skills for performance
- Export for dispatcher/knowledge graph
- Skill-to-plugin mapping
**Usage:**
```python
from lib.plugin_skill_loader import get_plugin_skill_loader
loader = get_plugin_skill_loader()
skills = loader.generate_skills_from_plugins()
matched = loader.find_skills_for_task('simplify this code')
```
**Generated Skills:**
```
code-simplifier:simplify_code (Code Simplifier)
code-simplifier:detect_complexity (Code Simplifier)
code-simplifier:suggest_improvements (Code Simplifier)
code-reviewer:security_review (Code Reviewer)
code-reviewer:performance_review (Code Reviewer)
code-reviewer:best_practices_review (Code Reviewer)
api-integration:generate_api_client (API Integration Helper)
api-integration:validate_api_spec (API Integration Helper)
```
### 3. **DispatcherPluginBridge** (`dispatcher_plugin_integration.py`)
Seamlessly integrates plugins into the task dispatch system.
**Features:**
- Enhance task context with plugin skills
- Generate recommendations for task handling
- Build execution sequences based on capabilities
- Plugin-aware task dispatch
**Usage:**
```python
from lib.dispatcher_plugin_integration import (
DispatcherPluginBridge,
PluginAwareTaskDispatcher
)
bridge = DispatcherPluginBridge()
context = bridge.enhance_task_context(
'Review code for security',
'myproject',
'job-123'
)
dispatcher = PluginAwareTaskDispatcher(bridge)
result = dispatcher.dispatch_with_plugin_context(
'Optimize this function',
'myproject',
'job-456'
)
```
### 4. **PluginKnowledgeGraphExporter** (`plugin_kg_integration.py`)
Exports plugin data to shared knowledge graph format.
**Features:**
- Export plugins as entities
- Export skills as entities
- Export relationships (plugin→skill, skill→category)
- Complete KG export with metadata
**Usage:**
```python
from lib.plugin_kg_integration import export_plugins_to_kg
# Export to files
exports = export_plugins_to_kg()
# Each export type:
# - plugins_entities.json
# - skills_entities.json
# - relationships.json
# - complete_export.json
```
### 5. **PluginCLI** (`plugin_cli.py`)
Command-line interface for plugin operations.
**Commands:**
```bash
# List all plugins
luzia plugins list
# Show plugin details
luzia plugins code-simplifier
luzia plugins code-reviewer
# List all skills
luzia plugins skills
# Find plugins for a task
luzia plugins find "review code for security"
# Export plugin data
luzia plugins export
# Show statistics
luzia plugins stats
# Help
luzia plugins help
```
## Plugin Definitions
### Code Simplifier
**ID:** `code-simplifier`
**Vendor:** Anthropic
**Trust:** Trusted
**Capabilities:**
- `simplify_code` - Analyze and simplify code for readability
- `detect_complexity` - Identify overly complex code patterns
- `suggest_improvements` - Suggest code improvements and best practices
### Code Reviewer
**ID:** `code-reviewer`
**Vendor:** Anthropic
**Trust:** Trusted
**Capabilities:**
- `security_review` - Identify security vulnerabilities
- `performance_review` - Analyze performance bottlenecks
- `best_practices_review` - Check against best practices
### API Integration Helper
**ID:** `api-integration`
**Vendor:** Anthropic
**Trust:** Trusted
**Capabilities:**
- `generate_api_client` - Generate API client from specs
- `validate_api_spec` - Validate OpenAPI/Swagger specs
## Task Matching Flow
1. **Task Received**: "Review this code for security and performance"
2. **Keyword Extraction**: ['security', 'performance', 'review', 'code']
3. **Plugin Matching**: Code Reviewer (2.9 relevance score)
4. **Skill Matching**:
- `code-reviewer:security_review` (relevance: 2.9)
- `code-reviewer:performance_review` (relevance: 2.9)
5. **Recommendations Generated**:
- Primary: security_review
- Alternatives: performance_review
- Sequence: security → performance
6. **Task Dispatched** with plugin context
## Knowledge Graph Integration
Plugin data is exported to shared knowledge graph with:
### Entities
- **Plugins**: name, vendor, version, description, trust_level, capabilities
- **Skills**: name, category, tags, keywords, trust_level, plugin_id
- **Categories**: code-analysis, security, performance, integration
### Relations
- `plugin provides_capability skill`
- `plugin supports_category category`
- `skill belongs_to_category category`
### Access
```bash
# Query plugins for a task
luzia docs search "plugin marketplace"
# Show plugin entity
luzia docs --show "Code Simplifier"
# Show knowledge graph stats
luzia docs --stats
```
## Testing
Comprehensive test suite available:
```bash
# Run all tests
python3 tests/test_plugin_system.py
# Test output shows:
# - Registry tests (5 tests)
# - Skill tests (7 tests)
# - Capability matching (4 tests)
# - Dispatcher integration (5 tests)
# - KG export tests (6 tests)
# Results: 27/27 tests passed ✓
```
## Usage Examples
### Example 1: Find Plugins for Task
```python
import sys
sys.path.insert(0, 'lib')
from plugin_skill_loader import get_plugin_skill_loader
loader = get_plugin_skill_loader()
task = "simplify and optimize this Python function"
matched = loader.find_skills_for_task(task, min_relevance=0.3)
for skill in matched:
print(f"{skill['name']}: {skill['description']}")
```
### Example 2: Dispatch Task with Plugin Context
```python
from dispatcher_plugin_integration import PluginAwareTaskDispatcher
dispatcher = PluginAwareTaskDispatcher()
result = dispatcher.dispatch_with_plugin_context(
task_description='Review code for security issues',
project='my-project',
job_id='job-789'
)
print(f"Matched {len(result['plugin_context']['plugin_analysis']['matched_skills'])} skills")
print(f"Primary recommendation: {result['plugin_context']['recommended_plugins']['primary_skill']['name']}")
```
### Example 3: CLI Usage
```bash
# List all plugins and their capabilities
$ luzia plugins list
Name Vendor Trust Capabilities
Code Simplifier anthropic trusted 3
Code Reviewer anthropic trusted 3
API Integration Helper anthropic trusted 2
# Find plugins for a specific task
$ luzia plugins find "identify security vulnerabilities"
{
"query": "identify security vulnerabilities",
"matched_skills": [
{
"skill_id": "code-reviewer:security_review",
"name": "security_review (Code Reviewer)",
"relevance_score": 2.9,
...
}
],
"count": 1
}
# Export all plugin data
$ luzia plugins export
{
"action": "export_plugins",
"status": "success",
"files": {
"plugins_entities": "/tmp/.luzia-kg-exports/plugins_entities.json",
"skills_entities": "/tmp/.luzia-kg-exports/skills_entities.json",
"relationships": "/tmp/.luzia-kg-exports/relationships.json",
"complete_export": "/tmp/.luzia-kg-exports/complete_export.json"
},
"count": 4
}
```
## Configuration
Plugins are configured in `lib/plugin_marketplace.py` in the `OFFICIAL_PLUGINS` dictionary:
```python
OFFICIAL_PLUGINS = {
'plugin-id': {
'id': 'plugin-id',
'name': 'Plugin Name',
'description': 'Description',
'vendor': 'anthropic', # All official plugins
'version': '1.0.0',
'url': 'https://marketplace.claude.ai/plugins/...',
'capabilities': [
{
'name': 'capability_name',
'description': 'What it does',
'category': 'category',
'tags': ['tag1', 'tag2']
}
],
'trust_level': 'trusted',
'tags': ['plugin', 'tags']
}
}
```
## Adding New Plugins
To add a new official plugin:
1. Add entry to `OFFICIAL_PLUGINS` dict in `plugin_marketplace.py`
2. Define capabilities with categories and tags
3. Set trust_level to 'trusted' for official plugins
4. Regenerate skills: `python3 -c "from lib.plugin_skill_loader import generate_all_skills; generate_all_skills()"`
5. Export to KG: `python3 -c "from lib.plugin_kg_integration import export_plugins_to_kg; export_plugins_to_kg()"`
## Performance
- **Plugin Loading**: ~50ms (3 plugins)
- **Skill Generation**: ~100ms (8 skills from 3 plugins)
- **Task Matching**: ~10ms per task
- **Cache Hit**: <1ms
## Storage
- **Plugin Registry Cache**: `/tmp/.luzia-plugins/registry.json`
- **Plugin Skills Cache**: `/tmp/.luzia-plugin-skills/skills.json`
- **KG Exports**: `/tmp/.luzia-kg-exports/`
- `plugins_entities.json`: Plugin entities for KG
- `skills_entities.json`: Skill entities for KG
- `relationships.json`: All relationships
- `complete_export.json`: Complete export with metadata
## Future Enhancements
1. **Marketplace API Integration**: Fetch plugins from marketplace.claude.ai
2. **Dynamic Plugin Discovery**: Load plugins from URLs
3. **Plugin Authentication**: Support for authenticated plugins
4. **Custom Plugins**: User-defined plugins and capabilities
5. **Plugin Performance Metrics**: Track plugin effectiveness
6. **Community Plugins**: Support for community-contributed plugins (with separate trust level)
## Troubleshooting
### No plugins found
```bash
# Check if plugins are loaded
python3 -c "from lib.plugin_marketplace import get_marketplace_registry; r = get_marketplace_registry(); print(len(r.plugins))"
# Should output: 3 (for the official plugins)
```
### Skills not matching
```bash
# Check skill generation
python3 -c "from lib.plugin_skill_loader import get_plugin_skill_loader; l = get_plugin_skill_loader(); print(len(l.skills))"
# Should output: 8 (for the 3 official plugins)
```
### KG export fails
```bash
# Check export directory
ls -la /tmp/.luzia-kg-exports/
# Should contain 4 JSON files
```
## Integration with Luzia Flow
The plugin system integrates at multiple points:
1. **Task Dispatch**: Plugins matched before dispatcher queues task
2. **Skill Recommendation**: Top 3-5 skills suggested for execution
3. **Context Hydration**: Task context enriched with plugin metadata
4. **Knowledge Graph**: Plugin skills indexed for search/discovery
## See Also
- `tests/test_plugin_system.py` - Test suite
- `lib/responsive_dispatcher.py` - Task dispatcher
- `lib/flow_intelligence.py` - Task flow management
- `/opt/server-agents/orchestrator/docs/` - Full documentation

250
docs/README_HELP.md Normal file
View File

@@ -0,0 +1,250 @@
# Luzia Help & Documentation
Welcome to the Luzia command help system. This directory contains comprehensive documentation for using Luzia.
## Quick Start
Get help immediately:
```bash
luzia --help
luzia help
luzia -h
```
This displays a comprehensive overview of all available commands with examples.
## Documentation Files
### 1. **In-Command Help** (Immediate)
```bash
luzia --help
```
- **91 lines** of well-organized command categories
- Quick start examples
- All command syntax
- Global flags
- **Best for:** Quick reference while using CLI
### 2. **Cheat Sheet** (Quick Lookup)
📄 **[LUZIA_CHEAT_SHEET.md](./LUZIA_CHEAT_SHEET.md)**
- Essential commands (4 lines)
- Troubleshooting patterns
- Common workflows
- Exit codes
- **Best for:** Finding common patterns quickly
```bash
cat docs/LUZIA_CHEAT_SHEET.md
```
### 3. **Full Command Reference** (Complete)
📄 **[LUZIA_COMMAND_REFERENCE.md](./LUZIA_COMMAND_REFERENCE.md)**
- All 30+ commands with descriptions
- Detailed usage examples
- Pattern explanations
- Configuration guide
- Troubleshooting section
- **Best for:** Understanding all features thoroughly
```bash
cat docs/LUZIA_COMMAND_REFERENCE.md
```
### 4. **Update Summary**
📄 **[HELP_UPDATE_SUMMARY.md](./HELP_UPDATE_SUMMARY.md)**
- What was changed
- Testing results
- Feature coverage
- Files modified/created
- **Best for:** Understanding recent updates
## Command Categories
### 🚀 Quick Start (4 essential commands)
```bash
luzia --help # Show help
luzia list # List projects
luzia status # Check status
luzia <project> <task> # Run task
```
### 💼 Core Project Commands
- Execute tasks in Docker containers
- Interactive sessions
- Manage project containers
- View project history
### 🔧 Maintenance & System
- Full cleanup with preview
- Job and container management
- System recommendations
- Log viewing
### 🔄 Failure Management
- List failures by exit code
- Show detailed error information
- Smart retry (single or batch)
- Kill stuck jobs
### 📚 Knowledge Graph & QA
- QA validation
- Documentation search (multi-domain)
- Entity lookup
- Statistics
- Sync capabilities
### 🔍 Research (3-Phase Flow)
- Research initiation
- Session management
- Knowledge graph display
- Context → search → synthesize
### 🏗️ Code Analysis
- Structure analysis
- Project-specific analysis
- Subdirectory analysis
- JSON output
### 🧠 Advanced Features
- Deep reasoning (Zen + Gemini 3)
- Troubleshooting assistant
- Queue management
- Notifications
### 🔨 Low-Level Operations
- Raw command execution
- File read/write
- Context retrieval
- JSON output
## Common Workflows
### Check System Health
```bash
luzia list # See all projects
luzia status # Overall status
luzia maintenance # System recommendations
```
### Run a Task
```bash
luzia <project> <task> [args]
```
### Troubleshoot Failures
```bash
luzia failures # See what failed
luzia failures --summary # Breakdown by error code
luzia retry <job_id> # Retry one
luzia failures --auto-retry # Retry all fixable
```
### Research a Topic
```bash
luzia research musica "your question"
luzia research-show <session_id>
luzia research-knowledge musica
```
### Analyze Code
```bash
luzia structure dss --json
luzia docs "search term"
luzia qa --sync
```
### Clean Up
```bash
luzia cleanup --dry-run # Preview
luzia cleanup # Execute
```
## Help System Organization
```
┌─ QUICK HELP (In-Command)
│ └─ luzia --help (91 lines, organized by category)
│
├─ CHEAT SHEET (Quick Patterns)
│ └─ Common workflows, quick lookups
│
├─ FULL REFERENCE (Complete Details)
│ └─ All commands, examples, patterns
│
└─ UPDATE SUMMARY (Change Log)
└─ What changed, testing, coverage
```
## Finding What You Need
**I want to...**
- ✅ See all commands → `luzia --help` or `LUZIA_CHEAT_SHEET.md`
- ✅ Find a command → `LUZIA_COMMAND_REFERENCE.md`
- ✅ See examples → `LUZIA_CHEAT_SHEET.md` (patterns) or `LUZIA_COMMAND_REFERENCE.md` (details)
- ✅ Understand a feature → `LUZIA_COMMAND_REFERENCE.md`
- ✅ Troubleshoot → `LUZIA_COMMAND_REFERENCE.md` (troubleshooting section)
- ✅ Know what changed → `HELP_UPDATE_SUMMARY.md`
- ✅ Work interactively → `luzia --help` (in-command reference)
## Global Flags
Available with any command:
```bash
luzia --help # Show this help
luzia --verbose # Detailed output
luzia --fg # Run in foreground (don't background)
```
## Exit Codes
| Code | Meaning |
|------|---------|
| 0 | Success |
| 1 | General error |
| 2 | Invalid arguments |
| 3 | Project not found |
| 4 | Container error |
## Getting Help
1. **Immediate Help:** `luzia --help`
2. **Quick Patterns:** `cat docs/LUZIA_CHEAT_SHEET.md`
3. **Full Details:** `cat docs/LUZIA_COMMAND_REFERENCE.md`
4. **Recent Changes:** `cat docs/HELP_UPDATE_SUMMARY.md`
## Contributing
When adding new commands to Luzia:
1. Update the main help docstring in `bin/luzia` (lines 1-92)
2. Add examples to `LUZIA_CHEAT_SHEET.md`
3. Add full documentation to `LUZIA_COMMAND_REFERENCE.md`
4. Update this README if significant changes
## File Locations
```
/opt/server-agents/orchestrator/
├── bin/
│ └── luzia # Main script (help in docstring)
└── docs/
├── README_HELP.md # This file
├── LUZIA_CHEAT_SHEET.md # Quick reference
├── LUZIA_COMMAND_REFERENCE.md # Full documentation
├── HELP_UPDATE_SUMMARY.md # Recent changes
└── ...
```
## Support
For issues or questions:
1. Check `luzia --help` for command syntax
2. Read `LUZIA_COMMAND_REFERENCE.md` for usage patterns
3. See `LUZIA_CHEAT_SHEET.md` for common workflows
4. Review `HELP_UPDATE_SUMMARY.md` for recent changes
---
**Last Updated:** January 9, 2026
**Help System Version:** 2.0
**Coverage:** 27 command handlers, 9 categories, 90+ examples

429
docs/RESPONSIVE-DISPATCHER.md Normal file
View File

@@ -0,0 +1,429 @@
# Responsive Dispatcher - Non-blocking Task Dispatch
## Overview
The Responsive Dispatcher is a new subsystem in Luzia that enables **non-blocking task dispatch** with **immediate job_id return** and **live status tracking**. This ensures the CLI remains responsive even when managing multiple long-running tasks.
### Key Features
1. **Immediate Return**: Task dispatch returns a job_id within milliseconds
2. **Background Processing**: All job monitoring happens asynchronously
3. **Status Polling**: Check job status without blocking the main CLI
4. **Concurrent Management**: Track multiple concurrent tasks independently
5. **Live Feedback**: Pretty-printed status updates with progress indicators
6. **Status Caching**: Fast status retrieval with intelligent cache invalidation
## Architecture
### Components
```
┌─────────────────────┐
│ CLI (Luzia) │
│ "luzia <proj>..." │
└──────────┬──────────┘
│
▼
┌─────────────────────────────────────────┐
│ EnhancedDispatcher │
│ - dispatch_and_report() │
│ - get_status_and_display() │
│ - show_jobs_summary() │
└──────────┬──────────────────────────────┘
│
┌────┴────┐
▼ ▼
┌──────────┐ ┌──────────────────────┐
│Response │ │ Background Monitor │
│Dispatcher│ │ (Thread) │
└──────────┘ │ - Polls job status │
│ - Updates status.json│
│ - Detects completion │
└──────────────────────┘
Job Status (persisted):
- /var/lib/luzia/jobs/<job_id>/
├── status.json (updated by monitor)
├── output.log (agent output)
├── meta.json (job metadata)
└── progress.md (progress tracking)
```
### Task Dispatch Flow
```
1. User: luzia project "natural language task"
↓
2. CLI: route_project_task()
↓
3. Enhanced Dispatcher: dispatch_and_report()
├─ Create job directory (/var/lib/luzia/jobs/<job_id>/)
├─ Write initial status.json (dispatched)
├─ Queue job for background monitoring
└─ Return job_id immediately (<100ms)
↓
4. CLI Output: "agent:project:job_id"
↓
5. Background (async):
├─ Monitor waits for agent to start
├─ Polls output.log for progress
├─ Updates status.json with live info
└─ Detects completion and exit code
6. User: luzia jobs job_id (anytime)
↓
7. CLI: display current status
└─ No waiting, instant feedback
```
## Usage Guide
### Dispatching Tasks
Tasks now return immediately:
```bash
$ luzia overbits "fix the login button"
✓ Dispatched
Job ID: 113754-a2f5
Project: overbits
Use: luzia jobs to view status
luzia jobs 113754-a2f5 for details
```
The job runs in the background while you can continue using the CLI.
### Checking Job Status
View a specific job:
```bash
$ luzia jobs 113754-a2f5
113754-a2f5 running 42% overbits Building solution...
Details:
Job ID: 113754-a2f5
Project: overbits
Status: running
Progress: 42%
Message: Building solution...
Created: 2025-01-09T10:23:45.123456
Updated: 2025-01-09T10:24:12.456789
```
### List All Jobs
See all recent jobs:
```bash
$ luzia jobs
Recent Jobs:
Job ID Status Prog Project Message
----------------------------------------------------------------------------------------------------
113754-a2f5 running 42% overbits Building solution...
113754-8e4b running 65% musica Analyzing audio...
113754-7f2d completed 100% dss Task completed
113754-5c9a failed 50% librechat Connection error
```
### Monitor Specific Job (Interactive)
Watch a job's progress in real-time:
```bash
$ luzia jobs 113754-a2f5 --watch
Monitoring job: 113754-a2f5
starting [░░░░░░░░░░░░░░░░░░░░] 5% Agent initialization
running [██████░░░░░░░░░░░░░░] 30% Installing dependencies
running [████████████░░░░░░░░] 65% Building project
running [██████████████████░░] 95% Running tests
completed [██████████████████████] 100% Task completed
Final Status:
Details:
Job ID: 113754-a2f5
Project: overbits
Status: completed
Progress: 100%
Message: Task completed
Exit Code: 0
```
### Multiple Concurrent Tasks
Dispatch multiple tasks at once:
```bash
$ luzia overbits "fix button"
agent:overbits:113754-a2f5
$ luzia musica "analyze audio"
agent:musica:113754-8e4b
$ luzia dss "verify signature"
agent:dss:113754-9f3c
$ luzia jobs
Task Summary:
Running: 3
Pending: 0
Completed: 0
Failed: 0
Currently Running:
113754-a2f5 running 42% overbits Building...
113754-8e4b running 65% musica Analyzing...
113754-9f3c starting 5% dss Initializing...
```
All tasks run concurrently without blocking each other!
## Implementation Details
### Status File Format
Each job has a `status.json` that tracks its state:
```json
{
"id": "113754-a2f5",
"project": "overbits",
"task": "fix the login button",
"status": "running",
"priority": 5,
"progress": 42,
"message": "Building solution...",
"dispatched_at": "2025-01-09T10:23:45.123456",
"updated_at": "2025-01-09T10:24:12.456789",
"exit_code": null
}
```
Status transitions:
- `dispatched` → `starting` → `running` → `completed`
- `running` → `failed` (if exit code != 0)
- `running` → `stalled` (if no output for 30+ seconds)
- Any state → `killed` (if manually killed)
### Background Monitor
The responsive dispatcher starts a background monitor thread that:
1. Polls job queues for new tasks
2. Waits for agents to start (checks output.log / meta.json)
3. Monitors execution (reads output.log size, parses exit codes)
4. Updates status.json atomically
5. Detects stalled jobs (no output for 30 seconds)
6. Maintains job completion history
### Cache Strategy
Status caching ensures fast retrieval:
- Cache expires after **1 second** of no updates
- `get_status(job_id, use_cache=True)` returns instantly from cache
- `get_status(job_id, use_cache=False)` reads from disk (fresh data)
- Cache is automatically invalidated when status is updated
```python
# Fast cached read (if < 1 sec old)
status = dispatcher.get_status(job_id)
# Force fresh read from disk
status = dispatcher.get_status(job_id, use_cache=False)
```
## API Reference
### ResponseiveDispatcher
Core non-blocking dispatcher:
```python
from lib.responsive_dispatcher import ResponseiveDispatcher
dispatcher = ResponseiveDispatcher()
# Dispatch and get job_id immediately
job_id, status = dispatcher.dispatch_task(
project="overbits",
task="fix login button",
priority=5
)
# Get current status (with cache)
status = dispatcher.get_status(job_id)
# Update status (used by monitor)
dispatcher.update_status(
job_id,
status="running",
progress=50,
message="Processing..."
)
# List jobs
jobs = dispatcher.list_jobs(project="overbits", status_filter="running")
# Wait for completion (blocking)
final_status = dispatcher.wait_for_job(job_id, timeout=3600)
# Stream updates (for interactive display)
dispatcher.stream_status(job_id)
# Start background monitor
monitor_thread = dispatcher.start_background_monitor()
```
### CLIFeedback
Pretty-printed feedback for CLI:
```python
from lib.cli_feedback import CLIFeedback
feedback = CLIFeedback()
# Show job dispatch confirmation
feedback.job_dispatched(job_id, project, task)
# Display status with progress bar
feedback.show_status(status, show_full=True)
# List jobs formatted nicely
feedback.show_jobs_list(jobs)
# Show summary of concurrent jobs
feedback.show_concurrent_jobs(jobs)
```
### EnhancedDispatcher
High-level dispatcher with integrated feedback:
```python
from lib.dispatcher_enhancements import EnhancedDispatcher
enhanced = EnhancedDispatcher()
# Dispatch and show feedback automatically
job_id, status = enhanced.dispatch_and_report(
project="overbits",
task="fix button",
show_details=True,
show_feedback=True
)
# Get status and display
status = enhanced.get_status_and_display(job_id, show_full=True)
# Show jobs summary
enhanced.show_jobs_summary(project="overbits")
# Show all concurrent jobs
enhanced.show_concurrent_summary()
```
## Integration with Luzia CLI
The responsive dispatcher is integrated into the main Luzia CLI:
```python
# In route_project_task() handler:
dispatcher = get_enhanced_dispatcher()
job_id, status = dispatcher.dispatch_and_report(
project,
task,
show_details=True,
show_feedback=True
)
# Output job_id for tracking
print(f"agent:{project}:{job_id}")
```
## Testing
Run the comprehensive test suite:
```bash
python3 tests/test_responsive_dispatcher.py
```
Tests cover:
- ✓ Immediate dispatch with sub-millisecond response
- ✓ Job status retrieval and updates
- ✓ Concurrent job handling
- ✓ Status caching behavior
- ✓ CLI feedback rendering
- ✓ Progress bar visualization
- ✓ Background monitoring queue
## Performance
Dispatch latency (measured):
- **Dispatch only**: <50ms
- **With feedback**: <100ms
- **Status retrieval (cached)**: <1ms
- **Status retrieval (fresh)**: <5ms
- **Job listing**: <20ms
Memory overhead:
- Per job: ~2KB (status.json + metadata)
- Monitor thread: ~5MB
- Cache: ~100KB per 1000 jobs
## Configuration
Dispatcher behavior can be customized via environment variables:
```bash
# Cache expiration (seconds)
export LUZIA_CACHE_TTL=2
# Monitor poll interval (seconds)
export LUZIA_MONITOR_INTERVAL=1
# Max job history
export LUZIA_MAX_JOBS=500
```
## Troubleshooting
### Job stuck in "dispatched" status
The agent may have failed to start. Check:
```bash
cat /var/lib/luzia/jobs/<job_id>/output.log
cat /var/lib/luzia/jobs/<job_id>/meta.json
```
### Status not updating
Ensure background monitor is running:
```bash
luzia monitor status
```
### Cache returning stale status
Force fresh read:
```python
status = dispatcher.get_status(job_id, use_cache=False)
```
## Future Enhancements
- [ ] Web dashboard for job monitoring
- [ ] WebSocket support for real-time updates
- [ ] Job retry with exponential backoff
- [ ] Job cancellation with graceful shutdown
- [ ] Resource-aware scheduling
- [ ] Job dependencies and DAG execution
- [ ] Slack/email notifications on completion

235
docs/SKILL_LEARNING_QUICKSTART.md Normal file
View File

@@ -0,0 +1,235 @@
# Skill Learning System - Quick Start
## TL;DR
The skill learning system automatically learns from successful tasks and QA passes, storing learnings in the knowledge graph to improve future recommendations.
**Enable it in one line:**
```bash
python3 lib/qa_validator.py --learn --sync --verbose
```
## How It Works
1. **Task Completes** → QA validation passes
2. **System Analyzes** → Extracts skills used, patterns, tools
3. **Learning Created** → Stores in knowledge graph with metadata
4. **Future Tasks** → System recommends relevant skills based on prompt
## Basic Usage
### Run QA with Learning Extraction
```bash
# Standard QA validation only
python3 lib/qa_validator.py --sync --verbose
# With learning extraction enabled
python3 lib/qa_validator.py --learn --sync --verbose
```
### Extract Learnings from Completed Task
```python
from lib.skill_learning_engine import SkillLearningSystem
system = SkillLearningSystem()
task_data = {
"task_id": "my_task",
"prompt": "Refactor authentication module",
"project": "overbits",
"status": "success",
"tools_used": ["Bash", "Read", "Edit"],
"duration": 45.2,
"result_summary": "Successfully refactored",
"qa_passed": True,
"timestamp": "2026-01-09T12:00:00"
}
qa_results = {
"passed": True,
"results": {"syntax": True, "routes": True},
"summary": {"errors": 0}
}
result = system.process_task_completion(task_data, qa_results)
print(f"Learning ID: {result['learning_id']}")
```
### Get Recommendations for a Task
```python
system = SkillLearningSystem()
recommendations = system.get_recommendations(
task_prompt="Debug authentication issue",
project="overbits"
)
for rec in recommendations:
print(f"{rec['skill']}: {rec['confidence']:.0%} confidence")
```
### View Learned Skills Profile
```python
profile = system.get_learning_summary()
print(f"Total learnings: {profile['total_learnings']}")
print(f"Top skills: {profile['top_skills']}")
```
## What Gets Learned
The system extracts and learns:
### Tool Usage
- Which tools are used for which tasks
- Tool frequency and patterns
- Tool combinations that work well together
### Decision Patterns
- **Optimization**: Performance improvement approaches
- **Debugging**: Error diagnosis and fixing strategies
- **Testing**: Validation and verification techniques
- **Refactoring**: Code improvement methods
- **Documentation**: Documentation practices
- **Integration**: System integration approaches
- **Automation**: Automation and scheduling patterns
### Project Knowledge
- Which projects benefit from which approaches
- Project-specific tool combinations
- Project patterns and best practices
### Quality Metrics
- Success rates by tool combination
- Task completion times
- QA pass rates by category
## Storage
All learnings stored in the **research knowledge graph**:
```
/etc/luz-knowledge/research.db
```
Query learnings:
```bash
python3 lib/knowledge_graph.py search "optimization"
python3 lib/knowledge_graph.py list research finding
```
## Examples
### Example 1: Learn from Database Optimization
```bash
# Task completes successfully with QA passing
python3 lib/qa_validator.py --learn --sync
# System automatically:
# - Identifies tools used: Bash, Read, Edit
# - Recognizes pattern: optimization
# - Stores learning about database optimization
# - Creates relations between tools and pattern
```
### Example 2: Get Recommendations
```python
# Later, for similar task:
recommendations = system.get_recommendations(
"Optimize API endpoint performance",
project="overbits"
)
# Might suggest:
# - Use Bash for performance analysis
# - Use Edit for code changes
# - Watch for optimization patterns
# - Similar to previous successful tasks
```
### Example 3: Build Team Knowledge
Run multiple tasks with learning enabled:
```bash
# Day 1: Deploy task with --learn
python3 lib/qa_validator.py --learn --sync
# Day 2: Optimization task with --learn
python3 lib/qa_validator.py --learn --sync
# Day 3: Similar deployment task
# System now has learnings from both previous tasks
recommendations = system.get_recommendations("Deploy new version")
```
## Statistics and Monitoring
View learning system statistics:
```bash
python3 lib/qa_learning_integration.py --stats
```
Output:
```
=== QA Learning Integration Statistics ===
total_events: 42
qa_passed: 40
learnings_extracted: 38
extraction_rate: 0.95
last_event: 2026-01-09T12:00:00
```
## Testing
Quick test of the system:
```bash
python3 lib/skill_learning_engine.py test
```
Full test suite:
```bash
python3 -m pytest tests/test_skill_learning.py -v
```
## Troubleshooting
### "No learnings extracted"
- Check that QA actually passed
- Verify knowledge graph is accessible
- Run with `--verbose` to see details
### "Empty recommendations"
- Need to complete tasks with `--learn` first
- Task prompt must match learning keywords
- Check knowledge graph has entries:
```bash
python3 lib/knowledge_graph.py list research finding
```
### "Permission denied"
- Check `/etc/luz-knowledge/` permissions
- Ensure user is in `ai-users` group
- Check knowledge graph domain permissions
## Next Steps
1. **Start collecting learnings**: Run tasks with `--learn`
2. **Monitor learnings**: Check statistics and knowledge graph
3. **Use recommendations**: Integrate into task routing
4. **Refine patterns**: Add custom extraction patterns as needed
## Learn More
- Full documentation: [SKILL_LEARNING_SYSTEM.md](./SKILL_LEARNING_SYSTEM.md)
- Source code: `lib/skill_learning_engine.py`
- Integration: `lib/qa_learning_integration.py`
- Tests: `tests/test_skill_learning.py`

425
docs/SKILL_LEARNING_SYSTEM.md Normal file
View File

@@ -0,0 +1,425 @@
# Skill and Knowledge Learning System
## Overview
The Skill and Knowledge Learning System automatically extracts learnings from completed tasks and QA passes, storing them in the shared knowledge graph for future skill recommendations and continuous decision-making improvements.
This system enables Luzia to:
- **Learn from successes**: Extract patterns from passing QA validations
- **Build skill profiles**: Aggregate tool usage, patterns, and decision-making approaches
- **Make recommendations**: Suggest effective approaches for similar future tasks
- **Improve over time**: Store learnings persistently for cross-session learning
## Architecture
### Components
```
TaskExecution
↓
TaskAnalyzer → Patterns & Metadata
↓
SkillExtractor → Extracted Skills
↓
LearningEngine → Learning Objects
↓
KnowledgeGraph (research domain)
↓
SkillRecommender → Task Recommendations
```
### Core Classes
#### 1. **TaskAnalyzer**
Analyzes task executions to extract patterns and metadata.
```python
from lib.skill_learning_engine import TaskAnalyzer
analyzer = TaskAnalyzer()
# Analyze a single task
execution = analyzer.analyze_task({
"task_id": "task_001",
"prompt": "Refactor database schema",
"project": "overbits",
"status": "success",
"tools_used": ["Bash", "Read", "Edit"],
"duration": 45.2,
"result_summary": "Schema refactored successfully",
"qa_passed": True,
"timestamp": "2026-01-09T12:00:00"
})
# Extract patterns from multiple executions
patterns = analyzer.extract_patterns(executions)
# Returns: success_rate, average_duration, common_tools, etc.
```
#### 2. **SkillExtractor**
Extracts skills from task executions and QA results.
```python
from lib.skill_learning_engine import SkillExtractor
extractor = SkillExtractor()
# Extract skills from task
skills = extractor.extract_from_task(execution)
# Extract skills from QA results
qa_skills = extractor.extract_from_qa_results(qa_results)
# Aggregate multiple skill extractions
aggregated = extractor.aggregate_skills(all_skills)
```
**Skill Categories:**
- `tool_usage`: Tools used in task (Read, Bash, Edit, etc.)
- `pattern`: Task patterns (optimization, debugging, testing, etc.)
- `decision`: Decision-making approaches
- `architecture`: Project/system knowledge
#### 3. **LearningEngine**
Processes and stores learnings in the knowledge graph.
```python
from lib.skill_learning_engine import LearningEngine
engine = LearningEngine()
# Extract a learning from successful task
learning = engine.extract_learning(execution, skills, qa_results)
# Store in knowledge graph
learning_id = engine.store_learning(learning)
# Create skill entities
skill_id = engine.create_skill_entity(skill)
```
#### 4. **SkillRecommender**
Recommends skills for future tasks based on stored learnings.
```python
from lib.skill_learning_engine import SkillRecommender
recommender = SkillRecommender()
# Get recommendations for a task
recommendations = recommender.recommend_for_task(
task_prompt="Optimize database performance",
project="overbits"
)
# Get overall skill profile
profile = recommender.get_skill_profile()
```
#### 5. **SkillLearningSystem**
Unified orchestrator for the complete learning pipeline.
```python
from lib.skill_learning_engine import SkillLearningSystem
system = SkillLearningSystem()
# Process a completed task with QA results
result = system.process_task_completion(task_data, qa_results)
# Result includes: skills_extracted, learning_created, learning_id
# Get recommendations
recommendations = system.get_recommendations(prompt, project)
# Get learning summary
summary = system.get_learning_summary()
```
## Integration with QA Validator
The learning system integrates seamlessly with the QA validator:
### Manual Integration
```python
from lib.qa_learning_integration import QALearningIntegrator
integrator = QALearningIntegrator()
# Run QA with automatic learning extraction
result = integrator.run_qa_and_sync_with_learning(sync=True, verbose=True)
```
### Via CLI
```bash
# Standard QA validation
python3 lib/qa_validator.py
# QA validation with learning extraction
python3 lib/qa_validator.py --learn --sync --verbose
# Get statistics on learning integration
python3 lib/qa_learning_integration.py --stats
```
## Knowledge Graph Storage
Learnings are stored in the `research` domain of the knowledge graph:
```
Entity Type: finding
Name: learning_20260109_120000_Refactor_Database_Schema
Content:
- Title: Refactor Database Schema
- Description: Task execution details
- Skills Used: tool_bash, tool_read, tool_edit, ...
- Pattern: refactoring_pattern
- Applicability: overbits, tool_bash, decision, ...
- Confidence: 0.85
Metadata:
- skills: [list of skill names]
- pattern: refactoring_pattern
- confidence: 0.85
- applicability: [projects, tools, categories]
- extraction_time: ISO timestamp
```
### Accessing Stored Learnings
```python
from lib.knowledge_graph import KnowledgeGraph
kg = KnowledgeGraph("research")
# Search for learnings
learnings = kg.search("database optimization", limit=10)
# Get specific learning
learning = kg.get_entity("learning_20260109_120000_Refactor_Database_Schema")
# Get related skills
relations = kg.get_relations("learning_20260109_120000_...")
# List all learnings
all_learnings = kg.list_entities(entity_type="finding")
```
## Usage Examples
### Example 1: Extract Learnings from Task Completion
```python
from lib.skill_learning_engine import SkillLearningSystem
system = SkillLearningSystem()
# Task data from execution
task_data = {
"task_id": "deploy_overbits_v2",
"prompt": "Deploy new frontend build to production with zero downtime",
"project": "overbits",
"status": "success",
"tools_used": ["Bash", "Read", "Edit"],
"duration": 120.5,
"result_summary": "Successfully deployed with no downtime, 100% rollback verified",
"qa_passed": True,
"timestamp": "2026-01-09T15:30:00"
}
# QA validation results
qa_results = {
"passed": True,
"results": {
"syntax": True,
"routes": True,
"command_docs": True,
},
"summary": {
"errors": 0,
"warnings": 0,
"info": 5,
}
}
# Process and extract learnings
result = system.process_task_completion(task_data, qa_results)
print(f"Skills extracted: {result['skills_extracted']}")
print(f"Learning created: {result['learning_id']}")
```
### Example 2: Get Recommendations for Similar Task
```python
# Later, for a similar deployment task
new_prompt = "Deploy database migration to production"
recommendations = system.get_recommendations(new_prompt, project="overbits")
for rec in recommendations:
print(f"Skill: {rec['skill']}")
print(f"From learning: {rec['source_learning']}")
print(f"Confidence: {rec['confidence']:.1%}")
```
### Example 3: Build Skill Profile
```python
# Get overview of learned skills
profile = system.get_learning_summary()
print(f"Total learnings: {profile['total_learnings']}")
print(f"Skills by category: {profile['by_category']}")
print(f"Top 5 skills:")
for skill, count in profile['top_skills'][:5]:
print(f" {skill}: {count} occurrences")
```
## Testing
Run the comprehensive test suite:
```bash
python3 -m pytest tests/test_skill_learning.py -v
```
**Test Coverage:**
- Task analysis and pattern extraction
- Skill extraction from tasks and QA results
- Decision pattern recognition
- Skill aggregation
- Learning extraction and storage
- Skill recommendations
- Full integration pipeline
All tests pass with mocked knowledge graph to avoid dependencies.
## Configuration
The system is configured in the QA validator integration:
**File:** `lib/qa_learning_integration.py`
Key settings:
- **Knowledge Graph Domain**: `research` (all learnings stored here)
- **Learning Extraction Trigger**: QA pass with all validations successful
- **Skill Categories**: tool_usage, pattern, decision, architecture
- **Confidence Calculation**: Weighted average of skill confidence and QA pass rate
## Data Flow
```
Task Execution
↓
Task Analysis
├─→ Success rate: 85%
├─→ Average duration: 45 min
├─→ Common tools: [Bash, Read, Edit]
└─→ Project distribution: {overbits: 60%, dss: 40%}
↓
Skill Extraction
├─→ Tool skills (from tools_used)
├─→ Decision patterns (from prompt)
├─→ Project knowledge (from project)
└─→ QA validation skills
↓
Learning Creation
├─→ Title & description
├─→ Skill aggregation
├─→ Pattern classification
├─→ Confidence scoring
└─→ Applicability determination
↓
Knowledge Graph Storage
└─→ Entity: finding
Relations: skill → learning
Metadata: skills, pattern, confidence, applicability
↓
Future Recommendations
└─→ Search similar tasks
Extract applicable skills
Rank by confidence
```
## Performance Considerations
**Learning Extraction:**
- Runs only on successful QA passes (not a bottleneck)
- Async-ready (future enhancement)
- Minimal overhead (~100ms per extraction)
**Recommendation:**
- Uses FTS5 full-text search on KG
- Limited to top 10 results
- Confidence-ranked sorting
**Storage:**
- SQLite with FTS5 (efficient)
- Automatic indexing and triggers
- Scales to thousands of learnings
## Future Enhancements
1. **Async Extraction**: Background learning extraction during deployment
2. **Confidence Evolution**: Learnings gain/lose confidence based on outcomes
3. **Skill Decay**: Unused skills decrease in relevance over time
4. **Cross-Project Learning**: Share learnings between similar projects
5. **Decision Tracing**: Link recommendations back to specific successful tasks
6. **Feedback Loop**: Update learning confidence based on task outcomes
7. **Skill Trees**: Build hierarchies of related skills
8. **Collaborative Learning**: Share learnings across team instances
## Troubleshooting
### Learnings Not Being Created
Check:
1. QA validation passes (`qa_results["passed"] == True`)
2. Knowledge graph is accessible and writable
3. No errors in `qa_learning_integration.py` output
```bash
python3 lib/qa_validator.py --learn --verbose
```
### Recommendations Are Empty
Possible causes:
1. No learnings stored yet (run a successful task with `--learn`)
2. Task prompt doesn't match stored learning titles
3. Knowledge graph search not finding results
Test with:
```bash
python3 lib/skill_learning_engine.py recommend --task-prompt "Your task" --project overbits
```
### Knowledge Graph Issues
Check knowledge graph status:
```bash
python3 lib/knowledge_graph.py stats
python3 lib/knowledge_graph.py search "learning"
```
## API Reference
See inline documentation in:
- `lib/skill_learning_engine.py` - Main system classes
- `lib/qa_learning_integration.py` - QA integration
- `tests/test_skill_learning.py` - Usage examples via tests
## Contributing
To add new skill extraction patterns:
1. Add pattern to `SkillExtractor._extract_decision_patterns()`
2. Update test cases in `TestSkillExtractor.test_extract_decision_patterns()`
3. Test with: `python3 lib/skill_learning_engine.py test`
4. Document pattern in this guide
## License
Part of Luzia Orchestrator. See parent project license.

549
docs/SUB_AGENT_CONTEXT_FEATURE.md Normal file
View File

@@ -0,0 +1,549 @@
# Sub-Agent Context Feature - Complete Guide
## Overview
The Sub-Agent Context Feature enables intelligent task context propagation from parent tasks to sub-agents, facilitating multi-project coordination and intelligent workflow execution. This is a Phase 1 implementation that provides the foundational infrastructure for cross-project task coordination.
**Release Date:** 2026-01-09
**Status:** ✅ Production Ready (Phase 1)
**Test Coverage:** 20/20 tests passing (100%)
## Architecture
### Core Components
#### 1. **SubAgentContext** (Data Model)
Represents the execution context passed from a parent task to a sub-agent.
```python
@dataclass
class SubAgentContext:
parent_task_id: str # Reference to parent task
parent_project: str # Parent project name
parent_description: str # Parent task description
sub_agent_id: str # Unique sub-agent ID
created_at: str # Creation timestamp
parent_context: Dict[str, Any] # Context data from parent
parent_tags: List[str] # Tags/labels from parent
parent_metadata: Dict[str, Any] # Additional metadata
phase_progression: List[FlowPhase] # 9-phase flow tracking
sibling_agents: Set[str] # IDs of sibling agents
coordination_messages: List[Dict] # Inter-agent messages
```
#### 2. **SubAgentContextManager** (Core Manager)
Manages creation, retrieval, persistence, and coordination of sub-agent contexts.
**Key Methods:**
- `create_sub_agent_context()` - Create context for new sub-agent
- `get_sub_agent_context()` - Retrieve context by ID
- `update_phase()` - Update phase status with output/error
- `get_current_phase()` - Get current active phase
- `send_message_to_sibling()` - Coordinate with sibling agents
- `get_sibling_agents()` - Discover related agents
- `save_context()` / `load_contexts()` - Persistence management
#### 3. **SubAgentFlowIntegrator** (Flow Execution)
Integrates sub-agent context with the 9-phase Luzia flow for coordinated execution.
**Key Methods:**
- `execute_sub_agent_flow()` - Execute full 9-phase flow
- `execute_phase()` - Execute single phase
- `register_phase_handler()` - Register custom phase logic
- `get_sub_agent_progress()` - Progress reporting
- `coordinate_sub_agents()` - Multi-agent coordination
- `collect_sub_agent_results()` - Result aggregation
### 9-Phase Flow Execution
Each sub-agent executes through the standard 9-phase Luzia flow:
```
1. CONTEXT_PREP → Prepare parent context for processing
2. RECEIVED → Register sub-agent in system
3. PREDICTING → Predict requirements based on parent
4. ANALYZING → Analyze parent task characteristics
5. CONSENSUS_CHECK → Check coordination with siblings
6. AWAITING_APPROVAL → Wait for approval to proceed
7. STRATEGIZING → Plan execution strategy
8. EXECUTING → Execute sub-agent task
9. LEARNING → Learn from execution results
```
Each phase tracks:
- Status: pending, in_progress, completed, failed
- Output/results from phase execution
- Duration/performance metrics
- Error information if failed
- Timestamps for audit trail
### Sibling Discovery & Coordination
Sub-agents automatically discover siblings (other sub-agents from same parent):
```
Parent Task (e.g., "Implement authentication system")
├── Sub-Agent 1 (Build auth service)
│ └── Siblings: [Sub-Agent 2, Sub-Agent 3]
├── Sub-Agent 2 (Create UI components)
│ └── Siblings: [Sub-Agent 1, Sub-Agent 3]
└── Sub-Agent 3 (Write tests)
└── Siblings: [Sub-Agent 1, Sub-Agent 2]
```
**Coordination Message Types:**
- `request` - Ask sibling for data/assistance
- `update` - Share progress update
- `result` - Send completion result
- `dependency` - Indicate blocking dependency
## Usage Patterns
### Pattern 1: Simple Sub-Agent Creation
```python
from sub_agent_context import SubAgentContextManager
manager = SubAgentContextManager()
# Create context for a sub-agent
context = manager.create_sub_agent_context(
parent_task_id="task-123",
parent_project="admin",
parent_description="Setup production environment",
parent_context={"environment": "prod", "region": "us-east-1"},
parent_tags=["deployment", "critical"],
)
print(f"Sub-agent ID: {context.sub_agent_id}")
print(f"Current phase: {manager.get_current_phase(context.sub_agent_id)}")
```
### Pattern 2: Phase Progression Tracking
```python
# Execute and track phases
sub_agent_id = context.sub_agent_id
# Process each phase
for phase in context.phase_progression:
# Mark phase as in progress
manager.update_phase(sub_agent_id, phase.phase_name, "in_progress")
# Execute phase logic (replace with actual implementation)
try:
result = execute_phase_logic(phase.phase_name)
manager.update_phase(
sub_agent_id,
phase.phase_name,
"completed",
output=str(result)
)
except Exception as e:
manager.update_phase(
sub_agent_id,
phase.phase_name,
"failed",
error=str(e)
)
```
### Pattern 3: Sibling Coordination
```python
# Send coordination message to sibling
manager.send_message_to_sibling(
from_agent_id=sub_agent_1_id,
to_agent_id=sub_agent_2_id,
message_type="dependency",
content={
"depends_on": "database_setup",
"waits_until": "sub_agent_2_completes_schema"
}
)
# Check sibling relationships
siblings = manager.get_sibling_agents(sub_agent_1_id)
print(f"Siblings: {siblings}")
```
### Pattern 4: Flow Integration with Execution
```python
from sub_agent_flow_integration import SubAgentFlowIntegrator
integrator = SubAgentFlowIntegrator()
# Execute full sub-agent flow
results = integrator.execute_sub_agent_flow(
parent_task_id="task-456",
parent_project="admin",
parent_description="Deploy microservices",
parent_context={
"services": ["auth", "api", "database"],
"deployment_type": "kubernetes"
},
parent_tags=["deployment", "infrastructure"]
)
# Get progress
progress = integrator.get_sub_agent_progress(results["sub_agent_id"])
print(f"Progress: {progress['progress_percentage']:.1f}%")
print(f"Completed phases: {progress['completed_phases']}/{progress['total_phases']}")
```
### Pattern 5: Multi-Agent Coordination
```python
# Coordinate multiple sub-agents for same parent
coordination = integrator.coordinate_sub_agents(
parent_task_id="task-789",
coordination_strategy="sequential" # or "parallel", "dependency-based"
)
# Collect results from all sub-agents
results = integrator.collect_sub_agent_results("task-789")
print(f"Sub-agents: {results['sub_agents_total']}")
print(f"All complete: {results['all_sub_agents_complete']}")
for sub_agent in results['sub_agents']:
print(f"{sub_agent['sub_agent_id']}: {sub_agent['progress']['progress_percentage']:.1f}%")
```
### Pattern 6: Custom Phase Handlers
```python
# Register custom handler for specific phase
def handle_consensus_check(context):
"""Custom logic for CONSENSUS_CHECK phase"""
siblings = context.sibling_agents
ready = all(
integrator.context_manager.get_current_phase(sibling)
for sibling in siblings
)
return {"consensus_reached": ready, "siblings_ready": len(siblings)}
integrator.register_phase_handler("CONSENSUS_CHECK", handle_consensus_check)
# Phase will now use custom handler
result = integrator.execute_phase(sub_agent_id, "CONSENSUS_CHECK")
```
## API Reference
### SubAgentContextManager
#### `create_sub_agent_context(parent_task_id, parent_project, parent_description, ...)`
Creates a new sub-agent context.
**Parameters:**
- `parent_task_id` (str): ID of parent task
- `parent_project` (str): Name of parent project
- `parent_description` (str): Description of parent task
- `parent_context` (Dict, optional): Context data from parent
- `parent_tags` (List, optional): Tags/labels from parent
- `parent_metadata` (Dict, optional): Additional metadata
**Returns:** `SubAgentContext` instance
**Example:**
```python
context = manager.create_sub_agent_context(
parent_task_id="task-001",
parent_project="admin",
parent_description="Build authentication system",
parent_context={"environment": "production"},
parent_tags=["backend", "security"]
)
```
#### `update_phase(sub_agent_id, phase_name, status, output=None, error=None)`
Updates the status of a phase.
**Parameters:**
- `sub_agent_id` (str): ID of sub-agent
- `phase_name` (str): Name of phase to update
- `status` (str): New status (pending, in_progress, completed, failed)
- `output` (str, optional): Phase output/results
- `error` (str, optional): Error message if failed
**Returns:** `bool` - True if successful
#### `send_message_to_sibling(from_agent_id, to_agent_id, message_type, content)`
Sends coordination message to sibling agent.
**Parameters:**
- `from_agent_id` (str): Sending sub-agent ID
- `to_agent_id` (str): Receiving sub-agent ID
- `message_type` (str): Type (request, update, result, dependency)
- `content` (Dict): Message content
**Returns:** `bool` - True if message sent successfully
#### `get_context_summary(sub_agent_id)`
Gets human-readable summary of sub-agent context.
**Returns:** `Dict` with summary information
### SubAgentFlowIntegrator
#### `execute_sub_agent_flow(...)`
Executes complete 9-phase flow for sub-agent.
**Returns:** `Dict` with results from all phases
#### `execute_phase(sub_agent_id, phase_name)`
Executes single phase for sub-agent.
**Parameters:**
- `sub_agent_id` (str): ID of sub-agent
- `phase_name` (str): Name of phase to execute
**Returns:** `Dict` with phase execution results
#### `get_sub_agent_progress(sub_agent_id)`
Gets progress report for sub-agent.
**Returns:** `Dict` with progress metrics:
```python
{
"sub_agent_id": "...",
"total_phases": 9,
"completed_phases": 3,
"in_progress_phases": 1,
"failed_phases": 0,
"current_phase": "ANALYZING",
"progress_percentage": 33.3,
"total_duration_seconds": 1.234,
"phase_details": [...]
}
```
#### `coordinate_sub_agents(parent_task_id, coordination_strategy="sequential")`
Coordinates execution of multiple sub-agents.
**Parameters:**
- `parent_task_id` (str): ID of parent task
- `coordination_strategy` (str): "sequential", "parallel", or "dependency-based"
**Returns:** `Dict` with coordination plan
## Real-World Example: Multi-Project Feature Implementation
### Scenario
Implementing a new feature that requires work across multiple projects:
1. **librechat** - Frontend UI components
2. **musica** - Audio engine updates
3. **admin** - Configuration and testing
### Implementation
```python
from sub_agent_context import SubAgentContextManager
from sub_agent_flow_integration import SubAgentFlowIntegrator
manager = SubAgentContextManager()
integrator = SubAgentFlowIntegrator(manager)
# Create parent task context
parent_task_id = "feature-001"
parent_description = "Implement real-time audio collaboration"
parent_tags = ["feature", "audio", "collaboration"]
# Create sub-agents for each project
sub_agents = {}
projects = ["librechat", "musica", "admin"]
for project in projects:
results = integrator.execute_sub_agent_flow(
parent_task_id=parent_task_id,
parent_project=project,
parent_description=f"Implement collaboration in {project}",
parent_context={"feature": "audio_collab", "timeline": "2 weeks"},
parent_tags=parent_tags
)
sub_agents[project] = results["sub_agent_id"]
# Get overall progress
overall_results = integrator.collect_sub_agent_results(parent_task_id)
print(f"Feature implementation progress: {len(overall_results['sub_agents'])}/{len(projects)} started")
# Monitor coordination
for project, sub_agent_id in sub_agents.items():
progress = integrator.get_sub_agent_progress(sub_agent_id)
print(f"{project}: {progress['progress_percentage']:.0f}% complete")
```
## Performance Characteristics
### Context Operations (Local)
- Create sub-agent context: ~0.5ms
- Update phase status: ~1ms
- Retrieve context: ~0.1ms (in-memory), ~2ms (from disk)
- Send coordination message: ~0.5ms
### Phase Execution (Default Handlers)
- Execute single phase: ~1-5ms
- Full flow (9 phases): ~15-50ms
### Persistence
- Save context to disk: ~2-5ms (JSON serialization)
- Load context from disk: ~2-5ms (JSON deserialization)
### Scaling
- 10 sub-agents: <10ms total coordination
- 100 sub-agents: ~50ms total coordination
- 1000 sub-agents: ~500ms total coordination (linear scaling)
## Integration Points
### With Luzia Flow Orchestration
```python
# In luzia orchestrator when dispatching sub-tasks
integrator = SubAgentFlowIntegrator()
# Dispatch sub-agent for other project
results = integrator.execute_sub_agent_flow(
parent_task_id=current_task_id,
parent_project=target_project,
parent_description=subtask_description,
parent_context=current_context,
parent_tags=current_tags
)
```
### With Luzia CLI
```bash
# luzia will automatically create sub-agent context
luzia librechat implement ui-components
# Creates sub-agent that understands parent task context
```
### With Knowledge Graph
```python
# Store sub-agent coordination in shared KG
from shared_projects_memory import store_fact
store_fact(
entity_source_name=sub_agent_id,
relation="coordinates_with",
entity_target_name=sibling_id,
source_type="SubAgent",
target_type="SubAgent"
)
```
## Testing
### Running Tests
```bash
# Run all sub-agent context tests
python3 -m pytest tests/test_sub_agent_context.py -v
# Run specific test class
python3 -m pytest tests/test_sub_agent_context.py::TestFlowIntegration -v
# Run with coverage
python3 -m pytest tests/test_sub_agent_context.py --cov=lib/sub_agent_context
```
### Test Coverage (20/20 passing)
- ✅ Context creation and retrieval (3 tests)
- ✅ Sibling discovery (3 tests)
- ✅ Phase progression (4 tests)
- ✅ Sub-agent coordination (3 tests)
- ✅ Context persistence (1 test)
- ✅ Flow integration (4 tests)
- ✅ Context summary generation (1 test)
### Example Test
```python
def test_multiple_sub_agents_discover_siblings():
"""Test multiple sub-agents discover each other"""
agent1 = manager.create_sub_agent_context(
parent_task_id="parent-2",
parent_project="admin",
parent_description="Agent 1"
)
agent2 = manager.create_sub_agent_context(
parent_task_id="parent-2",
parent_project="admin",
parent_description="Agent 2"
)
assert agent2.sub_agent_id in manager.get_sibling_agents(agent1.sub_agent_id)
assert agent1.sub_agent_id in manager.get_sibling_agents(agent2.sub_agent_id)
```
## Phase 2 Roadmap (Future Enhancements)
1. **Advanced Coordination Strategies**
- Dependency graphs between sub-agents
- Resource-aware scheduling
- Priority-based execution
2. **Context Enrichment**
- Automatic parent context analysis
- Intelligent context filtering per sub-agent
- Context inheritance chains
3. **Monitoring & Observability**
- Real-time progress dashboards
- Performance analytics
- Execution traces and debugging
4. **Error Recovery**
- Automatic retry strategies
- Fallback execution paths
- Graceful degradation
5. **Integration Extensions**
- Git/VCS integration for sub-agent branching
- CI/CD pipeline hooks
- Deployment orchestration
## Troubleshooting
### Sub-agents not discovering siblings
**Cause:** Created with different `parent_task_id`
**Solution:** Ensure all related sub-agents use the same parent task ID
### Phase stuck in "in_progress"
**Cause:** Update call didn't complete successfully
**Solution:** Check manager.update_phase() return value and error logs
### Coordination messages not visible to recipient
**Cause:** Sub-agents not actually siblings
**Solution:** Verify recipient is in sender's sibling_agents set
### Context not persisting across restarts
**Cause:** Custom context_dir not configured
**Solution:** Specify persistent context_dir when creating SubAgentContextManager
## Contributing
When extending this feature:
1. **Add new phase handlers** in SubAgentFlowIntegrator
2. **Update tests** in test_sub_agent_context.py
3. **Document** coordination patterns
4. **Benchmark** performance impact
## License
Part of Luzia Governance Framework - MIT License
## References
- Parent Task: Luzia Governance LangChain Integration
- Related: Flow Intelligence (flow_intelligence.py)
- Integration: Luzia CLI (luzia_cli.py)
- Orchestration: Luzia Flow Engine

309
docs/TIME_METRICS.md Normal file
View File

@@ -0,0 +1,309 @@
# Luzia Time Metrics Integration
**Last Updated:** 2026-01-11
## Overview
Luzia now includes comprehensive time-based metrics and context tracking for task execution. This enables:
- **Task Timing**: Track dispatch, start, and completion times for all tasks
- **Duration Tracking**: Calculate and display task durations in various formats
- **System Context**: Capture CPU load, memory, and disk usage at dispatch and completion
- **Performance Baselines**: Establish normal task durations and detect anomalies
- **Causality Tracking**: Understand task sequencing and dependencies
## Quick Start
```bash
# View jobs with timing information
luzia jobs --timing
# View specific job with detailed timing
luzia jobs 123456-abcd
# View logs with timing header
luzia logs 123456-abcd
# View aggregate metrics
luzia metrics
# View project-specific metrics
luzia metrics musica --days 30
```
## Commands
### `luzia jobs`
Lists jobs with optional timing display.
```bash
# Standard output (shows elapsed time for running jobs)
luzia jobs
# Detailed timing columns
luzia jobs --timing
# Specific job details
luzia jobs <job_id>
```
Example output with `--timing`:
```
Job ID Project Status Dispatch Duration CPU
--------------------------------------------------------------------------------
123456-abcd admin completed 10:30:00 00:05:30 0.52
234567-efgh musica running 11:00:00 00:15:42 0.48
```
### `luzia logs`
Shows job output with timing header.
```bash
# With timing header
luzia logs <job_id>
# Without timing header
luzia logs <job_id> --no-header
```
Example header:
```
═════════════════════════════════════════════════════════════════
Job: agent:admin:123456-abcd
Agent: admin
Dispatched: 2026-01-11T10:30:00Z (America/Montevideo: 07:30:00)
Status: completed (took 00:05:30)
System: CPU 0.52, Memory 65%, Disk 45%
═════════════════════════════════════════════════════════════════
```
### `luzia metrics`
Shows aggregate task metrics and performance statistics.
```bash
# All projects summary
luzia metrics
# Specific project
luzia metrics <project>
# Custom time period
luzia metrics --days 30
# Success rate by duration bucket
luzia metrics --by-bucket
# Performance baseline
luzia metrics <project> --baseline
```
Example output:
```
=== Luzia Task Metrics (Last 7 Days) ===
Total Tasks: 45
Total Time: 02:15:30
By Project:
Project Tasks Time Avg Success
-------------------------------------------------------
admin 15 01:00:00 00:04:00 93.3%
musica 20 00:45:30 00:02:17 95.0%
dss 10 00:30:00 00:03:00 90.0%
Longest Running Tasks:
1. dss: 00:15:42
2. admin: 00:12:30
```
## Metadata Structure
Time metrics are stored in job `meta.json`:
```json
{
"id": "123456-abcd",
"project": "admin",
"task": "implement feature X",
"status": "completed",
"started": "2026-01-11T10:30:00.123456",
"time_metrics": {
"dispatch": {
"utc_time": "2026-01-11T10:30:00Z",
"agent_timezone": "America/Montevideo",
"system_load": [0.52, 0.48, 0.45],
"memory_percent": 65,
"disk_percent": 45
},
"completion": {
"utc_time": "2026-01-11T10:35:30Z",
"duration_seconds": 330,
"duration_formatted": "00:05:30",
"exit_code": 0,
"system_load": [0.48, 0.50, 0.47],
"memory_percent": 67,
"disk_percent": 45
}
},
"time_tracker_data": {
"task_id": "123456-abcd",
"project": "admin",
"dispatch_time": "2026-01-11T10:30:00Z",
"agent_timezone": "America/Montevideo"
}
}
```
## Performance Baselines
The system automatically collects task duration data to establish performance baselines:
```bash
# View current baseline for a project
luzia metrics musica --baseline
```
Output:
```
Performance Baseline:
Average: 00:02:17
Median: 00:01:45
P95: 00:05:30
Samples: 50
```
### Anomaly Detection
Tasks that run significantly longer than the baseline are flagged:
- **Normal**: Within P95
- **Slow**: > average + 2σ (flagged as potential issue)
- **Very Slow**: > average + 3σ (highlighted in reports)
- **Extreme**: > average + 4σ (should be investigated)
## Success Rate by Duration
Analyze success rates based on task duration:
```bash
luzia metrics --by-bucket
```
Output:
```
Success Rate by Duration:
Duration Total Success Rate
-----------------------------------------
< 1 minute 15 15 100.0%
1-5 minutes 20 19 95.0%
5-15 minutes 10 9 90.0%
15-30 minutes 5 4 80.0%
```
## Causality Tracking
Tasks can reference prior task completions to establish causal relationships:
```python
from time_metrics import find_prior_task
# Find the most recent task that completed before this dispatch
prior = find_prior_task(dispatch_time, project="admin")
if prior:
print(f"Started {prior['gap_formatted']} after {prior['task_id']} completed")
```
## Implementation Details
### Module Location
```
/opt/server-agents/orchestrator/lib/time_metrics.py
```
### Key Functions
| Function | Description |
|----------|-------------|
| `get_utc_now()` | Get current UTC timestamp |
| `calculate_duration_seconds(start, end)` | Calculate duration between timestamps |
| `format_duration(seconds)` | Format as HH:MM:SS |
| `format_duration_human(seconds)` | Format as "1h 30m 45s" |
| `capture_system_context()` | Get CPU, memory, disk snapshot |
| `create_task_time_metadata(task_id, project)` | Create dispatch metadata |
| `update_task_completion_metadata(meta, exit_code)` | Add completion info |
### Storage
- **Per-task metadata**: `/var/log/luz-orchestrator/jobs/<job_id>/meta.json`
- **Performance baselines**: `/var/lib/luzia/metrics/baselines.db`
## Timezone Support
All internal timestamps are in UTC. Display functions can convert to local time:
```python
from time_metrics import convert_to_local_time, format_timestamp_with_local
utc_ts = "2026-01-11T10:30:00Z"
# Convert to local time
local = convert_to_local_time(utc_ts, "America/Montevideo")
# Returns: "07:30:00"
# Format with both
formatted = format_timestamp_with_local(utc_ts, "America/Montevideo")
# Returns: "2026-01-11T10:30:00Z (America/Montevideo: 07:30:00)"
```
## Integration with Time MCP
The time metrics system can be enhanced with the Time MCP for more accurate timezone handling:
```python
# Via MCP
mcp__time__get_current_time(timezone="UTC")
mcp__time__convert_time(source_timezone="UTC", time="10:30", target_timezone="America/Montevideo")
```
## Best Practices
1. **Always use UTC for storage** - Only convert for display
2. **Capture context at dispatch** - System state affects task performance
3. **Monitor baselines regularly** - Detect performance regressions
4. **Use duration buckets** - Identify problematic task lengths
5. **Track causality** - Understand task dependencies
## Troubleshooting
### Time metrics not appearing
Check that the module is loaded:
```bash
python3 -c "from time_metrics import TIME_METRICS_AVAILABLE; print(TIME_METRICS_AVAILABLE)"
```
### Baseline database missing
Create the metrics directory:
```bash
mkdir -p /var/lib/luzia/metrics
```
### Duration showing as unknown
Ensure both dispatch and completion times are recorded:
```bash
cat /var/log/luz-orchestrator/jobs/<job_id>/meta.json | jq '.time_metrics'
```
## Testing
Run the test suite:
```bash
cd /opt/server-agents/orchestrator
python3 -m pytest tests/test_time_metrics.py -v
```