评估与自改进

openjiuwen/
├── auto_harness/                    # 教学基线上的编排器，本 commit 已空（仅 __init__.py）
├── rsi/                             # 递归自我改进
│   ├── auto_harness/orchestrator.py # ← 编排器迁移到这里
│   │   └── agents / stages / pipelines / rails / skills / experience ...
│   ├── evaluator/case_runner.py                (1493 行)
│   ├── evaluation_result_analyzer/analyzer.py  (2902 行)
│   ├── member_optimizer/action_planner.py      (1714 行)
│   ├── member_optimizer/action_executor.py     (2208 行)
│   └── single_harness/iterative.py             (2758 行)
└── agent_evolving/                  # 经验沉淀与调用优化
    ├── experience/  lifecycle · scorer · archive · tracker · rebuild
    ├── optimizer/   llm_call · memory_call · skill_call · tool_call
    ├── evaluator/ · dataset/ · checkpointing/ · agent_rl/

The pipeline is triggered when a root session starts, and only if:

- the session is not ephemeral
- the memory feature is enabled
- the session is not a sub-agent session
- the state DB is available

It runs asynchronously in the background and executes two phases in order: Phase 1, then Phase 2.

## Phase 1: Rollout Extraction (per-thread)
# → 领取到期 rollout、过滤出与记忆相关的响应项、并发送模型抽取结构化记忆、
#   脱敏后写回状态库；租约防重复，失败带退避重试。

## Phase 2: Global Consolidation
# → 取全局锁，按 usage_count / last_usage 排序筛选，合并成 raw_memories.md
#   与 rollout_summaries/，记忆根目录维护为 git 基线并写出 phase2_workspace_diff.md。

# feedback/ — recorded human feedback

The feedback family exposes two deliberately separate contracts: an immutable
remark in the canonical Session log, and editable feedback attached to one
assistant message in a local sidecar. Neither form enters the model conversation.

# packages/feedback/
# ├── command-feedback/   斜杠命令入口
# └── message-feedback/   挂在单条助手消息上的可编辑反馈（本地边车）

# BENCHMARK.md：基准测试走 Python SDK 的 jsonrpc-agent 最小变体运行，
#               要求每个基准任务使用独立 workspace 与 session id。

# Pi evals

Pi evals are behavioral, model-backed checks for Pi workflows. They adapt a real
`AgentSession` to `vitest-evals`, run it in isolated temporary project and agent
directories, and attach native Pi session artifacts.
Use them to measure end-to-end behavior and compare prompts, tools, skills,
models, or other harness configurations.

## Running evals

npm run eval -- --provider openai --model gpt-5.6-sol

# packages/evals/src/
# ├── pi-harness.ts      被测 Harness 的适配层
# ├── smoke.eval.ts
# └── extensions.eval.ts