Agent 评估:轨迹、工具调用与完成率

Agent 与普通 LLM 应用的差别

普通 QA:问题 → 答案(单步)。Agent:问题 → [工具1 → 观察 → 工具2 → 观察 → ...] → 答案(多步)。这多出来的"..."里藏着所有的脆弱性:

• 工具选对了吗?
• 参数填对了吗?
• 观察到错误后会调整吗?
• 会不会陷入无效循环?
• 是不是用了 10 步做了 3 步的事?

四维评估框架

Trajectory 评估的两种视角

视角 A:Exact Trajectory Match(严格匹配)

拿 Agent 实际走的轨迹与"参考轨迹"比对。适合固定 SOP 的场景。

# 工具调用序列的严格匹配
expected = [
    ("search_order", {"order_id": "12345"}),
    ("get_refund_policy", {"category": "electronics"}),
    ("initiate_refund", {"order_id": "12345", "reason": "defective"}),
]

actual = [
    ("search_order", {"order_id": "12345"}),
    ("get_refund_policy", {"category": "electronics"}),
    ("initiate_refund", {"order_id": "12345", "reason": "defective"}),
]

def trajectory_exact_match(expected, actual):
    return expected == actual

def tool_sequence_match(expected, actual):
    # 忽略参数,只看工具名序列
    return [t[0] for t in expected] == [t[0] for t in actual]

视角 B:LLM 裁判轨迹合理性

让 Judge 模型看完整条轨迹,判断每一步是否合理。

TRAJECTORY_JUDGE = """你是 Agent 行为审核员。下面是一个用户问题、Agent 走过的轨迹、最终答案。

用户问题: {question}
轨迹:
{trajectory}
最终答案: {answer}

请独立评估 4 个维度(1-5):

1. task_completion: 最终答案是否解决了用户问题
2. tool_selection:  每一步选的工具是否合适
3. arg_correctness: 工具参数是否正确
4. efficiency:      是否存在冗余/循环/不必要的步骤

返回 JSON:
{{
  "task_completion": {{"score": 1-5, "reason": "..."}},
  "tool_selection":  {{"score": 1-5, "reason": "...", "bad_steps": [1, 3]}},
  "arg_correctness": {{"score": 1-5, "reason": "..."}},
  "efficiency":      {{"score": 1-5, "reason": "...", "redundant_steps": [5]}}
}}"""

常见 Agent 失败模式清单

效率指标的量化

def agent_efficiency_metrics(trajectory, reference_steps=None):
    n_steps = len(trajectory)
    unique = len(set((t.name, frozenset(t.args.items())) for t in trajectory))
    dup_ratio = 1 - unique / n_steps if n_steps else 0

    total_tokens = sum(t.tokens for t in trajectory)
    total_latency = sum(t.latency_ms for t in trajectory)

    return {
        "n_steps": n_steps,
        "dup_ratio": dup_ratio,
        "total_tokens": total_tokens,
        "total_latency_ms": total_latency,
        "overshoot": n_steps / reference_steps if reference_steps else None,
    }

带环境的 Agent:成功判定

当 Agent 与真实环境交互(更新数据库、发邮件、操作浏览器),评估变成"目标状态是否达成"而非"输出看起来对不对"。

# 例:订单状态 Agent
def check_env_success(db, expected_order_id, expected_status):
    order = db.get_order(expected_order_id)
    return order and order.status == expected_status

# 评估时
def run_eval_case(case):
    env = create_isolated_env(case.initial_state)   # 每个 case 独立沙箱
    agent = build_agent(env)
    trajectory = agent.run(case.user_input)
    success = check_env_success(env.db, case.expected_order_id, case.expected_status)
    return {"success": success, "trajectory": trajectory, "env_final": env.snapshot()}

Benchmark 参考

评估新 Agent 时,公开 benchmark 是基线参考:

分步归因:错在哪一步

Agent 评估最重要的不是"成功率 72%",而是"失败的 28% 错在哪个环节"。一种实用分类:

def classify_failure(trajectory, final_correct):
    if final_correct:
        return "success"
    # 未调用任何工具就答
    if len(trajectory) == 0:
        return "no_tool_used"
    # 有死循环
    if has_loop(trajectory):
        return "infinite_loop"
    # 任一工具报错且未重试
    if any(t.error for t in trajectory) and not has_retry(trajectory):
        return "unhandled_error"
    # 参数校验失败
    if any(not validate_args(t) for t in trajectory):
        return "bad_arguments"
    # 步数超阈值
    if len(trajectory) >= MAX_STEPS:
        return "step_budget_exceeded"
    return "reasoning_failure"  # 剩下的多半是推理错

一个完整的 Agent 评估报告

# 评估结果(模拟输出)
n_cases: 200
success_rate: 0.72
avg_steps: 4.3
avg_tokens: 3200
avg_latency_ms: 8400

Failure breakdown:
  - reasoning_failure:   18 (9%)
  - bad_arguments:       12 (6%)
  - step_budget:          8 (4%)
  - infinite_loop:        6 (3%)
  - unhandled_error:      5 (2.5%)
  - no_tool_used:         5 (2.5%)

By difficulty:
  easy:   90% (70 cases)
  medium: 68% (90 cases)
  hard:   50% (40 cases)

Tool selection accuracy: 86%
Arg correctness:         83%
Trajectory efficiency:   0.79 (vs reference avg)

本章小结