Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions cmd/aiscan/profile.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import (
nativeext "github.com/chainreactors/cyber/exts/native"
nodeext "github.com/chainreactors/cyber/exts/node"
observeext "github.com/chainreactors/cyber/exts/observe"
otelext "github.com/chainreactors/cyber/exts/otel"
proxyext "github.com/chainreactors/cyber/exts/proxy"
ptyext "github.com/chainreactors/cyber/exts/pty"
recapext "github.com/chainreactors/cyber/exts/recap"
Expand Down Expand Up @@ -188,8 +189,27 @@ func buildAIScanProfile(config config) (*aiscanProfile, error) {
}
values = append(values, output)
}
if len(config.Observe) > 0 {
observer, err := observeext.New(observeext.Options{Kinds: config.Observe})
lookup := envLookup(config.Option)
options, err := otelext.OptionsFromEnv(lookup)
if err != nil {
return nil, err
}
otelEnabled := options.Endpoint != "" || options.TraceEndpoint != "" || options.LogEndpoint != "" || options.MetricEndpoint != ""
if otelEnabled {
options.ProviderName = config.Base.Provider.Config.Provider
observer, err := otelext.New(options)
if err != nil {
return nil, err
}
values = append(values, observer)
}
observeKinds := config.Observe
// Export existing operation facts whenever any OTel signal is configured.
if otelEnabled && len(observeKinds) == 0 {
observeKinds = []observeext.Kind{observeext.Tools, observeext.Commands, observeext.Processes}
}
if len(observeKinds) > 0 {
observer, err := observeext.New(observeext.Options{Kinds: observeKinds})
if err != nil {
return nil, err
}
Expand Down
2 changes: 2 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ cyber-harness 将模型、工具和运行环境组合为可以持续执行任务

## 开发者指南

现有 AOP 接入 OpenTelemetry、Agent 生命周期、token/工具计量和 SigNoz / Phoenix 配置见[Agent 可观测性](observability.md)。

[开发者指南](development.md)从可运行的 Go 工具组合推进到会话应用;[扩展开发](developer/extensions.md)说明工具、命令、知识与共享服务的贡献方式;[宿主集成](developer/hosting.md)说明生命周期及终端、Web、协议和协作接入。

构建入口、产物和依赖见[源码构建](../README_CN.md#构建与嵌入),选择所需能力见[自定义最小应用](../README_CN.md#自定义最小应用)。非 Go 客户端见[第三方语言集成](integration.md),字段与错误语义见[API 参考](api.md)。
Expand Down
161 changes: 161 additions & 0 deletions docs/observability.md

Large diffs are not rendered by default.

94 changes: 94 additions & 0 deletions examples/observability/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Cyber Harness 接入 SigNoz / Phoenix

正式 Cyber Agent 发布现有 AOP,`exts/otel` 将其转换为 traces、结构化日志和指标。Collector 将三种信号写入 SigNoz,并把相同 traces 写入 Phoenix。Agent / 工具无需新增 OTel hooks 或协议定义。映射及边界见 [Agent 可观测性](../../docs/observability.md)。

## 启动本地后端

```powershell
docker compose -f examples/observability/docker/docker-compose.yaml up -d --wait --wait-timeout 240
```

| 服务 | 地址 |
| --- | --- |
| SigNoz | http://127.0.0.1:24880 |
| Phoenix | http://127.0.0.1:24006 |
| OTLP/HTTP 基地址 | http://127.0.0.1:24318 |
| OTLP/gRPC Collector | 127.0.0.1:24317 |
| Collector 健康检查 | http://127.0.0.1:23133 |

固定版本为 SigNoz v0.129.0、Collector v0.144.5、Phoenix 20.20.0、ClickHouse 25.5.6。SigNoz 配置来自 [官方 v0.129.0 Compose](https://github.com/SigNoz/signoz/tree/v0.129.0/deploy/docker),附带的授权文本见 [SIGNOZ-LICENSE](common/SIGNOZ-LICENSE)。服务绑定本机,数据使用命名卷保存。首次访问 SigNoz 创建本地管理员;Phoenix 本地实例关闭身份验证。这组配置用于本地开发。

## 执行正式 Agent

在仓库根目录执行,模型密钥与配置沿用正常 Cyber 配置:

```powershell
$env:OTEL_EXPORTER_OTLP_ENDPOINT = 'http://127.0.0.1:24318'
$env:OTEL_EXPORTER_OTLP_PROTOCOL = 'http/protobuf'
$env:OTEL_SERVICE_NAME = 'cyber-harness'
$env:CYBER_OTEL_PROJECT = 'cyber-harness'
go run ./cmd/aiscan agent --prompt '用 Bash 执行 pwd,然后解释结果'
```

OTel 开启且未显式选择 `--observe` 时,自动启用现有 tools / commands / processes 观测。需要文件/HTTP 事实时使用 `--observe tools,commands,processes,files,http`;显式选择保持原样。OTLP 客户端使用 HTTP,gRPC 端口供其他客户端使用。

也可仅配置 `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` / `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` / `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` 启用对应信号;这些是完整 URL,优先于全局 endpoint,路径不会被改写。对应 signal 的 protocol 设置同样优先,目前只支持 `http/protobuf`。GenAI 语义固定到 v1.41.0,Resource 中可查看 `cyber.otel.gen_ai.version`。

复杂任务可用正常 `--task-file`,例如审查代码、生成统计程序、执行已有测试并输出报告。`--output <路径>` 保存 canonical AOP JSONL,可与后端记录对照;`--output-format json` 输出最终结果及 usage。JSONL 中 protobuf uint64 是数字字符串,统计只累计顶层 usage,不能再次累计 TurnEnded 的 usage 摘要或 assistant 消息内的工具调用。

默认不导出消息正文。`CYBER_OTEL_CAPTURE_CONTENT=true` 开启最多 4096 字节的输入输出和事件正文。此开关不采集原始 provider frames 或流式 delta;AOP 日志不等于自动采集任意 stdout。

## 查看数据

SigNoz 按 `service.name` 筛选;每个进程中的扩展实例还有独立 `service.instance.id`,可关联同一次执行的三个信号。Traces 中根 `invoke_agent [agent name]` 展开 `chat [model]`、`execute_tool <name>`、命令和进程;Logs 中可见 Session/Turn、operation、usage、outcome 和 trace/span ID。Phoenix 的对应项目使用 AGENT / LLM / TOOL / CHAIN 展示同一 trace。

| 指标 | 用途 |
| --- | --- |
| `cyber.agent.sessions` / `.active` / `cyber.agent.session.duration` | Session 开始/结束转换、活跃数与耗时 |
| `cyber.agent.turns` / `.active` / `cyber.agent.turn.duration` | Turn outcome、活跃数与耗时 |
| `cyber.scope.completed` / `.active` / `.duration` | tool / command / process 和其他机制的状态及耗时 |
| `cyber.model.requests` / `cyber.model.usage.missing` | 逻辑模型调用数(包含自动重试)与缺失 usage 覆盖 |
| `cyber.model.tokens` | input / output / total;总量只筛 `gen_ai.token.type=total` |
| `gen_ai.client.operation.duration` / `gen_ai.client.token.usage` | 模型耗时和 token histogram |
| `cyber.aop.events` / `cyber.otel.events.dropped` | 消费的事实和已观察到的订阅丢弃 |

指标不以 Session/Turn/operation ID 为标签。`cyber.agent.sessions` 是状态转换计数,筛 `cyber.lifecycle.state=started` 才表示启动的 Session 数。累计 input/output/total 不可三者相加。标准模型指标带 operation/provider 和已知 request model;失败 duration 的 `error.type` 与 trace 一致。两个 histogram 都按同一逻辑调用记录,SigNoz 保存为 `.sum` / `.count` / `.bucket` 序列。

任务进度按生命周期和正在执行的工具展示,不估算百分比。tokens 在 AOP usage 到达后更新;当前 StandardLoop 在本轮工具结果收集后发布 usage,长工具会延后该响应计量。模型 active 表示请求投影尚未收尾,不保证此时网络请求仍在进行。startup / recap 等辅助请求没有完整 AOP usage,因此已报告 tokens 不等于整个进程或网关账单。

短任务可以对每条指标序列取累计末值。查询 ClickHouse 时按 fingerprint + metric_name 去重,不能用 Map 属性的物理顺序区分序列:

```sql
SELECT sum(v) AS reported_tokens
FROM (
SELECT argMax(s.value, s.unix_milli) AS v
FROM signoz_metrics.distributed_samples_v4 s
INNER JOIN (
SELECT fingerprint, metric_name, any(attrs) AS attrs
FROM signoz_metrics.distributed_time_series_v4
WHERE resource_attrs['service.instance.id'] = '<本次实例ID>'
GROUP BY fingerprint, metric_name
) t ON s.fingerprint = t.fingerprint AND s.metric_name = t.metric_name
WHERE t.metric_name = 'cyber.model.tokens'
AND t.attrs['gen_ai.token.type'] = 'total'
GROUP BY t.fingerprint
)
```

该查询用于累计末值核对,不表示当前时间窗口内的速率。长期服务可在 SigNoz 创建按时间范围计算的看板和告警。

2026-10-11 本轮验收使用正式 CLI 和真实 `gpt-6-sol` 分析 AOP 历史归档、编写 Python analyzer、执行自检并恢复工具失败:114,242 tokens、10 个逻辑模型调用、10 个工具调用(4 个失败)、158 logs / 29 spans。JSONL、CLI usage、SigNoz trace/log/metric 与 Phoenix 的 29 spans 核对一致,98 个非流式 AOP facts 均有对应日志,任务成功且活跃数最终归零。生成报表另行验收,正确计算历史输入的 138,297 tokens / 22 请求 / 28 工具,这不属于本轮 token 消耗。

本地 AOP 回放另覆盖 delegation、compact/eval、guardrail、JEV、retry、取消和未完成:7 traces / 23 spans / 210 tokens、113 logs、15 类指标 / 99 序列;真实 Harness + HTTP/SSE 夹具覆盖 7 spans / 250 tokens。回放验收与真实模型覆盖分开记录。测试代码、验收夹具、运行脚本、截图、凭据和临时产物保留在本地,未纳入此变更。

## 嵌入式宿主与关闭

在事件流提供者之后、执行者之前安装 `otelext.New(Options{Endpoint: ...})`;需要工具/命令/进程生命周期时安装现有 `observe`。三个 SDK 独立,Resource 一致,不修改全局 provider。MetricInterval 默认 10 秒,Flush / Close 也导出指标。

先结束生产者、Session 与后台工作,再关闭 OTel。SessionEnded 立即收尾所属孤儿 Turn/scope,不提前关闭不同 Session 的后台 Agent。Flush 不强制结束运行中的任务;Close 将剩余 scope/Turn/Session 标记 incomplete 并归零活跃数。Close 超时返回 `ErrCloseIncomplete`,扩展继续清理,后续 Close 可等待完成。导出失败和已观察到的 AOP 订阅丢弃由 Flush/Close 返回;Log SDK 内部丢弃尚未计量。其他事实缺口见设计文档与 [Issue #175](https://github.com/chainreactors/cyber-harness/issues/175)。

```powershell
docker compose -f examples/observability/docker/docker-compose.yaml stop
```

停止服务保留历史数据;再次启动继续使用命名卷。
25 changes: 25 additions & 0 deletions examples/observability/common/SIGNOZ-LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
Copyright (c) 2020-present SigNoz Inc.

Portions of this software are licensed as follows:

* All content that resides under the "ee/" and the "cmd/enterprise/" directory of this repository, if that directory exists, is licensed under the license defined in "ee/LICENSE".
* All third party components incorporated into the SigNoz Software are licensed under the original license provided by the owner of the applicable component.
* Content outside of the above mentioned directories or restrictions above is available under the "MIT Expat" license as defined below.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
75 changes: 75 additions & 0 deletions examples/observability/common/clickhouse/cluster.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
<?xml version="1.0"?>
<clickhouse>
<!-- ZooKeeper is used to store metadata about replicas, when using Replicated tables.
Optional. If you don't use replicated tables, you could omit that.

See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/
-->
<zookeeper>
<node index="1">
<host>zookeeper-1</host>
<port>2181</port>
</node>
<!-- <node index="2">
<host>zookeeper-2</host>
<port>2181</port>
</node>
<node index="3">
<host>zookeeper-3</host>
<port>2181</port>
</node> -->
</zookeeper>

<!-- Configuration of clusters that could be used in Distributed tables.
https://clickhouse.com/docs/en/operations/table_engines/distributed/
-->
<remote_servers>
<cluster>
<!-- Inter-server per-cluster secret for Distributed queries
default: no secret (no authentication will be performed)

If set, then Distributed queries will be validated on shards, so at least:
- such cluster should exist on the shard,
- such cluster should have the same secret.

And also (and which is more important), the initial_user will
be used as current user for the query.

Right now the protocol is pretty simple and it only takes into account:
- cluster name
- query

Also it will be nice if the following will be implemented:
- source hostname (see interserver_http_host), but then it will depends from DNS,
it can use IP address instead, but then the you need to get correct on the initiator node.
- target hostname / ip address (same notes as for source hostname)
- time-based security tokens
-->
<!-- <secret></secret> -->
<shard>
<!-- Optional. Whether to write data to just one of the replicas. Default: false (write data to all replicas). -->
<!-- <internal_replication>false</internal_replication> -->
<!-- Optional. Shard weight when writing data. Default: 1. -->
<!-- <weight>1</weight> -->
<replica>
<host>clickhouse</host>
<port>9000</port>
<!-- Optional. Priority of the replica for load_balancing. Default: 1 (less value has more priority). -->
<!-- <priority>1</priority> -->
</replica>
</shard>
<!-- <shard>
<replica>
<host>clickhouse-2</host>
<port>9000</port>
</replica>
</shard>
<shard>
<replica>
<host>clickhouse-3</host>
<port>9000</port>
</replica>
</shard> -->
</cluster>
</remote_servers>
</clickhouse>
Loading
Loading