mirror of
https://github.com/simonlin1212/TradingAgents-astock.git
synced 2026-10-30 09:29:56 +00:00
feat: 决策绩效统计 + 修复中英混排评级被静默判成 Hold (#61)
新增 tradingagents performance
决策与真实收益一直都在记忆日志里(store_decision → 下次分析同一只票时自动拉
行情回填 raw/alpha),缺的只是汇总——此前全仓连一个胜率都算不出来。零 LLM 调用。
- 整体胜率 / alpha 胜率 / 平均与中位收益,按评级、按标的分组
- 评级区分度检验:五档从 Buy 到 Sell 平均 alpha 是否单调递减;仅在五档都有
样本时给结论
- --json 输出
三条不许骗人的设计:收益解析不出来的记录跳过而非计 0%;样本 <20 条报告自己标
注是噪音;结尾写明这不是回测(窗口重叠、无仓位管理、未计成本、可能有选择偏差)。
刻意不算夏普/年化——离散且重叠的决策点年化没有意义。
指标口径参考 Vibe-Trading(MIT)的 agent/backtest/metrics.py。
修复:`最终评级:Buy` 被静默判成 Hold
中文标签 + 英文评级词的混排躲过了原有每一条规则(英文标签规则要 "rating";
中文标签规则只认中文词;裸英文词扫描按空白切分,全角冒号 strip 不掉)。
决策评级被悄悄改写且报告里看不出来,是 #78/#80 的未修变种。这个洞会直接污染
上面的绩效统计,是用真实数据验证报告时撞出来的。
测试:新增 20 例,全量回归 277 passed
This commit is contained in:
@@ -6,6 +6,51 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
Breaking changes within the 0.x line are called out explicitly.
|
||||
|
||||
## [0.5.2] — 2026-08-06
|
||||
|
||||
新增决策绩效统计,并修掉一个会污染它的评级解析漏洞。
|
||||
|
||||
### 新增:`tradingagents performance`([#61](https://github.com/simonlin1212/TradingAgents-astock/issues/61))
|
||||
|
||||
决策和真实收益其实一直都在存——`TradingMemoryLog` 每次分析落一条决策,下次分析
|
||||
同一只股票时自动拉真实行情回填收益与 alpha。缺的只是汇总:在此之前整个仓库连一个
|
||||
胜率都算不出来。
|
||||
|
||||
- **零 LLM 调用**,只读已落盘的结果。
|
||||
- 整体胜率 / **alpha 胜率**(对沪深 300)/ 平均与中位收益 / 最好最差。
|
||||
- 按评级、按标的分组。
|
||||
- **评级区分度检验**:五档评级从 Buy 到 Sell,平均 alpha 是否单调递减。不单调就说明
|
||||
评级没有实际区分能力——这比单看胜率有意义得多。仅在五档都有样本时才给结论。
|
||||
- `--json` 输出便于二次加工。
|
||||
|
||||
三条刻意的"不许骗人"设计:
|
||||
|
||||
1. **收益解析不出来的记录跳过,不当成 0%** —— 后者会把统计悄悄拉向中性。
|
||||
2. **样本少于 20 条时报告自己标注「这些比率基本是噪音」**,不等用户去数。
|
||||
3. **报告结尾写明这不是回测、不是策略业绩**:持有窗口重叠、无仓位管理、未计交易
|
||||
成本,样本可能有选择偏差。
|
||||
|
||||
指标口径参考 [Vibe-Trading](https://github.com/HKUDS/Vibe-Trading)(MIT)的
|
||||
`agent/backtest/metrics.py`:胜率与盈亏按已完成交易计。**刻意不算夏普、不做年化**
|
||||
——用离散且重叠的决策点去年化只会得到一个看着专业但没有意义的数字。
|
||||
|
||||
### 修复:中文标签 + 英文评级词被静默判成 Hold
|
||||
|
||||
`最终评级:Buy` 这种混排(`output_language` 为中文、模型却保留英文评级词)躲过了
|
||||
原有的每一条规则:英文标签规则要求出现 `rating`;中文标签规则只认中文评级词;裸
|
||||
英文词扫描按空白切分,`最终评级:Buy` 是**一个** token,`strip("*:.,")` 又剥不掉
|
||||
全角冒号。结果静默落到默认值 Hold —— 决策评级被悄悄改写,报告里完全看不出来,
|
||||
属于 #78 / #80 同类问题的未修变种。
|
||||
|
||||
这个漏洞会直接污染上面的绩效统计(评级分组全错),是在用真实数据验证报告时撞出来的。
|
||||
|
||||
### 测试
|
||||
|
||||
新增 `tests/test_performance.py`(16 例)与 4 例评级解析用例(含"中文评级词仍然
|
||||
优先"的反向边界)。全量回归 277 passed。
|
||||
|
||||
---
|
||||
|
||||
## [0.5.1] — 2026-08-06
|
||||
|
||||
修复未来函数:在历史日期上跑分析时,数据层会把"今天"的数据当成分析日当天的事实。
|
||||
|
||||
@@ -243,10 +243,32 @@ print(decision)
|
||||
### 4. CLI 方式
|
||||
|
||||
```bash
|
||||
tradingagents # 交互式 CLI
|
||||
tradingagents --help # 查看所有选项
|
||||
tradingagents # 交互式 CLI
|
||||
tradingagents analyze # 同上(默认命令)
|
||||
tradingagents performance # 决策绩效统计(见下)
|
||||
tradingagents --help # 查看所有选项
|
||||
```
|
||||
|
||||
### 5. 决策绩效统计(v0.5.2 新增)
|
||||
|
||||
想知道**这套流程过往的判断准不准**,跑:
|
||||
|
||||
```bash
|
||||
tradingagents performance # 人读的报告
|
||||
tradingagents performance --json # 机器读的 JSON
|
||||
```
|
||||
|
||||
数据来自记忆日志:每次分析会落一条决策,下次分析同一只股票时自动拉真实行情回填收益与 alpha(对沪深 300)。**统计本身零 LLM 调用**,只读已经落盘的结果。
|
||||
|
||||
输出包含整体胜率 / alpha 胜率 / 平均收益、按评级分组、按标的分组,以及一项**评级区分度检验**——五档评级从 Buy 到 Sell,平均 alpha 是否真的单调递减。评级不单调,说明这套流程的评级没有实际区分能力,这比单看胜率有意义得多。
|
||||
|
||||
几点务必注意:
|
||||
|
||||
- **这不是回测,也不是策略业绩。** 每条记录是「某天做出的判断在固定持有期后的表现」:持有窗口互相重叠、没有仓位管理、未计交易成本与冲击成本,样本还可能有选择偏差。
|
||||
- **A 股 beta 很强**,跟着大盘涨不代表判断对,所以 **alpha 胜率比胜率更值得看**。
|
||||
- **样本少于 20 条时报告会自己标注「这些比率基本是噪音」**,不要拿三五条记录下结论。
|
||||
- 收益解析不出来的记录会被**跳过**而不是当成 0%——后者会把统计悄悄拉向中性。
|
||||
|
||||
---
|
||||
|
||||
## Web UI
|
||||
|
||||
+24
-2
@@ -230,10 +230,32 @@ print(decision)
|
||||
### 4. CLI Mode
|
||||
|
||||
```bash
|
||||
tradingagents # Interactive CLI
|
||||
tradingagents --help # Show all options
|
||||
tradingagents # Interactive CLI
|
||||
tradingagents analyze # Same as above (default command)
|
||||
tradingagents performance # Decision performance report (see below)
|
||||
tradingagents --help # Show all options
|
||||
```
|
||||
|
||||
### 5. Decision performance report (added in v0.5.2)
|
||||
|
||||
To find out **how well the pipeline's past calls actually held up**:
|
||||
|
||||
```bash
|
||||
tradingagents performance # Human-readable report
|
||||
tradingagents performance --json # Machine-readable JSON
|
||||
```
|
||||
|
||||
The data comes from the memory log: every analysis records a decision, and the next analysis of the same ticker resolves it by fetching real prices and filling in the return and the alpha (against CSI 300). **The report itself makes zero LLM calls** — it only reads results already on disk.
|
||||
|
||||
It covers overall win rate / alpha win rate / average return, a breakdown by rating and by ticker, and a **rating-discrimination check**: across the five tiers from Buy to Sell, does average alpha actually decrease monotonically? A non-monotonic result means the ratings carry no real discriminating power, which tells you far more than the win rate alone.
|
||||
|
||||
Important caveats:
|
||||
|
||||
- **This is not a backtest, and not strategy performance.** Each record is "how one judgement made on one day looked after a fixed holding window": the windows overlap, there is no position sizing, no transaction or impact costs, and the sample may be selection-biased.
|
||||
- **A-share beta is strong** — rising with the index is not the same as being right, so **the alpha win rate matters more than the raw win rate**.
|
||||
- **Below 20 resolved records the report says so itself** ("these ratios are mostly noise"). Do not draw conclusions from a handful of entries.
|
||||
- Records whose return cannot be parsed are **skipped, not counted as 0%** — counting them would quietly drag every statistic toward neutral.
|
||||
|
||||
---
|
||||
## Web UI
|
||||
|
||||
|
||||
+23
@@ -1280,5 +1280,28 @@ def analyze(
|
||||
run_analysis(checkpoint=checkpoint)
|
||||
|
||||
|
||||
@app.command()
|
||||
def performance(
|
||||
json_out: bool = typer.Option(
|
||||
False, "--json", help="Print raw JSON instead of the formatted report."
|
||||
),
|
||||
):
|
||||
"""决策绩效统计:这套流程过往的判断准不准(零 LLM 调用,只读已结算的记录)。
|
||||
|
||||
数据来自记忆日志:每次分析会落一条决策,下次分析同一只股票时自动拉真实行情
|
||||
回填收益与 alpha。**这不是回测**,详见报告末尾的说明。
|
||||
"""
|
||||
import json as _json
|
||||
|
||||
from tradingagents.agents.utils.memory import TradingMemoryLog
|
||||
from tradingagents.performance import format_report, summarize
|
||||
|
||||
summary = summarize(TradingMemoryLog(DEFAULT_CONFIG).load_entries())
|
||||
if json_out:
|
||||
console.print_json(_json.dumps(summary, ensure_ascii=False))
|
||||
return
|
||||
console.print(Markdown(format_report(summary)))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
app()
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "tradingagents-astock"
|
||||
version = "0.5.1"
|
||||
version = "0.5.2"
|
||||
description = "A股多Agent投研框架 — 基于 TradingAgents 深度特化"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
"""决策绩效统计(#61)。
|
||||
|
||||
决策与真实收益一直都在记忆日志里,缺的只是汇总。这些用例除了验算术,更重要的是
|
||||
锁死三条"不许骗人"的规则:
|
||||
|
||||
1. 解析不出收益的记录必须**跳过**,不能当成 0%——那会把统计悄悄拉向中性;
|
||||
2. 样本太小要**自己说出来**,不能让人把 3 条记录的胜率当结论;
|
||||
3. 报告必须写明**这不是回测**,且未结算时不能显示一堆 0%。
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from tradingagents.performance import (
|
||||
MIN_MEANINGFUL_SAMPLE,
|
||||
format_report,
|
||||
load_records,
|
||||
summarize,
|
||||
)
|
||||
|
||||
|
||||
def entry(date, ticker, rating, raw=None, alpha=None, holding="5", pending=False):
|
||||
return {
|
||||
"date": date, "ticker": ticker, "rating": rating,
|
||||
"pending": pending, "raw": raw, "alpha": alpha, "holding": holding,
|
||||
"decision": "", "reflection": "",
|
||||
}
|
||||
|
||||
|
||||
def many(n, rating="Buy", raw="+5.0%", alpha="+2.0%", ticker="600519"):
|
||||
return [entry(f"2026-01-{i+1:02d}", ticker, rating, raw, alpha) for i in range(n)]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 解析
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_pending_entries_are_excluded():
|
||||
records = load_records([
|
||||
entry("2026-01-01", "600519", "Buy", pending=True),
|
||||
entry("2026-01-02", "600519", "Buy", "+3.0%", "+1.0%"),
|
||||
])
|
||||
assert len(records) == 1
|
||||
assert records[0].date == "2026-01-02"
|
||||
|
||||
|
||||
def test_unparseable_return_is_skipped_not_zeroed():
|
||||
"""把解析失败当成 0% 会把统计悄悄拉向中性——必须跳过。"""
|
||||
records = load_records([
|
||||
entry("2026-01-01", "600519", "Buy", "n/a", "n/a"),
|
||||
entry("2026-01-02", "600519", "Buy", "", ""),
|
||||
entry("2026-01-03", "600519", "Buy", "+3.0%", "+1.0%"),
|
||||
])
|
||||
assert len(records) == 1
|
||||
|
||||
|
||||
def test_percent_parsing():
|
||||
r = load_records([entry("2026-01-01", "600519", "Buy", "+5.0%", "-1.5%")])[0]
|
||||
assert r.raw_return == pytest.approx(0.05)
|
||||
assert r.alpha_return == pytest.approx(-0.015)
|
||||
|
||||
|
||||
def test_missing_alpha_is_none_not_zero():
|
||||
"""alpha 缺失和 alpha 为 0 是两回事,不能混。"""
|
||||
r = load_records([entry("2026-01-01", "600519", "Buy", "+5.0%", None)])[0]
|
||||
assert r.alpha_return is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 统计
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_win_rate_and_averages():
|
||||
s = summarize([
|
||||
entry("2026-01-01", "600519", "Buy", "+10.0%", "+5.0%"),
|
||||
entry("2026-01-02", "600519", "Buy", "-4.0%", "-2.0%"),
|
||||
entry("2026-01-03", "600519", "Buy", "+6.0%", "+3.0%"),
|
||||
])
|
||||
o = s["overall"]
|
||||
assert o["count"] == 3
|
||||
assert o["win_rate"] == pytest.approx(2 / 3)
|
||||
assert o["avg_return"] == pytest.approx(0.04)
|
||||
assert o["best"] == pytest.approx(0.10)
|
||||
assert o["worst"] == pytest.approx(-0.04)
|
||||
|
||||
|
||||
def test_alpha_win_rate_ignores_records_without_alpha():
|
||||
s = summarize([
|
||||
entry("2026-01-01", "600519", "Buy", "+10.0%", "+5.0%"),
|
||||
entry("2026-01-02", "600519", "Buy", "+1.0%", None),
|
||||
])
|
||||
# 只有 1 条有 alpha,且为正
|
||||
assert s["overall"]["alpha_win_rate"] == pytest.approx(1.0)
|
||||
|
||||
|
||||
def test_grouping_by_rating_and_ticker():
|
||||
s = summarize([
|
||||
entry("2026-01-01", "600519", "Buy", "+10.0%", "+5.0%"),
|
||||
entry("2026-01-02", "000001", "Sell", "-4.0%", "-2.0%"),
|
||||
])
|
||||
assert s["by_rating"]["Buy"]["count"] == 1
|
||||
assert s["by_rating"]["Sell"]["count"] == 1
|
||||
assert set(s["by_ticker"]) == {"600519", "000001"}
|
||||
|
||||
|
||||
def test_empty_rating_groups_are_dropped():
|
||||
"""没有样本的评级不该出现在报表里凑行数。"""
|
||||
s = summarize([entry("2026-01-01", "600519", "Buy", "+1.0%", "+1.0%")])
|
||||
assert set(s["by_rating"]) == {"Buy"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 评级区分度
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_monotonic_needs_every_tier():
|
||||
"""只有两三档有样本时不给结论——那样判断单调性没有意义。"""
|
||||
s = summarize(many(5, "Buy") + many(5, "Sell", "-5.0%", "-3.0%"))
|
||||
assert s["rating_monotonic"] is None
|
||||
|
||||
|
||||
def test_monotonic_true_when_ratings_rank_correctly():
|
||||
entries = []
|
||||
for rating, alpha in [
|
||||
("Buy", "+5.0%"), ("Overweight", "+3.0%"), ("Hold", "+1.0%"),
|
||||
("Underweight", "-1.0%"), ("Sell", "-3.0%"),
|
||||
]:
|
||||
entries += many(3, rating, "+1.0%", alpha)
|
||||
assert summarize(entries)["rating_monotonic"] is True
|
||||
|
||||
|
||||
def test_monotonic_false_when_ratings_are_inverted():
|
||||
entries = []
|
||||
for rating, alpha in [
|
||||
("Buy", "-3.0%"), ("Overweight", "+3.0%"), ("Hold", "+1.0%"),
|
||||
("Underweight", "-1.0%"), ("Sell", "+5.0%"),
|
||||
]:
|
||||
entries += many(3, rating, "+1.0%", alpha)
|
||||
assert summarize(entries)["rating_monotonic"] is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 不许骗人
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_small_sample_is_flagged():
|
||||
s = summarize(many(3))
|
||||
assert s["sample_is_meaningful"] is False
|
||||
assert "噪音" in format_report(s)
|
||||
|
||||
|
||||
def test_large_sample_is_not_flagged():
|
||||
s = summarize(many(MIN_MEANINGFUL_SAMPLE))
|
||||
assert s["sample_is_meaningful"] is True
|
||||
assert "噪音" not in format_report(s)
|
||||
|
||||
|
||||
def test_report_without_resolved_entries_says_so():
|
||||
"""一条都没结算时要说明怎么才会结算,而不是显示一堆 0%。"""
|
||||
report = format_report(summarize([
|
||||
entry("2026-01-01", "600519", "Buy", pending=True),
|
||||
]))
|
||||
assert "暂无可统计" in report
|
||||
assert "0.0%" not in report
|
||||
|
||||
|
||||
def test_report_states_it_is_not_a_backtest():
|
||||
"""必须写明这不是回测、不是策略业绩,否则很容易被当成收益承诺。"""
|
||||
report = format_report(summarize(many(MIN_MEANINGFUL_SAMPLE)))
|
||||
assert "不是回测" in report
|
||||
assert "不构成任何投资建议" in report
|
||||
|
||||
|
||||
def test_report_highlights_alpha_over_raw_win_rate():
|
||||
"""A 股 beta 强,光看胜率会高估判断力——报告要点破这一点。"""
|
||||
report = format_report(summarize(many(MIN_MEANINGFUL_SAMPLE)))
|
||||
assert "alpha 胜率" in report
|
||||
@@ -92,6 +92,27 @@ class TestParseRatingChinese:
|
||||
for cn, en in cases.items():
|
||||
assert parse_rating(f"最终评级:{cn}\n理由若干。") == en
|
||||
|
||||
def test_cn_label_with_english_rating_word(self):
|
||||
"""中文标签 + 英文评级词(最终评级:Buy)。
|
||||
|
||||
output_language 设为中文、模型却保留英文评级词时就是这个形状,而它躲过了
|
||||
原来每一条规则:英文标签规则要求出现 "rating";中文标签规则只认中文评级
|
||||
词;裸英文词扫描按空白切分,"最终评级:Buy" 是一个 token,strip 又剥不掉
|
||||
全角冒号 —— 结果静默落到默认 Hold,决策评级被悄悄改写。
|
||||
"""
|
||||
assert parse_rating("最终评级:Buy") == "Buy"
|
||||
assert parse_rating("最终评级:Sell\n理由若干。") == "Sell"
|
||||
assert parse_rating("投资建议:Overweight") == "Overweight"
|
||||
assert parse_rating("评级 - buy") == "Buy"
|
||||
|
||||
def test_cn_label_with_english_word_all_tiers(self):
|
||||
for r in RATINGS_5_TIER:
|
||||
assert parse_rating(f"最终评级:{r}") == r
|
||||
|
||||
def test_cn_label_still_prefers_chinese_term(self):
|
||||
"""中文评级词仍然优先——新增的混排规则不能抢在它前面。"""
|
||||
assert parse_rating("最终评级:卖出(英文可写作 Buy)") == "Sell"
|
||||
|
||||
def test_cn_label_variants(self):
|
||||
assert parse_rating("投资建议: **增持**\n分批建仓。") == "Overweight"
|
||||
assert parse_rating("评级:清仓") == "Sell"
|
||||
|
||||
@@ -46,10 +46,21 @@ _CN_RATING_MAP = {
|
||||
# same position (regex alternation is leftmost, first-listed among equals).
|
||||
_CN_ALT = "|".join(sorted(_CN_RATING_MAP, key=len, reverse=True))
|
||||
|
||||
# A labelled Chinese rating, e.g. "最终评级:卖出" / "投资建议: **增持**".
|
||||
_CN_LABEL_RE = re.compile(
|
||||
_CN_LABEL_PREFIX = (
|
||||
r"(?:最终评级|评级|投资评级|评级结论|最终投资建议|投资建议|操作建议|"
|
||||
r"推荐评级|建议|推荐)\s*[::\-]\s*\*{0,2}\s*(" + _CN_ALT + r")"
|
||||
r"推荐评级|建议|推荐)\s*[::\-]\s*\*{0,2}\s*"
|
||||
)
|
||||
|
||||
# A labelled Chinese rating, e.g. "最终评级:卖出" / "投资建议: **增持**".
|
||||
_CN_LABEL_RE = re.compile(_CN_LABEL_PREFIX + r"(" + _CN_ALT + r")")
|
||||
|
||||
# 中文标签后面跟**英文**评级词,例如 "最终评级:Buy"。output_language 设为中文、
|
||||
# 但模型保留了英文评级词时就是这个形状——而它躲过了上面每一条规则:
|
||||
# 英文标签规则要求出现 "rating";中文标签规则只认中文评级词;裸英文词扫描按
|
||||
# 空白切分,"最终评级:Buy" 是**一个** token,`strip("*:.,")` 又剥不掉全角冒号。
|
||||
# 结果是静默落到默认值 Hold —— 决策评级被悄悄改写,报告里完全看不出来。
|
||||
_CN_LABEL_EN_RE = re.compile(
|
||||
_CN_LABEL_PREFIX + r"(" + "|".join(RATINGS_5_TIER) + r")", re.IGNORECASE
|
||||
)
|
||||
# Bare Chinese rating term anywhere (last-resort fallback).
|
||||
_CN_TERM_RE = re.compile(_CN_ALT)
|
||||
@@ -60,7 +71,8 @@ def parse_rating(text: str, default: str = "Hold") -> str:
|
||||
|
||||
Pass order (first hit wins; explicit labels always beat bare words):
|
||||
1. English ``Rating: X`` label (tolerant of markdown bold).
|
||||
2. Chinese rating label, e.g. ``最终评级:卖出`` / ``投资建议: 增持``.
|
||||
2. Chinese rating label, e.g. ``最终评级:卖出`` / ``投资建议: 增持``;
|
||||
也接受中文标签后跟英文评级词的混排(``最终评级:Buy``)。
|
||||
3. First bare English 5-tier word found anywhere.
|
||||
4. First bare Chinese rating term found anywhere (longest match wins).
|
||||
|
||||
@@ -77,6 +89,11 @@ def parse_rating(text: str, default: str = "Hold") -> str:
|
||||
if m:
|
||||
return _CN_RATING_MAP[m.group(1)]
|
||||
|
||||
# 2b. Chinese label + English rating word (最终评级:Buy)
|
||||
m = _CN_LABEL_EN_RE.search(text)
|
||||
if m:
|
||||
return m.group(1).capitalize()
|
||||
|
||||
# 3. Bare English rating word
|
||||
for line in text.splitlines():
|
||||
for word in line.lower().split():
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
"""决策绩效统计 —— 这套 agent 到底准不准(#61)。
|
||||
|
||||
决策和真实收益其实一直都在存:`TradingMemoryLog` 每次分析落一条
|
||||
`[日期 | 代码 | 评级 | pending]`,下次跑同一只票时 `_resolve_pending_entries()`
|
||||
会拉真实行情回填 raw 收益与 alpha(对沪深 300)。缺的只是**汇总**——在此之前
|
||||
整个仓库连一个胜率都算不出来。
|
||||
|
||||
零 LLM 成本:只读已经落盘的结果,不重跑任何分析。
|
||||
|
||||
⚠️ 这不是回测。每条记录是"某天做的判断在固定持有期后的表现",不是一个可交易的
|
||||
组合:持有窗口互相重叠、没有仓位管理、没有交易成本、幸存下来的样本也可能有偏。
|
||||
所以这里**不算夏普、不做年化**——用离散、重叠的决策点去annualize 只会得到一个
|
||||
看着专业但没有意义的数字。
|
||||
|
||||
指标口径参考 Vibe-Trading(MIT,https://github.com/HKUDS/Vibe-Trading)的
|
||||
`agent/backtest/metrics.py`:胜率与盈亏统计按已完成交易计,不按逐 bar 收益计。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import statistics
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from tradingagents.agents.utils.rating import RATINGS_5_TIER
|
||||
|
||||
# 样本少于这个数时,任何比率都只是噪音。宁可明说,也不要让人把 3 条记录里的
|
||||
# "66.7% 胜率"当回事——这是这类统计最常见的误读。
|
||||
MIN_MEANINGFUL_SAMPLE = 20
|
||||
|
||||
|
||||
def _parse_pct(value: Optional[str]) -> Optional[float]:
|
||||
"""把记忆日志里的 "+5.0%" 解析回 0.05。解析不了返回 None(不当成 0)。"""
|
||||
if not value:
|
||||
return None
|
||||
text = str(value).strip().replace("%", "")
|
||||
try:
|
||||
return float(text) / 100.0
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
@dataclass
|
||||
class DecisionRecord:
|
||||
"""一条已结算的决策。"""
|
||||
|
||||
date: str
|
||||
ticker: str
|
||||
rating: str
|
||||
raw_return: float
|
||||
alpha_return: Optional[float]
|
||||
holding_days: Optional[int]
|
||||
|
||||
|
||||
@dataclass
|
||||
class GroupStats:
|
||||
"""一组决策的表现。"""
|
||||
|
||||
count: int = 0
|
||||
win_rate: Optional[float] = None
|
||||
alpha_win_rate: Optional[float] = None
|
||||
avg_return: Optional[float] = None
|
||||
median_return: Optional[float] = None
|
||||
avg_alpha: Optional[float] = None
|
||||
best: Optional[float] = None
|
||||
worst: Optional[float] = None
|
||||
|
||||
def as_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"count": self.count,
|
||||
"win_rate": self.win_rate,
|
||||
"alpha_win_rate": self.alpha_win_rate,
|
||||
"avg_return": self.avg_return,
|
||||
"median_return": self.median_return,
|
||||
"avg_alpha": self.avg_alpha,
|
||||
"best": self.best,
|
||||
"worst": self.worst,
|
||||
}
|
||||
|
||||
|
||||
def _stats(records: List[DecisionRecord]) -> GroupStats:
|
||||
if not records:
|
||||
return GroupStats()
|
||||
raws = [r.raw_return for r in records]
|
||||
alphas = [r.alpha_return for r in records if r.alpha_return is not None]
|
||||
return GroupStats(
|
||||
count=len(records),
|
||||
win_rate=sum(1 for v in raws if v > 0) / len(raws),
|
||||
# alpha 胜率单独算:A 股 beta 很强,跟着大盘涨不代表判断对。
|
||||
alpha_win_rate=(
|
||||
sum(1 for v in alphas if v > 0) / len(alphas) if alphas else None
|
||||
),
|
||||
avg_return=statistics.fmean(raws),
|
||||
median_return=statistics.median(raws),
|
||||
avg_alpha=statistics.fmean(alphas) if alphas else None,
|
||||
best=max(raws),
|
||||
worst=min(raws),
|
||||
)
|
||||
|
||||
|
||||
def load_records(entries: List[dict]) -> List[DecisionRecord]:
|
||||
"""从 TradingMemoryLog 解析出的条目里取已结算的那些。
|
||||
|
||||
未结算(pending)和 raw 收益解析不出来的都跳过——把解析失败当成 0% 收益会
|
||||
悄悄把统计拉向中性,正是这类报表最容易骗人的地方。
|
||||
"""
|
||||
records: List[DecisionRecord] = []
|
||||
for e in entries:
|
||||
if e.get("pending"):
|
||||
continue
|
||||
raw = _parse_pct(e.get("raw"))
|
||||
if raw is None:
|
||||
continue
|
||||
holding = e.get("holding")
|
||||
try:
|
||||
holding_days = int(str(holding).strip()) if holding else None
|
||||
except ValueError:
|
||||
holding_days = None
|
||||
records.append(
|
||||
DecisionRecord(
|
||||
date=e.get("date", ""),
|
||||
ticker=e.get("ticker", ""),
|
||||
rating=e.get("rating", "") or "Unknown",
|
||||
raw_return=raw,
|
||||
alpha_return=_parse_pct(e.get("alpha")),
|
||||
holding_days=holding_days,
|
||||
)
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def rating_monotonicity(by_rating: Dict[str, GroupStats]) -> Optional[bool]:
|
||||
"""更看好的评级,实际收益是不是真的更高?
|
||||
|
||||
这是"这套 agent 准不准"最直接的检验:五档评级从 Buy 到 Sell,平均 alpha
|
||||
应当单调递减。只在**每一档都有样本**时才给结论,否则返回 None——拿两三档
|
||||
去判断单调性没有意义。
|
||||
"""
|
||||
ordered = [by_rating.get(r) for r in RATINGS_5_TIER]
|
||||
if any(g is None or g.count == 0 or g.avg_alpha is None for g in ordered):
|
||||
return None
|
||||
values = [g.avg_alpha for g in ordered]
|
||||
return all(a >= b for a, b in zip(values, values[1:]))
|
||||
|
||||
|
||||
def summarize(entries: List[dict]) -> Dict[str, Any]:
|
||||
"""汇总记忆日志里的全部决策。
|
||||
|
||||
Args:
|
||||
entries: `TradingMemoryLog.load_entries()` 的返回值。
|
||||
|
||||
Returns:
|
||||
统计字典。`resolved` 为 0 时其余字段为空,调用方据此提示"还没有可统计的
|
||||
结果",而不是显示一堆 0%。
|
||||
"""
|
||||
total = len(entries)
|
||||
pending = sum(1 for e in entries if e.get("pending"))
|
||||
records = load_records(entries)
|
||||
|
||||
by_rating: Dict[str, GroupStats] = {}
|
||||
for rating in RATINGS_5_TIER:
|
||||
by_rating[rating] = _stats([r for r in records if r.rating == rating])
|
||||
others = [r for r in records if r.rating not in RATINGS_5_TIER]
|
||||
if others:
|
||||
by_rating["Unknown"] = _stats(others)
|
||||
|
||||
by_ticker: Dict[str, GroupStats] = {}
|
||||
for ticker in sorted({r.ticker for r in records}):
|
||||
by_ticker[ticker] = _stats([r for r in records if r.ticker == ticker])
|
||||
|
||||
overall = _stats(records)
|
||||
holdings = [r.holding_days for r in records if r.holding_days]
|
||||
|
||||
return {
|
||||
"total_decisions": total,
|
||||
"pending": pending,
|
||||
"resolved": len(records),
|
||||
# 样本太小的时候必须自己说出来,不能等用户自己去数
|
||||
"sample_is_meaningful": len(records) >= MIN_MEANINGFUL_SAMPLE,
|
||||
"min_meaningful_sample": MIN_MEANINGFUL_SAMPLE,
|
||||
"avg_holding_days": statistics.fmean(holdings) if holdings else None,
|
||||
"overall": overall.as_dict(),
|
||||
"by_rating": {k: v.as_dict() for k, v in by_rating.items() if v.count},
|
||||
"by_ticker": {k: v.as_dict() for k, v in by_ticker.items()},
|
||||
"rating_monotonic": rating_monotonicity(by_rating),
|
||||
"date_range": (
|
||||
(min(r.date for r in records), max(r.date for r in records))
|
||||
if records else None
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _pct(value: Optional[float], digits: int = 1) -> str:
|
||||
return "—" if value is None else f"{value:+.{digits}%}"
|
||||
|
||||
|
||||
def _rate(value: Optional[float]) -> str:
|
||||
return "—" if value is None else f"{value:.1%}"
|
||||
|
||||
|
||||
def format_report(summary: Dict[str, Any]) -> str:
|
||||
"""把统计结果排成给人看的文本。"""
|
||||
if summary["resolved"] == 0:
|
||||
return (
|
||||
f"暂无可统计的决策结果。\n"
|
||||
f"共 {summary['total_decisions']} 条决策记录,其中 {summary['pending']} 条"
|
||||
f"尚未结算。\n"
|
||||
f"结算发生在下一次分析同一只股票时(自动拉取真实行情回填收益),"
|
||||
f"或等持有期结束后重新跑一次该股票。"
|
||||
)
|
||||
|
||||
lines: List[str] = ["# 决策绩效统计", ""]
|
||||
if summary["date_range"]:
|
||||
lines.append(f"区间:{summary['date_range'][0]} → {summary['date_range'][1]}")
|
||||
lines.append(
|
||||
f"决策 {summary['total_decisions']} 条 · 已结算 {summary['resolved']} 条 · "
|
||||
f"待结算 {summary['pending']} 条"
|
||||
)
|
||||
if summary["avg_holding_days"]:
|
||||
lines.append(f"平均持有 {summary['avg_holding_days']:.1f} 个交易日")
|
||||
|
||||
if not summary["sample_is_meaningful"]:
|
||||
lines += [
|
||||
"",
|
||||
f"⚠️ 样本仅 {summary['resolved']} 条,少于 "
|
||||
f"{summary['min_meaningful_sample']} 条时下面的比率基本是噪音,"
|
||||
f"**不要据此判断这套流程有效或无效**。",
|
||||
]
|
||||
|
||||
o = summary["overall"]
|
||||
lines += [
|
||||
"",
|
||||
"## 整体",
|
||||
f"- 胜率(绝对收益为正):{_rate(o['win_rate'])}",
|
||||
f"- alpha 胜率(跑赢沪深300):{_rate(o['alpha_win_rate'])}",
|
||||
f"- 平均收益:{_pct(o['avg_return'])} 中位数:{_pct(o['median_return'])}",
|
||||
f"- 平均 alpha:{_pct(o['avg_alpha'])}",
|
||||
f"- 最好:{_pct(o['best'])} 最差:{_pct(o['worst'])}",
|
||||
"",
|
||||
"> A 股 beta 很强,跟着大盘涨不等于判断对——**alpha 胜率比胜率更能说明问题**。",
|
||||
]
|
||||
|
||||
if summary["by_rating"]:
|
||||
lines += ["", "## 按评级", "", "| 评级 | 条数 | 胜率 | alpha胜率 | 平均收益 | 平均alpha |",
|
||||
"|---|---|---|---|---|---|"]
|
||||
for rating, g in summary["by_rating"].items():
|
||||
lines.append(
|
||||
f"| {rating} | {g['count']} | {_rate(g['win_rate'])} | "
|
||||
f"{_rate(g['alpha_win_rate'])} | {_pct(g['avg_return'])} | "
|
||||
f"{_pct(g['avg_alpha'])} |"
|
||||
)
|
||||
mono = summary["rating_monotonic"]
|
||||
if mono is True:
|
||||
lines += ["", "✅ 评级越看好,平均 alpha 越高——评级带有区分度。"]
|
||||
elif mono is False:
|
||||
lines += ["", "⚠️ 评级与实际 alpha **不单调**:更看好的评级并没有带来更高的超额收益。"]
|
||||
else:
|
||||
lines += ["", "(五档评级尚未各自积累到样本,暂不判断评级是否有区分度。)"]
|
||||
|
||||
if summary["by_ticker"]:
|
||||
lines += ["", "## 按标的", "", "| 代码 | 条数 | 胜率 | 平均收益 | 平均alpha |",
|
||||
"|---|---|---|---|---|"]
|
||||
for ticker, g in summary["by_ticker"].items():
|
||||
lines.append(
|
||||
f"| {ticker} | {g['count']} | {_rate(g['win_rate'])} | "
|
||||
f"{_pct(g['avg_return'])} | {_pct(g['avg_alpha'])} |"
|
||||
)
|
||||
|
||||
lines += [
|
||||
"",
|
||||
"---",
|
||||
"⚠️ **这不是回测,也不是策略业绩。** 每条记录是「某天做出的判断在固定持有期后的"
|
||||
"表现」,持有窗口互相重叠、没有仓位管理、未计交易成本与冲击成本,样本也可能有"
|
||||
"选择偏差。它只用来观察这套分析流程自身的一致性,不构成任何投资建议。",
|
||||
]
|
||||
return "\n".join(lines)
|
||||
Reference in New Issue
Block a user