fix(relay): bound the wait for upstream response headers (fixes unbounded heap growth → OOM) (#6949)

* fix(relay): bound the wait for upstream response headers (fixes unbounded heap growth)

The relay transport sets a dial timeout, a TLS handshake timeout and an expect-continue
timeout, but nothing bounds how long it waits for the upstream *response headers* after
the request has been written. An upstream that accepts the connection and then never
answers -- without sending FIN/RST, which is what happens when a NAT/firewall silently
drops the flow or the provider hangs -- parks the goroutine in
net/http.(*persistConn).roundTrip forever.

That goroutine keeps the whole request alive, which in practice means three copies of the
request body stay reachable for the lifetime of the process: the raw bytes from
io.ReadAll in CreateBodyStorageFromReader, the decoded messages held as json.RawMessage,
and the re-marshalled upstream body from common.Marshal. BodyStorageCleanup cannot help
here: it runs after c.Next() returns, and for these requests c.Next() never returns.

Measured on v1.0.0-rc.23 in production (see #6947 for the full evidence):

  - 23 goroutines stuck in persistConn.roundTrip on a single 40h-old instance,
    blocked between 353 and 1894 minutes (5.9h to 31.5h)
  - 96.9% of the live heap, sampled after a forced GC, attributable to those three
    body copies (HeapAlloc 892 MiB surviving three GC cycles; HeapObjects dropping
    30x while bytes dropped only 25%)
  - the live floor grows with uptime: 33.7 MiB at 0.1h, 89.2 at 13.8h, 510.0 at 40.1h,
    955.2 at 146.8h, OOMKilled at 172.9h -- same image, same config, same load

Doubling the memory limit and adding GOMEMLIMIT only moved the OOM from 132h to 172.9h.

RELAY_TIMEOUT (http.Client.Timeout) cannot be used for this: it covers the whole response
read and would cut legitimate long streaming calls, which is why it defaults to 0.
ResponseHeaderTimeout only bounds the wait for the headers; streaming after they arrive is
unaffected.

The default is deliberately generous. Non-streaming upstreams usually send the response
headers only once generation has finished, so the value has to leave room for a long
completion. 1800s is 12x shorter than the shortest hang observed here while leaving
several times the headroom a normal non-streaming request needs; 0 restores the previous
unbounded behaviour.

The assignment goes next to the other transport.* lines rather than inside the else
branch: newRelayHTTPTransport() normally takes the http.DefaultTransport.Clone() path,
and DefaultTransport does not set ResponseHeaderTimeout either.

This repo already sets ResponseHeaderTimeout on its other outbound transports
(controller/model_sync.go, controller/ratio_sync.go); the relay path appears to have
been missed.

Refs #6947. Likely also the root cause of #6731, which reported the same symptom
(production OOM on /v1/responses after ~64h) but was closed for template reasons.

* review: clamp overflowing timeout values and switch the test to testify

Addresses the two CodeRabbit findings on this PR.

Overflow (common/init.go:113): a RELAY_RESPONSE_HEADER_TIMEOUT beyond ~9.2e9 seconds
overflows time.Duration and can wrap into a *tiny positive* timeout, which would cut
every relay request instead of only the stuck ones. The value is now clamped before the
conversion, with regression tests for both the negative and the overflowing input.

I did not add fail-on-startup validation for negative values, for two reasons: the
existing `if seconds > 0` guard already treats them as "disabled", and the neighbouring
env-driven timeouts in this file are less strict still -- RelayIdleConnTimeout is
converted with no guard at all. Failing startup on a bad value would be a behaviour
change out of step with the rest of the file; happy to add it if you'd prefer that
direction repo-wide.

Test style: switched to testify (require.Equal / require.Zero / require.Positive), which
is what every other test under service/ uses.

go build, go vet and go test ./common/... ./service/... pass.
(`go build ./...` fails on the `web/dist` embed both with and without this change -- the
frontend bundle is not checked in.)

.env.example

@@ -62,6 +62,10 @@
 # RELAY_TIMEOUT=0
 # Relay HTTP 客户端空闲连接超时时间，单位秒，默认跟随 Go 标准库，设置为0表示不限制
 # RELAY_IDLE_CONN_TIMEOUT=90
+# 等待上游返回响应头的超时时间，单位秒，默认 1800，设置为 0 表示不限制。
+# 仅约束「等待响应头」这一段；响应头返回之后的流式传输不受影响。
+# 注意：非流式请求通常要等上游生成完毕才会返回响应头，因此该值需留足余量。
+# RELAY_RESPONSE_HEADER_TIMEOUT=1800
 # 流模式无响应超时时间，单位秒，如果出现空补全可以尝试改为更大值
 # STREAMING_TIMEOUT=300
 

README.md

@@ -327,6 +327,7 @@ docker run --name new-api -d --restart always \
 | `SQL_DSN` | Database connection string | - |
 | `REDIS_CONN_STRING` | Redis connection string | - |
 | `RELAY_IDLE_CONN_TIMEOUT` | Idle keep-alive timeout for relay HTTP clients, seconds. Defaults to Go standard library behavior; set `0` to disable | `90` |
+| `RELAY_RESPONSE_HEADER_TIMEOUT` | How long the relay waits for upstream **response headers**, seconds; set `0` to disable. Only bounds the header wait -- streaming after the headers arrive is unaffected. Note that non-streaming upstreams usually send headers only once generation finishes, so leave headroom | `1800` |
 | `STREAMING_TIMEOUT` | Streaming timeout (seconds) | `300` |
 | `STREAM_SCANNER_MAX_BUFFER_MB` | Max per-line buffer (MB) for the stream scanner; increase when upstream sends huge image/base64 payloads | `64` |
 | `MAX_REQUEST_BODY_MB` | Max request body size (MB, counted **after decompression**; prevents huge requests/zip bombs from exhausting memory). Exceeding it returns `413` | `32` |

common/constants.go

@@ -163,6 +163,16 @@ var BatchUpdateInterval int
 var RelayTimeout int // unit is second
 
 var RelayIdleConnTimeout int // unit is second
+
+// RelayResponseHeaderTimeout limits how long the relay transport waits for the
+// upstream response headers after the request has been fully written.
+// 0 disables it (previous behaviour: wait forever).
+//
+// Note this is NOT the same as RelayTimeout (http.Client.Timeout), which covers
+// the whole response read and therefore breaks legitimate long streaming calls.
+// ResponseHeaderTimeout only bounds the wait for the response headers; once the
+// headers arrive, streaming is unaffected.
+var RelayResponseHeaderTimeout int // unit is second
 var RelayMaxIdleConns int
 var RelayMaxIdleConnsPerHost int
 

common/init.go

@@ -111,6 +111,7 @@ func InitEnv() {
 	BatchUpdateInterval = GetEnvOrDefault("BATCH_UPDATE_INTERVAL", 5)
 	RelayTimeout = GetEnvOrDefault("RELAY_TIMEOUT", 0)
 	RelayIdleConnTimeout = GetEnvOrDefault("RELAY_IDLE_CONN_TIMEOUT", 90)
+	RelayResponseHeaderTimeout = GetEnvOrDefault("RELAY_RESPONSE_HEADER_TIMEOUT", 1800)
 	RelayMaxIdleConns = GetEnvOrDefault("RELAY_MAX_IDLE_CONNS", 500)
 	RelayMaxIdleConnsPerHost = GetEnvOrDefault("RELAY_MAX_IDLE_CONNS_PER_HOST", 100)
 

service/http_client.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"crypto/tls"
 	"fmt"
+	"math"
 	"net"
 	"net/http"
 	"net/url"
@@ -71,6 +72,10 @@ func ValidateSSRFProtectedFetchURL(urlStr string) error {
 	return validateURLWithCurrentFetchSetting(urlStr, true)
 }
 
+// maxTimeoutSeconds is the largest number of seconds that still converts to a
+// time.Duration without overflowing (~292 years).
+const maxTimeoutSeconds = int(math.MaxInt64 / int64(time.Second))
+
 func newRelayHTTPTransport() *http.Transport {
 	var transport *http.Transport
 	if defaultTransport, ok := http.DefaultTransport.(*http.Transport); ok && defaultTransport != nil {
@@ -91,6 +96,24 @@ func newRelayHTTPTransport() *http.Transport {
 	transport.MaxIdleConns = common.RelayMaxIdleConns
 	transport.MaxIdleConnsPerHost = common.RelayMaxIdleConnsPerHost
 	transport.IdleConnTimeout = time.Duration(common.RelayIdleConnTimeout) * time.Second
+	// Bound the wait for upstream response headers. Without it, an upstream that
+	// accepts the connection but never responds (and never sends FIN/RST) parks the
+	// goroutine forever, and every buffer that request owns -- the raw body read by
+	// io.ReadAll, the decoded messages, and the re-marshalled upstream body -- stays
+	// reachable for the lifetime of the process.
+	//
+	// This only covers the wait for the headers; streaming after the headers arrive
+	// is not affected. Set RELAY_RESPONSE_HEADER_TIMEOUT=0 to restore the old
+	// unbounded behaviour.
+	if seconds := common.RelayResponseHeaderTimeout; seconds > 0 {
+		// Clamp before converting: seconds beyond maxTimeoutSeconds overflow
+		// time.Duration and can wrap into a tiny positive timeout, which would cut
+		// every relay request instead of only the stuck ones.
+		if seconds > maxTimeoutSeconds {
+			seconds = maxTimeoutSeconds
+		}
+		transport.ResponseHeaderTimeout = time.Duration(seconds) * time.Second
+	}
 	transport.ForceAttemptHTTP2 = true
 	if common.TLSInsecureSkipVerify {
 		transport.TLSClientConfig = common.InsecureTLSConfig