777 Commits
Author SHA1 Message Date
Calcium-Ion 0ed497f066 feat(relay): hosted-tool conversion fidelity, reasoning normalization, and billing usage integrity (#7137)
* feat(relaykit): preserve hosted tools across conversions

- add protocol-neutral hosted-tool DTOs, conversion metadata, and loss policies
- bridge citations, grounding metadata, and hosted-tool stream lifecycles
- document the public conversion behavior and channel policy controls

* refactor(relaykit): normalize reasoning and thinking intent

- centralize provider-neutral reasoning intent, effort, and budget mappings
- parse model suffixes at the host entry boundary while preserving provider-owned tails
- keep adaptive Claude thinking and explicit zero-token compatibility consistent

* fix(billing): preserve authoritative usage across relay hops

- carry native BillingUsage sidecars through direct and streamed protocol bridges
- merge partial and terminal usage monotonically with safe fallback settlement
- retain cache metadata, penultimate usage, and per-call Gemini tool surcharges

* feat(relay): bridge Responses with Claude and Gemini protocols

- add direct request, response, and stream converters across supported relay formats
- expose Claude count_tokens and Chat-to-Responses compatibility endpoints
- carry conversion diagnostics through the host while retaining the curated public goldens

* fix(relay): wire relaykit conversions into host channels

- connect handlers, adaptors, and channel settings to the standalone conversion layer
- keep model mapping, pricing identity, retries, and provider-specific suffix behavior aligned
- ignore local audit artifacts and retain focused public regression coverage
2026-09-01 21:53:35 +08:00
Seefs b7017c251b fix(model): do not treat no-op system task state writes as lock loss (#7135) 2026-09-01 20:54:17 +08:00
CaIon 27ff6a8767 fix(model): migrate legacy token key constraints 2026-08-31 11:51:59 +08:00
jimmyleocn 8c8c4153d4 fix(log): preserve quota in usage statistics (#7108)
* fix(log): preserve quota in usage statistics
2026-08-31 11:36:43 +08:00
CaIon 2b6f1dfefb fix(model): drop leftover prefill_groups unique constraints before AutoMigrate 2026-08-30 23:02:58 +08:00
Calcium-Ion 2bf0820f4b Revert "fix(model): drop leftover prefill_groups unique constraints before Au…" (#7101)
This reverts commit 69a41eeadc.
2026-08-30 22:55:17 +08:00
Seefs 69a41eeadc fix(model): drop leftover prefill_groups unique constraints before AutoMigrate (#7100) 2026-08-30 22:51:50 +08:00
CaIon 6eb6f35ed2 fix(model): return string from JSON column Valuers for pg simple protocol
With PrepareStmt disabled, PostgreSQL queries run over pgx's simple
protocol, which encodes every []byte parameter as a bytea hex literal
('\x...'). driver.Valuer implementations returning []byte from
json.Marshal therefore fail json-column writes with SQLSTATE 22P02
(reported on the channels UPDATE path via ChannelInfo).

Reproduced against a live PostgreSQL 16: []byte Valuer into a json
column fails under simple protocol, string succeeds; []byte into a
text column silently stores the hex literal (no such path exists in
the repo today — audited all Valuers, json.RawMessage fields, and raw
SQL call sites).

- ChannelInfo, Properties, TaskPrivateData, JSONValue Value() now
  return string; zero-value nil semantics unchanged. Task.Data
  (bare json.RawMessage) is unaffected — database/sql's default
  converter already passes it as expected.
- Their Scan() counterparts now accept both []byte and string via a
  shared jsonScanBytes helper: SQLite returns string for these columns
  once Value() emits string, and the old []byte-only assertions
  silently zeroed the field (caught by the model test suite).
- Add regression tests locking both contracts: json-column Valuers
  must return string (or nil for zero values), Scanners must accept
  []byte and string.

Verified end-to-end against PostgreSQL 16 with the real model types:
Channel create/update/read-back, Task json fields, PrefillGroup items.
2026-08-30 21:13:21 +08:00
CaIon 66031a09d9 fix(model): disable PostgreSQL prepared statements for pooler compatibility
GORM v1.25.2 closes cached prepared statements asynchronously on any SQL
error and immediately re-Parses the same deterministic name (pgx's
stmt_<sha256>) on the same client connection. Transaction-pooling proxies
(PgBouncer >=1.21 with max_prepared_statements, Neon, Supabase) respond
with FATAL "prepared statement name is already in use" (SQLSTATE 08P01)
and drop the connection. PreferSimpleProtocol only disables pgx's
implicit prepare and never covered GORM's explicit PrepareStmt cache.

- PostgreSQL now runs with PrepareStmt disabled entirely; named prepared
  statements are fundamentally session state and cannot be made safe
  under transaction pooling. Parse/plan cost is noise for this workload.
- Upgrade gorm to v1.25.12 so MySQL/SQLite statement caches (still
  enabled) no longer churn close/re-prepare on ordinary SQL errors;
  v1.25.9+ restricts eviction to driver.ErrBadConn. Deliberately not
  v1.26+, whose LRU eviction has an open use-after-close race (#7831).
- sanitizeDBError now attaches a remediation hint on 08P01/42P05 so
  affected deployments can self-diagnose from the log line.
2026-08-30 20:06:32 +08:00
CaIon 6c22550ea3 feat(task): resolve channel-mapped aliases and case variants for plugin models
Channel model_mapping keys exposed in a channel's model list now act as
first-class aliases for task-plugin models across the whole line:

- Derived alias view (model/task_model_alias.go): built from enabled
  channels' model_mapping, chain-following with cycle detection, declared
  names always win, cross-plugin conflicts dropped. Rebuilt on channel
  cache refresh, registry generation change, and a 60s TTL.
- Request path: PinTaskPluginEndpoint resolves declared-name case folds
  and mapping aliases before endpoint lookup (never rewriting the body
  until the endpoint is claimed), pins with MappedModel, and the decode
  contract accepts alias echoes without loosening model ownership for
  normal pins. Legacy /v1/tasks submit folds case variants the same way.
  Fixes aliases on POST /v1/responses silently falling through to the
  main relay against task channels.
- Mapping order: ModelMappedHelper now runs before the plugin submit
  hook builds and caches the upstream body, so channel model_mapping
  actually reaches the upstream request. Plugins receive the mapped
  name as ctx.upstreamModel in both decode and submit contexts.
- Billing: identity stays the origin name; when the alias has no tiered
  expression, the selected channel's mapping tail expression applies.
  Pricing page and billing-expr smoke tests resolve aliases to the
  owning plugin's usage schema.
- Case folding: ASCII-only fold with exact-match priority; same-plugin
  and cross-plugin fold collisions rejected at registration.
- Plugins: model-keyed rate tables, req_key derivation, and combo
  validation in doubao/kling/jimeng/hailuo/vidu/sunoapi now key on
  ctx.upstreamModel || ctx.model; render/echo paths keep ctx.model.
2026-08-30 19:13:51 +08:00
CaIon b80d633cf5 feat(auth): encrypt password login transport
Closes #6743
2026-08-29 20:11:36 +08:00
Calcium-Ion eb48396d5f feat(task): replace built-in task adaptors with a sandboxed JS plugin system (#7076) 2026-08-29 18:51:57 +08:00
Seefs a073f74b38 refactor: deprecate int32 (#7025)
* refactor: deprecate int32

* fix(db): reject legacy user quota schemas at startup

* fix(quota): enforce wallet bounds and saturating billing conversions

* fix(rate-limit): keep count*duration from wrapping int64

* fix: error message
2026-08-26 20:57:54 +08:00
Seefs 4add708ebe feat: channel test (#6917)
* feat: channel test

* fix: code smell
2026-08-18 18:03:59 +08:00
CaIon 47ba9d2c63 fix(topup): guard wallet quota during recharge 2026-08-14 17:34:57 +08:00
wans10andCaIon 58d4e9bd3b fix(billing): 异步任务退款时同步减少 used_quota (#6795)
* fix(billing): 异步任务退款时同步减少 used_quota
退款时仅恢复了 quota(剩余额度),但未同步减少 used_quota(已用额度),
导致"总额度"(quota + used_quota)随退款次数持续虚增,超出用户实际充值金额。

修复三处退款路径:
- RefundTaskQuota:任务失败完整退款
- RecalculateTaskQuota:差额结算退款分支
- controller/midjourney.go:Midjourney 任务失败退款

新增 model.UpdateUserUsedQuota 公开函数,仅调整 used_quota 不影响 request_count。

* fix(billing): 任务退款时同步扣减渠道 used_quota

* fix(billing): complete async task refund accounting

* style(model): group internal Midjourney fields

---------

Co-authored-by: CaIon <i@caion.me>
2026-08-13 22:06:40 +08:00
CaIon ccd535ef8e fix: harden concurrent quota and status updates 2026-08-11 22:03:47 +08:00
CaIon 50e5377ea5 fix(topup): settle recharge orders atomically 2026-08-11 22:03:47 +08:00
CaIon d7992672a6 fix(oauth): avoid overwriting user state when binding 2026-08-11 22:03:45 +08:00
RedwindA 0cd9dc85e3 Merge commit from fork 2026-08-06 17:53:55 +08:00
Calcium-Ion 0ab0202060 Feat/auto group (#6590)
* feat(token): support custom auto group order

* feat(keys): enhance auto group presentation

* fix(keys): rework Auto flow border and compact inherited order

The Auto group highlight previously tinted the whole control surface
with a gradient and animated only a 1px top sweep, which read as a
background color rather than a flowing border. Replace it with a
border-only effect: an aria-hidden, pointer-events-none overlay whose
conic gradient is masked down to a thin ring hugging the rounded
perimeter, so the highlight travels around all four edges and corners
every 3.2s. The interior stays neutral with a restrained static
primary border and glow; prefers-reduced-motion hides the moving
layer while keeping the static emphasis.

The inherited global Auto order also rendered as spacious two-line
rows with circular sequence markers, wasting drawer space. Render it
as a compact wrapping strip of one-line chips (index, name, ratio
badge) with descriptions kept accessible via title and sr-only text,
scrolling only past a much smaller max height.

Custom add/remove/reorder editing, empty-array inheritance semantics,
and the submit payload are unchanged.

* fix(keys): preserve Auto inheritance and unify effects

* refactor(keys): temporarily disable AutoGroupBadge in api-key-group-cell
2026-08-01 23:19:01 +08:00
Seefs 84834eee85 feat(logs): expose stream status to log owners (#6558) 2026-07-31 13:29:07 +08:00
feitianbubu c3db41407d fix: 慢查询/错误 SQL 日志参数化 (#6493)
* fix: parameterize slow/error SQL logs to avoid leaking credentials

* fix: validate SQL_SLOW_THRESHOLD_MS range

* fix: sanitize database driver error messages in SQL logs

* refactor: sanitize at gorm log writer seam to keep caller attribution
2026-07-27 22:21:31 +08:00
CaIon e99a9bd86f feat: add per-channel HTTP transport controls 2026-07-27 21:41:13 +08:00
CaIon b8bb3f40ac refactor: update import paths to use new types package 2026-07-27 16:45:02 +08:00
Calcium-Ion 86ac0f7745 refactor: extract protocol conversion layer into standalone relaykit module (#6369)
* test(relayconvert): add golden snapshot matrix and relaykit boundary guard

Phase 0 of the relaykit extraction plan: pin byte-level output of every
registered (from,to) request/response/stream conversion route, and
forbid kit-bound packages from growing host-only imports.

* wip(relayconvert): drop gin.Context from converter signatures; add convmeta draft

Phase 1 in progress: relayconvert now takes context.Context; host media
resolver adapts gin.Context back at the service boundary.

* refactor(relayconvert): decouple converters from RelayInfo, gin, and settings

Phase 1 of the relaykit extraction plan:
- converters now depend on convmeta.Meta (implemented by RelayInfo) instead
  of *relaycommon.RelayInfo; ClaudeConvertInfo and the format guesser move
  to convmeta with aliases left behind
- host settings reach converters via a convmeta.Options snapshot built in
  RelayInfo.ConvOptions; no more model_setting/reasoning global reads inside
  the conversion layer
- effort-suffix helpers move to service/relayconvert/reasoning (old package
  forwards); chat-to-responses upgrade policy moves to service (host routing
  logic, not conversion)
- golden conversion matrix unchanged

* test(relayconvert): tighten boundary — kit packages now free of gin/setting imports

* refactor(dto): drop gin and logger dependencies

Phase 2 (part 1): dto.Request.IsStream now takes *http.Request instead of
*gin.Context (Gemini's impl reads query/path off the std request); dto's
three logger calls become common.SysError. Boundary test allowlist is now
empty — kit-bound packages import no gin/setting/logger/model.

* refactor(kit): extract dependency-free kitutil; dto/types/relayconvert stop importing common

Phase 2 of the relaykit extraction plan:
- new service/relayconvert/kitutil holds the pure helpers the kit needs
  (JSON wrappers, pointer/string/uuid/timestamp utils, MaskSensitiveInfo,
  pluggable LogInfo/LogError hooks, Debug flag)
- dto, types, and all relayconvert packages now use kitutil; their only
  remaining internal deps are dto/types/constant
- common keeps every original symbol (MaskSensitiveInfo delegates to
  kitutil) so host code is untouched; main.go routes kit logging into
  common.SysLog/SysError and mirrors DebugEnabled
- golden conversion matrix unchanged

* refactor(kit): move EndpointType/FinishReason to types; OpenRouter dialect via Options

Kit packages (dto/types/relayconvert/reasonmap) no longer import constant:
- EndpointType and finish-reason values live in types; constant re-exports
- the OpenRouter special-case in claude->openai request conversion reads
  Options.OpenRouterDialect, set by the host from the channel type;
  InitChannelMeta invalidates the cached snapshot on channel switch

* refactor: extract relaykit submodule (dto/types/relayconvert/reasonmap)

Phase 3 of the relaykit extraction plan:
- new go module github.com/QuantumNous/new-api/relaykit containing dto
  (minus task family), types, relayconvert (with convmeta/kitutil/reasoning),
  and reasonmap; host consumes it via require + replace, go.work for dev
- task-family dto (task/suno/midjourney/video) stays in the host dto
  package; dual-consumer host files alias it as taskdto
- relaykit builds and tests standalone (GOWORK=off): no host imports,
  no gin, no DB, no settings
- golden conversion matrix unchanged

* build(docker): copy relaykit/go.mod before go mod download

The local-replace submodule's go.mod must exist inside the build context
for the main module graph to resolve.

* fix: address relaykit extraction regressions

* fix: address relaykit review regressions

* docs: document Meta nil receiver contract

* fix(relaykit): fail OpenAI→Claude conversion without max_tokens; reject negative default_max_tokens

The Claude Messages API requires max_tokens (omitting it is a 400
"Field required"), but with a nil Options.Claude.DefaultMaxTokens hook
the converters silently emitted a request the upstream is guaranteed to
reject. Both OpenAI Chat and Responses → Claude conversions now return
sharedclaude.ErrMissingMaxTokens when no path (client value, default
hook, thinking-adapter floor) supplied one. Unreachable in the host,
which always configures the hook.

Host side, claude.default_max_tokens now rejects negative values at the
option API before persisting — they would wrap into huge unsigned values
during conversion. Zero stays allowed: the current API treats
max_tokens: 0 as cache pre-warming.

* fix: make Gemini safety settings read path race-free
2026-07-27 15:56:21 +08:00
CaIon bc14c18f60 refactor: update task refund logic and remove legacy handling 2026-07-26 20:47:52 +08:00
CaIon 2d23cdf291 feat: configurable tool pricing, Sub2API channel, and alpha search billing
Add admin-configurable tool-call prices with cross-provider surcharge
settlement, Sub2API channel support, /v1/alpha/search relay, and usage-log
surcharge UI.
2026-07-26 20:05:15 +08:00
feitianbubuandCaIon e0d5156115 fix: prevent duplicate suno task refunds via cas status update (#6074)
* fix: prevent duplicate suno task refunds via cas status update

* fix: reconcile failed task refunds

---------

Co-authored-by: CaIon <i@caion.me>
2026-07-20 22:03:13 +08:00
yyhhyyyyyy e13d4033e5 fix(channel): improve proxy client compatibility and cache lifecycle (#6157)
* fix(channel): improve proxy client compatibility and cache lifecycle

* test(controller): use non-fatal assertions for channel tests
2026-07-20 18:11:22 +08:00
Calcium-Ion 31d70fca39 refactor(auth): replace dashboard sessions with stateless tokens and session control (#6329)
* refactor(auth): replace dashboard sessions with stateless tokens

* feat(auth): harden session issuance and distributed enforcement

* fix(proxy): preserve trusted proxy compatibility defaults

* refactor: address dashboard auth review feedback

* refactor: remove classic frontend and flatten web app
2026-07-20 16:48:43 +08:00
SeefsandCaIon a6cf42c0f1 feat: support upstream model fetch for advanced custom channels (#5971)
* feat: support upstream model fetch for advanced custom channels

* fix: add advanced custom routes as separate groups

* fix: select advanced custom route entry before adding

---------

Co-authored-by: CaIon <i@caion.me>
2026-07-18 13:39:53 +08:00
DawnMoon1542 506e41c116 fix: add server-side sorting to user list, prevent client-side sort on paged data (#6194)
* fix: add server-side sorting to user list, prevent client-side sort on paged data

The user management table applied client-side sorting to the current
page slice while pagination was handled server-side, causing ID-asc
views to show pages out of order (e.g. 23-42, 3-22, 1-2).

Backend: add sort_by/sort_order query params to GetAllUsers and
SearchUsers with a column whitelist for safe ORDER BY generation.
Frontend: pass sorting state to the API and reset to page 1 on sort
change. Generic useDataTable hook now disables client-side sorted row
model and sort UI for tables with manual pagination but no server-side
sort handler.

* fix: add id tie-breaker to non-unique sort columns, remove side effect from state updater

Append secondary ORDER BY id DESC when sorting by non-unique columns
(quota, created_at, etc.) to prevent row duplication/skipping across
OFFSET pages.

Move onPaginationChange out of setSorting updater to avoid side effects
inside a pure function (React Strict Mode double-invocation safety).
2026-07-17 19:36:44 +08:00
fuxdevandduanxufu a63364d156 fix: infer MiniMax vendor for MiniMax models (#6164)
* fix: infer MiniMax vendor for MiniMax models

* Delete model/pricing_default_test.go

---------

Co-authored-by: duanxufu <duanxufu@rededa.com>
2026-07-14 20:32:48 +08:00
Seefs b6b97a66e3 fix: purge authentication data on hard user deletion (#6168)
* fix: purge authentication data on hard user deletion

* fix: fail closed when 2FA status lookup fails

* fix: reject stale Telegram login callbacks

* fix(twofa): prevent concurrent backup code and lockout bypasses

* fix(auth): harden user deletion and Telegram verification
2026-07-14 14:25:54 +08:00
CaIon 7a2b9d86e8 feat: enhance model search functionality with status and sync filters 2026-07-11 22:36:09 +08:00
Calcium-Ion c36418c863 feat: enhance text protocol conversion and advanced custom routing (#5825)
* refactor: consolidate relay protocol converters

* refactor relayconvert text converters

* feat: refine relay converters and advanced custom routing

* refactor: enhance logging and add thought signature handling for Gemini requests

* refactor: enhance channel cache and pricing endpoint handling for advanced custom models

* feat: preserve billing usage semantics

* feat: add protocol-aware billing usage

* Delete useless files

* chore: update action versions in workflow files

* chore: update Docker action versions in workflow files

* fix: harden billing usage settlement and hot-path route matching

- estimate Gemini completion tokens locally when billable usageMetadata is
  prompt-only but output content was received (e.g. client aborts the stream
  before the final chunk), and rebuild the attached billing_usage as estimated
  so settlement does not bill zero output tokens
- guard NewClaudeMessagesBillingUsage against all-zero ClaudeUsage, matching
  the OpenAI/Gemini constructors, so a zero billing_usage cannot override a
  non-zero top-level usage during settlement
- cache compiled advanced-custom route model regexes; they run on the request
  hot path and were recompiled per request
- move the effectiveBillingUsage remap to PostTextConsumeQuota only, and
  document that calculateTextQuotaSummary expects remapped usage
- document the updatePricingLock -> channelSyncLock lock ordering that
  InitChannelCache/CacheUpdateChannel rely on, and the aux-struct pitfall in
  GeminiChatResponse.UnmarshalJSON
2026-07-11 20:44:12 +08:00
Seefs 4e570389dd fix: use GORM v2 row locking for subscription resets (#6057)
* fix: use GORM v2 row locking for subscription resets

* docs: codify GORM row locking rules
2026-07-10 10:21:00 +08:00
Seefs a72e5082e9 feat(system-info): add stale instance cleanup actions (#5953) 2026-07-07 21:22:49 +08:00
Seefs 9b93d61b7f feat(subscription): add admin quota reset actions (#5952)
* feat(subscription): add admin quota reset actions

* fix(subscription): keep quota reset in plan row actions

* refactor(subscription): move user subscription actions into menu
2026-07-07 12:47:41 +08:00
CaIon 70ea899e37 fix(model): centralize row locking in transactional flows 2026-07-07 12:40:09 +08:00
CaIon bae799ccb1 fix(billing): surface quota saturation events for admin auditing
Thread int32 saturation clamps from tiered settlement and video task
recompute into the consume/task logs under admin_info, so oversized or
malformed billing inputs stay auditable. Clamp negative audio duration
before token conversion and gate the saturation UI markers on admin.
2026-07-07 12:20:07 +08:00
feitianbubu 043720f9be fix: 任务差额结算后 quota 和阿里视频时长优化 (#5923)
* fix: apply default ali video duration when value is non-positive

* fix: persist task quota after async settlement
2026-07-06 11:49:24 +08:00
CaIon 5fc35e28a2 fix(user): harden account email and password handling
- normalize emails (trim + lowercase) and enforce uniqueness across
  registration, OAuth auto-registration, and email binding
- serialize concurrent writers on the same normalized email within a
  transaction to avoid duplicate accounts
- resolve password reset to a single matching account and reject
  ambiguous or absent matches
- require an existing password before self-service password change and
  reject login for accounts without a usable password
2026-07-05 13:15:41 +08:00
CaIon 12603a7765 fix(redemption): add status filtering and cleanup action 2026-07-04 16:15:47 +08:00
Calcium-Ion dfc0d6324b Merge commit from fork
* Harden user setting cache updates

* Fix user update test isolation
2026-07-03 15:25:33 +08:00
CaIon bfddc5fea0 fix: omit access_token from user queries 2026-07-03 15:15:16 +08:00
CaIon 8874d1929f Make quota logging synchronous and delay startup log 2026-07-02 21:58:41 +08:00
CaIon df44a75d53 fix: adapt ClickHouse log LIKE filters 2026-06-27 17:03:25 +08:00
Calcium-Ion 4aee5f7d5a feat: better admin permissions (#5755)
* feat: add casbin admin permissions

* feat: improve audit logging to associate logs with actual operators and target users

* feat: enhance admin permissions and UI interactions for sensitive actions

* Refactor authz RBAC and tighten channel permissions

* Split channel authz field policy

* Address channel authz review findings
2026-06-27 17:01:59 +08:00