HYPERFRAMES_RUN_ID (trimmed, 128-char cap) attaches as run_id to the
generic cli_command / cli_command_result events, absent when unset, so
an orchestrator setting it per design element can group a verify loop's
invocations in analytics. check additionally emits one check_report
event per invocation (including lint-short-circuited and failing runs):
gate booleans, per-class error/warning counts, launch/seek/contrast
phase timings, sample counts, ok and exit code. Timings stay internal;
no command output changes.
* feat(cli): associate signed-in HeyGen account with telemetry
Sign-in telemetry currently attributes everything to the anonymous
install id, so the sign-in funnel can be counted but a completed sign-in
can't be tied to the account it produced. This associates the two.
- On a completed sign-in, emit a PostHog `$identify` alias whose
`$anon_distinct_id` is the install's anonymousId, so events recorded
before sign-in stitch to the same person, and tag `auth_login_completed`
with the account identity (the pre-plumbed `distinctId`).
- `/v3/users/me` exposes no opaque user_id, so the identity key is the
account email, falling back to username (single `identityKey` helper).
- Both no-op under the `telemetry disable` opt-out and only fire after
the user chooses to sign in.
Privacy disclosure updated in lockstep, since this is the first PII the
CLI attaches: the first-run telemetry notice and the telemetry section
of docs/packages/cli.mdx now state that signing in links your account
email to your usage.
Tests: identifyUser payload + no-op, completion attribution incl.
username fallback and no-identity-on-reject/empty. Verified end-to-end
against the built CLI: pre-auth events anonymous, $identify carries
$anon_distinct_id, completion carries the account email.
* docs(cli): disclose the username identity fallback
Review gating item: identityKey is `email ?? username`, but the
first-run notice and cli.mdx said only "email", so an emailless
account's username would reach PostHog undisclosed. `/v3/users/me`
treats email as optional (pickString), so the fallback is live code,
not dead — disclose it rather than assert an unverifiable email
guarantee. Both surfaces now say "email, or username if the account
has no email".
Also soften the identityKey comment: it implied username is "less
identifying", but HeyGen usernames are often email-shaped, so the note
now states username is a fallback, not a privacy win.
* feat(cli): emit sign-in lifecycle telemetry
The CLI tracks command and render lifecycles but emits nothing for
`auth login`, so sign-in outcomes are invisible on the observability
dashboards — a completed sign-in, an abandoned browser flow, and a
rejected key all look identical (absent). This leaves a blind spot in
the same funnel the render events already cover.
Add three events mirroring the existing `trackX` pattern:
- auth_login_started (method: oauth | api_key)
- auth_login_completed (method)
- auth_login_failed (method, reason)
`reason` is a fixed low-cardinality enum (flow_error / no_credential /
rejected / invalid_input). No token, key, identity, email, or free text
is ever attached — consistent with the existing anonymous telemetry and
the `telemetry disable` opt-out. Wired into both the OAuth and
--api-key paths in `auth login`, with unit coverage for the new events.
* fix(cli): close sign-in telemetry funnel dropout gaps
Follow-up so `started` reconciles to `completed + failed` on the common
abandonment paths, which the first cut missed:
- Interactive prompt cancel (Ctrl-C) now surfaces as a throw that the
single catch in the api-key path records as `aborted`, instead of a
bare exit with no event.
- A stdin read that times out in non-TTY `--api-key` mode now records
`aborted` before the error propagates, rather than exiting silently.
- OAuth split: a timed-out browser callback (user closed the tab) is
tagged `flow_timeout`, separated from real `flow_error` (IdP/network),
since the walk-away timeout is the dominant non-error dropout.
Also pre-plumb an optional `distinctId` on the three trackers, mirroring
trackRenderComplete/trackRenderError. Unused today; it lets a later
identity-level attribution be a one-line callsite change rather than a
signature sweep. Coverage added for the new reasons and forwarding.
Closes the observability gaps on the figma integration:
- withFigmaErrors takes a command label (figma:asset|tokens|component) and
reports the failure inline before its process.exit — the top-level
trackCommandFailures wrapper never sees self-exiting commands, so typed
codes (NO_TOKEN, BAD_TOKEN, FORBIDDEN, RATE_LIMITED) were invisible.
FigmaClientError codes surface as the error name for dashboarding the
first-run funnel (NO_TOKEN -> later success = onboarding conversion).
- new figma_import event per import: phase, duration, reused (dedup
effectiveness), tokens variables-vs-styles mode + entry count
(Enterprise gating rate), unresolved-binding + rasterized-node counts
(fidelity degradation). No fileKeys, node ids, names, or descriptions.
- /figma skill fires the events beacon (figma-motion / figma-shaders /
figma-storyboard) for the MCP phases that never touch the CLI.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The P1-3 aspect/alpha/HDR pre-flight (#1843) aborts an incompatible render before any browser/ffmpeg work, but that "save" was invisible on dashboard 1783183 — indistinguishable from a deep failure or a user giving up.
checkRenderResolutionPreflight now returns { message, kind } (kind = the existing low-cardinality OutputResolutionIssueKind), and the render command emits render_preflight_rejected { kind } before exiting. No parsers change — the helper already carried kind. trackRenderPreflightRejected is typed to the union so the metric can't carry free text.
Tests: preflight tests assert kind for all five kinds; an events test locks the emit.
Further follow-up (still log-only): encoder-frame-0-exit counter and a P1-4 doctor cli_env_check event.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The standalone `feedback` command runs separately from `render`, so it
has no access to the prior render's elapsed time, yet it always passed
renderDurationMs: 0 to the feedback analytics event. Since that path is
the one used in practice (the auto-prompt returns early for agent and
non-interactive runtimes), nearly every feedback event recorded a render
duration of exactly 0, which is misleading rather than absent.
Make renderDurationMs optional and only include render_duration_ms in the
event when a real value is supplied. The standalone command no longer
passes a duration; the auto-prompt path still forwards the real elapsed
time.
Studio-triggered renders emit render_complete / render_error from the CLI
preview-server process, which stamps every event with the install's
anonymousId (client.ts drainQueueToPayload). The browser, meanwhile, fires
studio_session_start / studio_render_start under its own getAnonymousId(). So
the render outcome and the render start never share a person_id — verified in
data: of 15,125 users who started a studio render in 30d, ZERO have any
render_complete under any source, and the 898 studio-tagged completers are
disjoint server UUIDs. The studio render funnel — the product's core value
moment and strongest retention signal — is therefore unmeasurable.
Thread the browser's telemetry id through to the render-outcome events:
- client.ts: trackEvent takes an optional distinctId; drainQueueToPayload uses
`event.distinctId ?? config.anonymousId`. CLI renders unchanged.
- events.ts: trackRenderComplete/trackRenderError forward an optional distinctId.
- studioRenderTelemetry.ts: emitStudioRender* pass opts.distinctId through.
- core studio-api (types.ts + routes/render.ts): the render route reads
`telemetryDistinctId` from the request body (validated string) and passes it
to the adapter's startRender, which already forwards opts to the emitters.
- studio (useRenderQueue.ts): include getAnonymousId() as telemetryDistinctId
in the render POST — the same id studio_* events already use.
Result: studio render_complete/error now carry the browser user's id and join
studio_session_start / studio_render_start. Older clients that don't send the
field fall back to anonymousId (no regression). No new tracking surface — it's
the existing anonymous studio id.
Tests: per-event override forwarding (events), studio render distinctId
threading + older-client fallback (studioRenderTelemetry), and route body →
adapter forwarding incl. non-string rejection (core render route).
Observability showed `browser` (~75% fail, ~1.3k users/day) and `info` (~60%
fail) failing at high rates with no captured reason — only
`cli_command_result success=false`. citty's `runMain` catches a command's
thrown error and `process.exit(1)`s without re-throwing, so a thrown failure
never reached the existing `cli_error` telemetry (which only fired from the
uncaughtException / unhandledRejection handlers).
Wrap every command's `run()` at the dispatch boundary (cli.ts) so a thrown
failure reports its reason via `cli_error` (kind=command_error) before being
re-thrown unchanged — citty's print + exit-1 behavior is preserved. This
de-blinds every throw-style command at once: `browser ensure` (Chrome
download), `tts`, `inspect`, `render`, etc.
Paths that bypass the wrapper are handled inline:
- `browser` self-exits (`path` download failure, unknown subcommand) — report
inline; the ARM64 `ensure` branch previously swallowed a failed install and
returned success, now reports and exits 1.
- `resolveProject()` self-exits on InvalidProjectError (the dominant `info`
failure — run outside a project) — report inline before exit.
Hardening:
- PII: `trackCliError` now redacts error_message + stack_trace via
redactTelemetryString (matching render_* events) — CLI errors and stacks
carry absolute install paths / cache dirs / user args.
- Race: the wrapper awaits an on-demand telemetry import before re-throwing, so
a command that fails before the lazy telemetry import settles still reports
(a telemetry failure is swallowed and never masks the real error).
Pure helpers in utils/command-failure-tracking.ts with unit tests for the
throw / success / no-run / onFailure-rejection cases, the reporter wiring, and
trackCliError redaction. CommandDef<any> mirrors citty's SubCommandsDef.
Known scope: commands that print + `process.exit(1)` on their own validation
paths (tts/validate/lint argument errors) remain wrapper-blind — follow-up.
Transcribe failures are recorded as `cli_command_result success=false` but
without a reason: the command catches its own error, prints it, and
`process.exit(1)` — the message never reaches telemetry. `cli_error` was only
emitted from the uncaughtException / unhandledRejection handlers, so
self-handled command failures were invisible. That makes a high failure rate
countable but not debuggable.
Add `trackCommandFailure(command, err)` — a thin wrapper over the existing
`trackCliError({ kind: "command_error" })` that normalizes an unknown reason to
name/message/stack. It enqueues synchronously, so the process `exit` handler's
flushSync ships it alongside `cli_command_result`. Respects the telemetry
opt-out (gated in trackEvent) and reuses the existing PII redaction.
Wire it into all three of transcribe's failure exits (file-not-found,
empty-transcript import, and the transcribe() catch — ffmpeg / whisper-binary /
model-download errors). Now each failure carries its reason, so we can see how
much of the failure rate is environment vs user input.
The helper is generic — the same one-liner can be dropped into other commands'
failure paths, or centralized at the runMain boundary, as a follow-up.
Miguel Angel Simon Sierra