Files
hyperframes/packages/cli/src/telemetry/events.ts
T
Miguel Ángel f822200fb8 feat(telemetry): measure which lint rules fire, cost, and fail to converge (#3367)
* feat(telemetry): measure which lint rules fire, cost, and fail to converge

Lint rule changes are currently argued from anecdote. This adds the three
measurements needed to argue them from data.

`lint_report`, once per `hyperframes lint` or `hyperframes check`:
- `code_counts` / `codes` — which rules actually fire, and how often
- `rule_group_ms` — milliseconds per rule-source module (core, gsap, media, ...)
- `slowest_rule` / `slowest_rule_ms` — slowest single rule as `<group>#<index>`
- `rule_count` — how many rules this build ran

`lint_rule_streak`, once per finding that survives an edit to its file:
- `edits` — how many edits the finding survived
- `cleared` — whether it eventually went away

The streak event is the one that matters. A lint pass costs about 5ms, so
per-rule CPU is not what makes the authoring loop slow; a rule an agent cannot
satisfy is, because every failed attempt costs a full edit-and-relint cycle. A
single run cannot see that, so `lint_rule_streak` reconstructs it across runs:
high `edits` with `cleared: false` is a rule nobody can fix, and the
`cleared: true` distribution is the baseline to judge it against.

An iteration is counted only when the file's content digest CHANGED and the
finding is still there. Re-linting an untouched project is not an attempt,
which is what stops `check` (which lints on every invocation) from inflating
the numbers.

Rule identity is the source module plus an index within it. Naming all 86
rules would make the timings prettier but it is a refactor this measurement
does not need: the group locates the file, and the index locates the rule.

Version, agent runtime, CI flag, and invocation id are already attached to
every event by `trackEvent`, so lint pain can be split by CLI version and by
which agent produced it without adding anything here.

Privacy: only rule codes, counts, and timings are sent. Streak state lives in
~/.hyperframes/lint-streaks.json alongside config.json (so `rm -rf
~/.hyperframes` is still a full reset) and stores digests only — no file
paths, no project names, no composition source. Nothing is written and nothing
is emitted when telemetry is off. Entries expire after 14 days and are capped
at 500 files.

`EventProperties` gains string arrays and numeric maps. `codes` and
`code_counts` are inherently a set and a histogram; flattening them into
dynamic top-level keys would make them unqueryable. PostHog stores both
natively.

`trackLintRun` is the single call site shared by `lint` and `check`, and it
swallows every error — telemetry must never turn a green lint red.

* feat(telemetry): emit per-group rule counts so slowest_rule stays comparable

Review catch on #3367: `slowest_rule` is the one positional key in either
event. It is `<group>#<index>`, so adding or removing a rule renumbers every
later slot in that group and the same string means different rules in two
builds. #3366 does exactly that to 34 of 81 surviving slots, and `rule_count`
alone says only THAT the ruleset moved, not which groups.

`rule_group_counts` carries the per-group sizes alongside it, so a consumer
comparing two builds can tell which groups' indices still mean the same thing
without anyone having to remember which release dropped rules. `codes`,
`code_counts` and `rule_group_ms` are keyed by name and were never affected.

Also corrects the rule count in the RULE_GROUPS comment: 86, not ~60, as
LINT_RULE_COUNT in the same file computes.
2026-08-20 18:26:14 -04:00

895 lines
37 KiB
TypeScript

import { redactTelemetryString, type OutputResolutionIssueKind } from "@hyperframes/core";
import type { SubTimelineWaitOutcome } from "@hyperframes/engine";
import { FEEDBACK_RATING_SCALE } from "../utils/feedbackRating.js";
import { flush, shouldTrack, trackEvent } from "./client.js";
import { readConfig } from "./config.js";
import { getPowerState } from "./system.js";
// Power state is volatile (a laptop docks/undocks mid-session), so it is
// sampled per render event rather than cached with SystemMeta. Attached to
// render_complete AND render_error: the DE fleet is macOS laptops whose
// power management shifts render perf ~1.8x with no other telemetry signal,
// and perf/soak analysis needs to segment by it (see getPowerState).
//
// shouldTrack() is checked HERE, not just inside trackEvent: this helper is
// spread into the properties object at the CALL SITE, so it runs before
// trackEvent's own `if (!shouldTrack()) return` guard. Without this an
// opted-out install would still pay two blocking `pmset` subprocess spawns
// per render for an event that is then discarded (review finding).
// shouldTrack() memoizes, so this costs nothing on the tracked path.
function powerStateFields(): { on_battery?: boolean; low_power_mode?: boolean } {
if (!shouldTrack()) return {};
const power = getPowerState();
return {
on_battery: power.on_battery ?? undefined,
low_power_mode: power.low_power_mode ?? undefined,
};
}
// run_id is attached only when the orchestrator set HYPERFRAMES_RUN_ID — an
// absent property, never null/"" (PostHog treats those as real values).
function runIdField(runId: string | undefined): { run_id?: string } {
return runId !== undefined ? { run_id: runId } : {};
}
export interface RenderObservabilityTelemetryPayload {
/** Worst sub-composition timeline wait outcome across sessions. */
subTimelineWait?: SubTimelineWaitOutcome;
observabilityRenderJobId?: string;
observabilityCompositionHash?: string;
observabilityEventCount?: number;
observabilityLastPhase?: string;
observabilityLastStatus?: string;
observabilityFailedPhase?: string;
browserDiagnosticCount?: number;
browserDiagnosticErrors?: number;
browserDiagnosticPageErrors?: number;
browserDiagnosticRequestFailed?: number;
browserDiagnosticHttpErrors?: number;
browserDiagnosticNavigationStarts?: number;
browserDiagnosticNavigationFailures?: number;
browserDiagnosticConsoleErrors?: number;
browserDiagnosticConsoleWarnings?: number;
captureMode?: string;
captureForceScreenshot?: boolean;
captureWorkerCount?: number;
captureUseStreamingEncode?: boolean;
captureUseLayeredComposite?: boolean;
captureUsePageSideCompositing?: boolean;
captureHasHdrContent?: boolean;
captureBrowserGpuMode?: string;
captureProtocolTimeoutMs?: number;
capturePageNavigationTimeoutMs?: number;
capturePlayerReadyTimeoutMs?: number;
captureTransientRetries?: number;
captureMemoryExhaustionDetected?: boolean;
// Mirror of the DE inversion/router state on `RenderCaptureObservability` —
// sourced from the live-mutated capture object rather than `perfSummary`,
// so a hard failure (crash, OOM, timeout) that never reaches perfSummary
// construction still reports which DE experiment cohort it was in. Mapped
// to the SAME `de_*` event keys `trackRenderComplete` sets explicitly from
// `perfSummary.drawElement`; the caller must spread this payload FIRST so
// the more authoritative perfSummary value wins when both are present.
captureDeWorkerInversion?: string;
captureDePreInversionWorkers?: number;
captureCompositionElementCount?: number;
captureCompositionElementCountSource?: string;
captureDeShortBand?: string;
captureDeParallelRouter?: string;
captureDeGpuRenderer?: string;
captureDePreRouterWorkers?: number;
captureDeSelfVerifyFallback?: boolean;
captureDeFallbackReason?: string;
captureDeFallbackFailedDb?: number;
captureDeFallbackFrameIndex?: number;
captureDeFallbackThresholdDb?: number;
/** Non-DE parallel-streaming router outcome ("screenshot" | "beginframe" —
* routed; "eligible_off" — would route but the kill switch is off). */
captureParallelStream?: string;
observabilityExtractVideoCount?: number;
observabilityExtractedVideoCount?: number;
observabilityExtractTotalFrames?: number;
observabilityExtractMaxFramesPerVideo?: number;
observabilityExtractAvgFramesPerVideo?: number;
observabilityExtractVfrProbeMs?: number;
observabilityExtractVfrPreflightMs?: number;
observabilityExtractVfrPreflightCount?: number;
observabilityExtractCacheHits?: number;
observabilityExtractCacheMisses?: number;
observabilityInitDurationMs?: number;
observabilityInitTweenCount?: number;
observabilityInitElementCount?: number;
}
function renderObservabilityEventProperties(props: RenderObservabilityTelemetryPayload) {
return {
sub_timeline_wait: props.subTimelineWait,
observability_render_job_id: props.observabilityRenderJobId,
observability_composition_hash: props.observabilityCompositionHash,
observability_event_count: props.observabilityEventCount,
observability_last_phase: props.observabilityLastPhase,
observability_last_status: props.observabilityLastStatus,
observability_failed_phase: props.observabilityFailedPhase,
browser_diagnostic_count: props.browserDiagnosticCount,
browser_diagnostic_errors: props.browserDiagnosticErrors,
browser_diagnostic_page_errors: props.browserDiagnosticPageErrors,
browser_diagnostic_request_failed: props.browserDiagnosticRequestFailed,
browser_diagnostic_http_errors: props.browserDiagnosticHttpErrors,
browser_diagnostic_navigation_starts: props.browserDiagnosticNavigationStarts,
browser_diagnostic_navigation_failures: props.browserDiagnosticNavigationFailures,
browser_diagnostic_console_errors: props.browserDiagnosticConsoleErrors,
browser_diagnostic_console_warnings: props.browserDiagnosticConsoleWarnings,
capture_mode: props.captureMode,
capture_force_screenshot: props.captureForceScreenshot,
capture_worker_count: props.captureWorkerCount,
capture_use_streaming_encode: props.captureUseStreamingEncode,
capture_use_layered_composite: props.captureUseLayeredComposite,
capture_use_page_side_compositing: props.captureUsePageSideCompositing,
capture_has_hdr_content: props.captureHasHdrContent,
capture_browser_gpu_mode: props.captureBrowserGpuMode,
capture_protocol_timeout_ms: props.captureProtocolTimeoutMs,
capture_page_navigation_timeout_ms: props.capturePageNavigationTimeoutMs,
capture_player_ready_timeout_ms: props.capturePlayerReadyTimeoutMs,
capture_transient_retries: props.captureTransientRetries,
capture_memory_exhaustion_detected: props.captureMemoryExhaustionDetected,
de_worker_inversion: props.captureDeWorkerInversion,
de_pre_inversion_workers: props.captureDePreInversionWorkers,
composition_element_count: props.captureCompositionElementCount,
composition_element_count_source: props.captureCompositionElementCountSource,
de_short_band: props.captureDeShortBand,
de_parallel_router: props.captureDeParallelRouter,
gpu_renderer: props.captureDeGpuRenderer,
de_pre_router_workers: props.captureDePreRouterWorkers,
de_self_verify_fallback: props.captureDeSelfVerifyFallback,
de_fallback_reason: props.captureDeFallbackReason,
de_fallback_failed_db: props.captureDeFallbackFailedDb,
de_fallback_frame_index: props.captureDeFallbackFrameIndex,
de_fallback_threshold_db: props.captureDeFallbackThresholdDb,
capture_parallel_stream: props.captureParallelStream,
observability_extract_video_count: props.observabilityExtractVideoCount,
observability_extracted_video_count: props.observabilityExtractedVideoCount,
observability_extract_total_frames: props.observabilityExtractTotalFrames,
observability_extract_max_frames_per_video: props.observabilityExtractMaxFramesPerVideo,
observability_extract_avg_frames_per_video: props.observabilityExtractAvgFramesPerVideo,
observability_extract_vfr_probe_ms: props.observabilityExtractVfrProbeMs,
observability_extract_vfr_preflight_ms: props.observabilityExtractVfrPreflightMs,
observability_extract_vfr_preflight_count: props.observabilityExtractVfrPreflightCount,
observability_extract_cache_hits: props.observabilityExtractCacheHits,
observability_extract_cache_misses: props.observabilityExtractCacheMisses,
observability_init_duration_ms: props.observabilityInitDurationMs,
observability_init_tween_count: props.observabilityInitTweenCount,
observability_init_element_count: props.observabilityInitElementCount,
};
}
function redactTelemetryMessage(value: string): string {
return redactTelemetryString(value);
}
export function trackCommand(command: string, runId?: string): void {
trackEvent("cli_command", {
command,
...runIdField(runId),
});
}
export function trackRenderComplete(
props: {
durationMs: number;
fps: number;
quality: string;
/** Authoring workflow skill that drove this render (e.g. "product-launch-video"). */
authoringSkill?: string;
workers?: number;
// Worker auto-sizing provenance (RenderPerfSummary.workerSizing). Answers
// "why N workers?" fleet-wide, and validates the advisory per-worker heap
// budget before it's enforced (field OOM: 6 auto workers on a 24GB/4GB-heap
// machine — see computeWorkerSizing in @hyperframes/engine).
workersBoundBy?: string;
workersCpuBased?: number;
workersMemoryBased?: number;
workersHeapBased?: number;
workersFrameBased?: number;
workersHeapLimitMb?: number;
workersExceedHeapAdvisory?: boolean;
docker: boolean;
gpu: boolean;
// Static-frame dedup outcome (opt-out HF_STATIC_DEDUP=false). Undefined on
// render paths with no capture session.
staticDedupEnabled?: boolean;
staticDedupArmed?: boolean;
staticDedupSkipReason?: string;
staticDedupPredictedFrames?: number;
staticDedupReusedFrames?: number;
// BeginFrame no-damage reuse outcome (Linux/Docker lastFrameCache — the BF
// counterpart of static dedup). Undefined outside beginframe capture mode.
beginFrameNoDamageFrames?: number;
beginFrameHasDamageFrames?: number;
// drawElement fast-capture outcome (default-on release visibility).
// Undefined on render paths with no capture session.
deCaptureMode?: string;
deCompileGate?: string;
deClampReason?: string;
deWorkerInversion?: string;
dePreInversionWorkers?: number;
compositionElementCount?: number;
compositionElementCountSource?: string;
deShortBand?: string;
deParallelRouter?: string;
dePreRouterWorkers?: number;
deGateReason?: string;
/** Low-cardinality GPU bucket from DE session init (`<backend>/<vendor>`, e.g. `d3d11/nvidia`). */
gpuRenderer?: string;
deWorkerEncode?: boolean;
deVerifyArmed?: number;
deVerifyChecked?: number;
deVerifyMinDb?: number;
deVerifyInitMs?: number;
deSelfVerifyFallback?: boolean;
deFallbackReason?: string;
deFallbackFailedDb?: number;
deFallbackFrameIndex?: number;
deFallbackThresholdDb?: number;
deBlankSuspects?: number;
deBlankDeterministicAccepts?: number;
deBlankRecaptures?: number;
deBoundaryFrames?: number;
deNcprFallbacks?: number;
// "cli" when triggered by `hyperframes render` (default), "studio" when
// triggered by a studio preview-server render (POST /api/projects/:id/render).
source?: "cli" | "studio";
// Composition metadata
compositionDurationMs?: number;
compositionWidth?: number;
compositionHeight?: number;
totalFrames?: number;
// Processing efficiency
speedRatio?: number;
captureAvgMs?: number;
/** Warmup-robust per-frame capture median (basis for speedup estimates). */
captureP50Ms?: number;
/** <video> element count (speedup segmentation: injection comps read lower). */
videoCount?: number;
capturePeakMs?: number;
// Resource usage
peakMemoryMb?: number;
memoryFreeMb?: number;
tmpPeakBytes?: number;
// Per-stage timings (subset of RenderPerfSummary.stages)
stageCompileMs?: number;
stageVideoExtractMs?: number;
stageAudioProcessMs?: number;
stageCaptureMs?: number;
stageCaptureSetupMs?: number;
stageCaptureFrameMs?: number;
stageEncodeMs?: number;
stageAssembleMs?: number;
// Video-extraction breakdown (from RenderPerfSummary.videoExtractBreakdown)
extractResolveMs?: number;
extractHdrProbeMs?: number;
extractHdrPreflightMs?: number;
extractHdrPreflightCount?: number;
extractVfrProbeMs?: number;
extractVfrPreflightMs?: number;
extractVfrPreflightCount?: number;
extractPhase3Ms?: number;
extractCacheHits?: number;
extractCacheMisses?: number;
// Attribute this event to a specific user (e.g. the browser user who
// triggered a studio render); defaults to the install anonymousId.
distinctId?: string;
} & RenderObservabilityTelemetryPayload,
): void {
trackEvent(
"render_complete",
{
// Spread first: explicit de_* keys below (sourced from the more
// authoritative perfSummary.drawElement, always present on this
// success path) must win over the observability-capture fallback
// this shares with trackRenderError's failure path.
...renderObservabilityEventProperties(props),
duration_ms: props.durationMs,
fps: props.fps,
quality: props.quality,
authoring_skill: props.authoringSkill,
workers: props.workers,
workers_bound_by: props.workersBoundBy,
workers_cpu_based: props.workersCpuBased,
workers_memory_based: props.workersMemoryBased,
workers_heap_based: props.workersHeapBased,
workers_frame_based: props.workersFrameBased,
workers_heap_limit_mb: props.workersHeapLimitMb,
workers_exceed_heap_advisory: props.workersExceedHeapAdvisory,
docker: props.docker,
gpu: props.gpu,
static_dedup_enabled: props.staticDedupEnabled,
static_dedup_armed: props.staticDedupArmed,
static_dedup_skip_reason: props.staticDedupSkipReason,
static_dedup_predicted_frames: props.staticDedupPredictedFrames,
static_dedup_reused_frames: props.staticDedupReusedFrames,
begin_frame_no_damage_frames: props.beginFrameNoDamageFrames,
begin_frame_has_damage_frames: props.beginFrameHasDamageFrames,
de_capture_mode: props.deCaptureMode,
de_compile_gate: props.deCompileGate,
de_clamp_reason: props.deClampReason,
de_worker_inversion: props.deWorkerInversion,
de_pre_inversion_workers: props.dePreInversionWorkers,
composition_element_count: props.compositionElementCount,
composition_element_count_source: props.compositionElementCountSource,
de_short_band: props.deShortBand,
de_parallel_router: props.deParallelRouter,
de_pre_router_workers: props.dePreRouterWorkers,
de_gate_reason: props.deGateReason,
gpu_renderer: props.gpuRenderer,
de_worker_encode: props.deWorkerEncode,
de_verify_armed: props.deVerifyArmed,
de_verify_checked: props.deVerifyChecked,
de_verify_min_db: props.deVerifyMinDb,
de_verify_init_ms: props.deVerifyInitMs,
de_self_verify_fallback: props.deSelfVerifyFallback,
de_fallback_reason: props.deFallbackReason,
de_fallback_failed_db: props.deFallbackFailedDb,
de_fallback_frame_index: props.deFallbackFrameIndex,
de_fallback_threshold_db: props.deFallbackThresholdDb,
de_blank_suspects: props.deBlankSuspects,
de_blank_deterministic_accepts: props.deBlankDeterministicAccepts,
de_blank_recaptures: props.deBlankRecaptures,
de_boundary_frames: props.deBoundaryFrames,
de_ncpr_fallbacks: props.deNcprFallbacks,
...powerStateFields(),
source: props.source ?? "cli",
composition_duration_ms: props.compositionDurationMs,
composition_width: props.compositionWidth,
composition_height: props.compositionHeight,
total_frames: props.totalFrames,
speed_ratio: props.speedRatio,
capture_avg_ms: props.captureAvgMs,
capture_p50_ms: props.captureP50Ms,
video_count: props.videoCount,
capture_peak_ms: props.capturePeakMs,
peak_memory_mb: props.peakMemoryMb,
memory_free_mb: props.memoryFreeMb,
tmp_peak_bytes: props.tmpPeakBytes,
stage_compile_ms: props.stageCompileMs,
stage_video_extract_ms: props.stageVideoExtractMs,
stage_audio_process_ms: props.stageAudioProcessMs,
stage_capture_ms: props.stageCaptureMs,
stage_capture_setup_ms: props.stageCaptureSetupMs,
stage_capture_frame_ms: props.stageCaptureFrameMs,
stage_encode_ms: props.stageEncodeMs,
stage_assemble_ms: props.stageAssembleMs,
extract_resolve_ms: props.extractResolveMs,
extract_hdr_probe_ms: props.extractHdrProbeMs,
extract_hdr_preflight_ms: props.extractHdrPreflightMs,
extract_hdr_preflight_count: props.extractHdrPreflightCount,
extract_vfr_probe_ms: props.extractVfrProbeMs,
extract_vfr_preflight_ms: props.extractVfrPreflightMs,
extract_vfr_preflight_count: props.extractVfrPreflightCount,
extract_phase3_ms: props.extractPhase3Ms,
extract_cache_hits: props.extractCacheHits,
extract_cache_misses: props.extractCacheMisses,
},
props.distinctId,
);
// Send immediately instead of waiting for the exit-time flush. The render
// command's normal teardown (agent-pipe EPIPE → process.exit(0), or an
// explicit process.exit) kills the lazy beforeExit flush mid-flight, which
// is why only ~10-15% of successful renders ever produced a render_complete
// — and the survivors skewed toward users with low RTT to PostHog. The
// process is alive and idle here; if it still dies mid-request, the queue
// keeps the event for the exit-time flushSync() fallback.
void flush();
}
export function trackRenderError(
props: {
fps: number;
quality: string;
/** Authoring workflow skill that drove this render (e.g. "product-launch-video"). */
authoringSkill?: string;
docker: boolean;
workers?: number;
gpu?: boolean;
source?: "cli" | "studio";
failedStage?: string;
errorMessage?: string;
elapsedMs?: number;
peakMemoryMb?: number;
memoryFreeMb?: number;
// Attribute this event to a specific user (e.g. the browser user who
// triggered a studio render); defaults to the install anonymousId.
distinctId?: string;
} & RenderObservabilityTelemetryPayload,
): void {
trackEvent(
"render_error",
{
fps: props.fps,
quality: props.quality,
authoring_skill: props.authoringSkill,
docker: props.docker,
workers: props.workers,
gpu: props.gpu,
source: props.source ?? "cli",
failed_stage: props.failedStage,
error_message: props.errorMessage ? redactTelemetryMessage(props.errorMessage) : undefined,
elapsed_ms: props.elapsedMs,
peak_memory_mb: props.peakMemoryMb,
memory_free_mb: props.memoryFreeMb,
...powerStateFields(),
// gpu_renderer arrives via renderObservabilityEventProperties below:
// on the failure path perfSummary is never built, so live capture
// observability is the only source. Backend attribution matters MOST
// here — a win32 D3D11 crash is what the rollout is watching for.
...renderObservabilityEventProperties(props),
},
props.distinctId,
);
// Same rationale as trackRenderComplete: error paths process.exit(1) before
// the lazy flush can win its race — send now, exit-time fallback covers the rest.
void flush();
}
export function trackRenderObservation(props: {
source?: "cli" | "studio";
renderJobId?: string;
phase?: string;
status?: string;
compositionHash?: string;
elapsedMs?: number;
durationMs?: number;
message?: string;
workerCount?: number;
forceScreenshot?: boolean;
useStreamingEncode?: boolean;
useLayeredComposite?: boolean;
usePageSideCompositing?: boolean;
hasHdrContent?: boolean;
captureMode?: string;
captureOperation?: string;
framesCompleted?: number;
totalFrames?: number;
heartbeatIndex?: number;
stageElapsedMs?: number;
videoCount?: number;
extractedVideoCount?: number;
totalFramesExtracted?: number;
maxFramesPerVideo?: number;
avgFramesPerExtractedVideo?: number;
vfrPreflightCount?: number;
vfrPreflightMs?: number;
cacheHits?: number;
cacheMisses?: number;
}): void {
trackEvent("render_observation", {
source: props.source ?? "cli",
render_job_id: props.renderJobId,
phase: props.phase,
status: props.status,
composition_hash: props.compositionHash,
elapsed_ms: props.elapsedMs,
duration_ms: props.durationMs,
message: props.message ? redactTelemetryMessage(props.message) : undefined,
worker_count: props.workerCount,
force_screenshot: props.forceScreenshot,
use_streaming_encode: props.useStreamingEncode,
use_layered_composite: props.useLayeredComposite,
use_page_side_compositing: props.usePageSideCompositing,
has_hdr_content: props.hasHdrContent,
capture_mode: props.captureMode,
capture_operation: props.captureOperation,
frames_completed: props.framesCompleted,
total_frames: props.totalFrames,
heartbeat_index: props.heartbeatIndex,
stage_elapsed_ms: props.stageElapsedMs,
video_count: props.videoCount,
extracted_video_count: props.extractedVideoCount,
total_frames_extracted: props.totalFramesExtracted,
max_frames_per_video: props.maxFramesPerVideo,
avg_frames_per_extracted_video: props.avgFramesPerExtractedVideo,
vfr_preflight_count: props.vfrPreflightCount,
vfr_preflight_ms: props.vfrPreflightMs,
extract_cache_hits: props.cacheHits,
extract_cache_misses: props.cacheMisses,
});
}
export function trackInitTemplate(templateId: string, props?: { tailwind?: boolean }): void {
trackEvent("init_template", { template: templateId, tailwind: props?.tailwind });
}
/**
* One event per registry item written into a project.
*
* `cli_command` records that `add` ran, never what it installed, so the
* catalog cannot be ranked by what people actually pull — and the registry is
* served from raw.githubusercontent.com, which gives us no per-item counter
* either. `add` is the only place an item lands in a project, so this is the
* one signal that answers "which block is worth building more of".
*
* `requested` separates the item the user named from the transitive
* `registryDependencies` dragged in behind it. A dependency installed
* alongside something else is not a vote for itself, and collapsing the two
* would rank a popular dependency above everything that depends on it.
*
* Item names are public registry identifiers, never user content or project
* data. This routes through `trackEvent`, so an install that opted out
* (`hyperframes telemetry disable`, `HYPERFRAMES_NO_TELEMETRY`, `DO_NOT_TRACK`)
* emits nothing.
*/
export function trackRegistryItemAdded(props: {
item: string;
itemType: string;
requested: boolean;
}): void {
trackEvent("registry_item_added", {
item: props.item,
item_type: props.itemType,
requested: props.requested,
});
}
export function trackBrowserInstall(): void {
trackEvent("browser_install", {});
}
// Sign-in lifecycle. The CLI tracks command and render lifecycles but never
// authentication, so `auth login` outcomes are invisible on the observability
// dashboards — a completed sign-in, a browser flow the user abandoned, and a
// rejected key all look identical (i.e. absent). These three events close that
// gap so the sign-in funnel is measurable like the render funnel already is.
// `method` is "oauth" (the default browser PKCE flow), "device" (attended
// RFC 8628 flow), or "api_key". No token,
// key, identity, email, or free text is ever attached — only the method and a
// low-cardinality outcome/reason.
//
// The three trackers accept an optional `distinctId`, forwarded to trackEvent
// exactly like trackRenderComplete/trackRenderError already do. It is unused
// today (events attribute to the install's anonymousId), but pre-plumbing it
// makes attributing a completed sign-in to a resolved identity later a one-line
// change at the callsite rather than a signature sweep.
export type AuthLoginMethod = "oauth" | "device" | "api_key";
export type AuthLoginFailureReason =
| "flow_error" // OAuth authorization/exchange threw a real error
| "flow_timeout" // OAuth callback wait elapsed (user closed the tab / walked away)
| "no_credential" // flow reported success but nothing was persisted
| "rejected" // backend rejected the supplied API key (401)
| "invalid_input" // key was empty, header-unsafe, or too short
| "aborted"; // prompt cancelled, or no key arrived on stdin before timeout
export function trackAuthLoginStarted(method: AuthLoginMethod, distinctId?: string): void {
trackEvent("auth_login_started", { method }, distinctId);
}
export function trackAuthLoginCompleted(method: AuthLoginMethod, distinctId?: string): void {
trackEvent("auth_login_completed", { method }, distinctId);
}
export function trackAuthLoginFailed(
method: AuthLoginMethod,
reason: AuthLoginFailureReason,
distinctId?: string,
): void {
trackEvent("auth_login_failed", { method, reason }, distinctId);
}
// Associate this install with the signed-in HeyGen account after a completed
// sign-in. Emits a PostHog `$identify` alias whose `$anon_distinct_id` is the
// install's anonymousId, so events recorded before sign-in stitch to the same
// person instead of stranding as a separate anonymous profile. Routed through
// trackEvent so it shares the opt-out gate and flush path — a no-op when
// telemetry is disabled. `distinctId` is the account email (else username);
// see the privacy notice in showTelemetryNotice and docs/packages/cli.mdx.
export function identifyUser(distinctId: string): void {
if (!distinctId) return;
trackEvent("$identify", { $anon_distinct_id: readConfig().anonymousId }, distinctId);
}
// A render was rejected by the output-resolution/alpha/HDR pre-flight (P1-3)
// before any browser/ffmpeg work. Counts the "caught early" saves on dashboard
// 1783183, distinct from deep render failures. `kind` is the low-cardinality
// `OutputResolutionIssueKind` (aspect-mismatch / alpha-incompatible / etc.),
// typed to the union so the metric can never carry free text.
export function trackRenderPreflightRejected(props: { kind: OutputResolutionIssueKind }): void {
trackEvent("render_preflight_rejected", { kind: props.kind });
}
export function trackCliError(props: {
error_name: string;
error_message: string;
stack_trace?: string;
command?: string;
kind: "uncaught_exception" | "unhandled_rejection" | "command_error";
/** Low-cardinality figma REST call label (e.g. "images", "files_nodes") —
* which endpoint failed, for FigmaClientError-backed failures only. */
endpoint?: string;
}): void {
trackEvent("cli_error", {
error_name: props.error_name,
// Redact before truncating — CLI messages and stack traces carry absolute
// install paths (/Users/...), cache dirs, and user-supplied args. Same
// redaction the render_* events already apply.
error_message: redactTelemetryMessage(props.error_message).slice(0, 1000),
stack_trace: props.stack_trace
? redactTelemetryMessage(props.stack_trace).slice(0, 2000)
: undefined,
command: props.command,
kind: props.kind,
endpoint: props.endpoint,
});
}
/**
* One figma import outcome (asset/tokens/component). Carries capability mix,
* dedup effectiveness, and fidelity-degradation counts — never fileKeys,
* node ids, names, or descriptions.
*/
export function trackFigmaImport(props: {
phase: "asset" | "tokens" | "component";
durationMs: number;
reused?: boolean;
tokensMode?: "variables" | "styles";
entryCount?: number;
unresolvedBindings?: number;
rasterizedNodes?: number;
rasterizeFailures?: number;
}): void {
trackEvent("figma_import", {
phase: props.phase,
duration_ms: props.durationMs,
...(props.reused !== undefined ? { reused: props.reused } : {}),
...(props.tokensMode !== undefined ? { tokens_mode: props.tokensMode } : {}),
...(props.entryCount !== undefined ? { entry_count: props.entryCount } : {}),
...(props.unresolvedBindings !== undefined
? { unresolved_bindings: props.unresolvedBindings }
: {}),
...(props.rasterizedNodes !== undefined ? { rasterized_nodes: props.rasterizedNodes } : {}),
...(props.rasterizeFailures !== undefined
? { rasterize_failures: props.rasterizeFailures }
: {}),
});
}
// Report why a command failed before it exits non-zero. cli_command_result
// records the failure but not the reason; this fills that gap via cli_error so
// command failures are diagnosable. Enqueues synchronously — the process `exit`
// handler flushes it. Drop this into any command's failure path.
export function trackCommandFailure(command: string, err: unknown): void {
const error = err instanceof Error ? err : new Error(String(err));
trackCliError({
error_name: error.name,
error_message: error.message,
stack_trace: error.stack,
command,
kind: "command_error",
});
}
// Whisper being absent/uninstallable is an environment prerequisite gap, not a
// command crash — track it on its own low-severity metric instead of cli_error
// so the command-failure budget reflects real bugs. `optional` records whether
// the caller (init / skill pipeline) treated captions as skippable.
export function trackTranscribeUnavailable(props: { optional: boolean }): void {
trackEvent("transcribe_unavailable", { optional: props.optional });
}
// grade-compare / compare stand up headless Chrome and render up to 16 cells.
// Cell count, truncation-cap hits, and whether the render-ready timeout fired
// are the signals needed before safely lifting the cap. Low-cardinality only.
export function trackCompareSheet(props: {
command: "grade-compare" | "compare";
cells: number;
truncated: boolean;
total: number;
renderReadyTimedOut: boolean;
}): void {
trackEvent("media_use_compare", {
command: props.command,
cells: props.cells,
truncated: props.truncated,
total: props.total,
render_ready_timed_out: props.renderReadyTimedOut,
});
}
// A skills install was skipped because a required prerequisite binary is
// absent from PATH (e.g. git on a fresh Windows box). Best-effort callers
// (init) skip cleanly rather than crash, so the skip is otherwise invisible;
// this surfaces the rare environments that hit it. `reason` is a low-cardinality
// binary tag (e.g. "git_missing"), never a path or free text.
export function trackSkillsInstallSkipped(props: { reason: string }): void {
trackEvent("cli skill install skipped", { reason: props.reason });
}
export function trackRenderFeedback(props: {
rating: number;
renderDurationMs?: number;
comment?: string;
doctorSummary?: string;
/**
* Join key shared with the forwarded feedback report (Slack/backend): the
* same uuid rides in the report's env string as `fid=…`, so a wild report
* resolves to exactly one PostHog `cli_render_feedback` event and vice versa.
*/
feedbackId?: string;
/** render_job_id values of this install's recent renders (newest last). */
recentRenderIds?: string[];
}): void {
// Plain product event, not a PostHog survey response: nothing here is served
// by the surveys product (no survey definition, no targeting, no popover).
trackEvent("cli_render_feedback", {
rating: props.rating,
rating_scale: FEEDBACK_RATING_SCALE,
...(props.comment ? { comment: props.comment } : {}),
...(props.renderDurationMs !== undefined ? { render_duration_ms: props.renderDurationMs } : {}),
...(props.doctorSummary ? { doctor_summary: props.doctorSummary } : {}),
...(props.feedbackId ? { feedback_id: props.feedbackId } : {}),
// Comma-joined: EventProperties values are scalars only.
...(props.recentRenderIds?.length
? { recent_render_ids: props.recentRenderIds.join(",") }
: {}),
});
}
/**
* A catalog search that found nothing worth installing.
*
* This is the only path that ever sends a query anywhere, and it is a separate
* deliberate command rather than something `catalog --query` does on its own:
* plain search stays entirely local, which is what the CLI promises. The query
* is the point of the report — it names a move the catalog does not have yet,
* so the gaps can be read directly rather than guessed from install counts.
*/
export function trackCatalogSearchMiss(props: {
query: string;
wanted?: string;
tier?: string;
}): void {
trackEvent("cli_catalog_search_miss", {
query: props.query,
...(props.wanted ? { wanted: props.wanted } : {}),
...(props.tier ? { tier: props.tier } : {}),
});
}
export function trackCommandResult(props: {
command: string;
success: boolean;
exitCode: number;
durationMs: number;
runId?: string;
}): void {
trackEvent("cli_command_result", {
command: props.command,
success: props.success,
exit_code: props.exitCode,
duration_ms: props.durationMs,
...runIdField(props.runId),
});
}
export function trackCheckReport(props: {
contrastGate: boolean;
motionGate: boolean;
captionZoneGate: boolean;
frameCheckGate: boolean;
snapshotsGate: boolean;
lintErrors: number;
lintWarnings: number;
runtimeErrors: number;
runtimeWarnings: number;
layoutErrors: number;
layoutWarnings: number;
motionErrors: number;
motionWarnings: number;
contrastErrors: number;
contrastWarnings: number;
launchSettleMs: number;
seekLoopMs: number;
contrastMs: number;
gridPoints: number;
contrastPoints: number;
ok: boolean;
exitCode: number;
runId?: string;
}): void {
trackEvent("check_report", {
gate_contrast: props.contrastGate,
gate_motion: props.motionGate,
gate_caption_zone: props.captionZoneGate,
gate_frame_check: props.frameCheckGate,
gate_snapshots: props.snapshotsGate,
lint_errors: props.lintErrors,
lint_warnings: props.lintWarnings,
runtime_errors: props.runtimeErrors,
runtime_warnings: props.runtimeWarnings,
layout_errors: props.layoutErrors,
layout_warnings: props.layoutWarnings,
motion_errors: props.motionErrors,
motion_warnings: props.motionWarnings,
contrast_errors: props.contrastErrors,
contrast_warnings: props.contrastWarnings,
launch_settle_ms: props.launchSettleMs,
seek_loop_ms: props.seekLoopMs,
contrast_ms: props.contrastMs,
grid_points: props.gridPoints,
contrast_points: props.contrastPoints,
ok: props.ok,
exit_code: props.exitCode,
...runIdField(props.runId),
});
}
/**
* One lint pass over a project. `code_counts` is what makes "which rules
* actually fire" answerable; `rule_group_ms` and `slowest_rule` are what make
* "which rules are expensive" answerable. Only lint rule codes and timings are
* sent — never file paths, project names, or composition source.
*/
export function trackLintReport(props: {
/** The command that ran the lint: "lint" or "check". */
command: string;
durationMs: number;
filesScanned: number;
errorCount: number;
warningCount: number;
infoCount: number;
/** Finding count keyed by lint rule code. */
codeCounts: Record<string, number>;
/** Milliseconds spent per rule-source module, summed across files. */
ruleGroupMs: Record<string, number>;
/** Slowest single rule as `<group>#<index>`, across every file in the run. */
slowestRule: string;
slowestRuleMs: number;
/** How many rules this build ran, so a ruleset change is visible in the data. */
ruleCount: number;
/**
* Rule count per group. `slowest_rule` is positional, so a group that changed
* size between two builds has indices that no longer mean the same thing.
*/
ruleGroupCounts: Record<string, number>;
runId?: string;
}): void {
trackEvent("lint_report", {
command: props.command,
duration_ms: Math.round(props.durationMs),
files_scanned: props.filesScanned,
error_count: props.errorCount,
warning_count: props.warningCount,
info_count: props.infoCount,
codes: Object.keys(props.codeCounts).sort(),
code_counts: props.codeCounts,
rule_group_ms: props.ruleGroupMs,
slowest_rule: props.slowestRule,
slowest_rule_ms: Math.round(props.slowestRuleMs),
rule_count: props.ruleCount,
rule_group_counts: props.ruleGroupCounts,
...runIdField(props.runId),
});
}
/**
* A finding that survived one or more edits to the file it was reported on.
*
* `cleared: false` with a high `edits` is the signal that matters most: a rule
* an agent kept trying and failing to satisfy. `cleared: true` gives the
* distribution to compare it against — how many edits a normal finding costs.
*/
export function trackLintRuleStreak(props: {
code: string;
severity: string;
edits: number;
cleared: boolean;
command: string;
runId?: string;
}): void {
trackEvent("lint_rule_streak", {
code: props.code,
severity: props.severity,
edits: props.edits,
cleared: props.cleared,
command: props.command,
...runIdField(props.runId),
});
}