Commit Graph
7 Commits
Author SHA1 Message Date
miga-heygenandMiguel Ángel 71bb88719e feat(telemetry): render output-shape, adapter, count, and environment properties (#4017)
* feat(telemetry): add output-shape request facts to render events

output_resolution_preset, output_format, hdr_mode, video_frame_format,
and gif_fps_capped join render_complete/render_error, all threaded
from already-resolved RenderOptions/RenderPlan fields rather than
recomputed: createRenderPlan already computes gifFpsCapped (clamping a
requested --fps above 30 to 30 for --format gif) but only used it for
a console warning; this threads that same decision into RenderOptions
and telemetry. All five are known before the pipeline starts, so both
events carry them even when a render fails before a perfSummary
exists.

Also wires output_format/output_resolution_preset into Studio's
separate emitStudioRenderComplete/emitStudioRenderError path, since
Studio's request genuinely varies those two per-request. hdr_mode,
video_frame_format, and gif_fps_capped stay CLI-only: Studio exposes
no HDR-mode, video-frame-format, or GIF output control to users at
all, so there is no per-request value to report for those three.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* feat(telemetry): add adapters_used to render events

adapters_used lists the runtime deterministic adapters
(packages/core/src/runtime/adapters/) a composition exercises.
Investigated the actual codebase rather than the ticket's guessed
7-adapter list and found 12 real registered adapters (gsap, css,
waapi, animejs, lottie, three, typegpu, d3, leaflet, mapbox, maplibre,
google-maps) — the map/data-source adapters share the exact same
window.__hf<Name> registration mechanism and are equally real
"adapters used" facts, so all 12 are included.

Detection unions a live page.evaluate probe (piggybacked on the same
probeSession resolveCompositionElementCount already uses) with a
static regex fallback over the compiled HTML, since this is purely
observational with no gating role (unlike compositionElementCount) —
union strictly increases recall rather than needing one source of
truth. Threaded through the same plumbing chain as the already-merged
composition_element_tags/aroll_video_count/heygen_video_count.

Studio's emitStudioRenderComplete/emitStudioRenderError get this field
automatically, with no Studio-side code change, via the same shared
capture-observability spread heygen_video_count already uses.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* feat(telemetry): add audio/image/sub-composition/color-grading counts

audio_count, image_count, sub_composition_count, audio_group_count,
color_grading_count, and has_lut join composition_element_tags in the
same single scanElementTags pass — audio/image/audio-group counts read
from the scan's uncapped tag map so a 50+-distinct-tag composition
can't push a named count into the "other" bucket. sub_composition_count
counts data-composition-src mounts directly rather than reusing
collectSubCompositionSrcs, which dedupes by src and drops
placeholder/remote mounts (a different, smaller fact than the mount
count this field reports).

has_lut decodes the HTML-entity-escaped form of data-color-grading
before parsing, since linkedom's serializer re-emits it
&quot;-escaped on every compile-pipeline round-trip — without the
decode this silently reported false for every LUT-graded composition.
The LUT-presence check now mirrors normalizeLut's own definition of
"has a LUT" exactly, closing a gap where a blank or empty lut value
would have reported true.

Threaded through the same plumbing chain as the already-merged
per-tag/provenance/adapter fields; Studio's emitter picks these up
automatically via the shared capture-observability spread.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* feat(telemetry): add browser_version_major and ffmpeg_version_major

Both are already-run subprocess calls whose output was discarded:
readToolVersion (checkFFmpeg's preflight) already runs `ffmpeg
-version`, and chromeLaunchOutcome already runs `--version` as a
Chrome launch sanity probe on every render. A shared
extractMajorVersion helper pulls the first X.Y-shaped number from
either banner, threaded onto EnvironmentCheckOutcome/
EnvironmentCheckResult, then RenderOptions right after
runEnvironmentChecks resolves in the local render path only.

Docker renders never populate these two RenderOptions fields (the
containerized producer runs its own preflight the host CLI never
sees), so both properties come back absent there rather than
reporting the host's own toolchain versions for a render that didn't
use them.

Studio does not call runEnvironmentChecks or this ffmpeg/Chrome
preflight at all before creating a render job, so its render_complete/
render_error events will not carry these two properties unless
Studio's own startup path is separately instrumented — a genuine
parity gap, not an oversight in this change.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* feat(telemetry): add authoring_skill_source and authoring_skill_invalid

Both derive from resolution logic createRenderPlan already runs:
authoring_skill_source names whichever step actually won attribution
(an explicit --skill flag, or the skill persisted in hyperframes.json)
using the same flagSkill/projectConfigSkill values already computed
for authoringSkill itself, not a second lookup. authoring_skill_invalid
surfaces the raw --skill value when it fails slug-shape validation
(normalizeSkillSlug rejects malformed input, not unregistered skill
names) — this value already exists as RenderPlan.invalidAuthoringSkill
and already drives a CLI warning in present.ts; this change threads
the same field into telemetry rather than adding a parallel one.

Threaded through both RenderOptions builders in execute.ts (single and
batch render) and all three telemetry call sites in render.ts,
including the Docker path — skill attribution is a request-level fact
independent of which backend executes the render.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* feat(telemetry): add root_body_mismatch and root_body_delta_px_bucket

Measures the composition root's authored data-width/data-height
against the scaffold's own html/body CSS size, straight from the
compiled document string scanElementTags already scans — the
source-level fact, not runtime DOM state. init.ts's applyCompositionSizing
(the #4001 fix) only patches the live DOM after the browser loads this
same HTML, so this measurement stays meaningful whether or not that fix
ran for a given render.

Reuses @hyperframes/parsers' canvasScaffoldPatterns regexes (the same
ones packages/lint's root_dimensions_mismatch rule reads) for the CSS
side. Root-tag detection here is a lighter first-data-composition-id-match
heuristic, not lint's more thorough findRootTag (which also handles a
leading decorative <svg> defs block) — that helper isn't part of
@hyperframes/lint's public API, and expanding it is out of scope here.

The delta is bucketed (0 / 1-10 / 11-50 / 51+ px) rather than reported
raw, and both fields come back absent — not a false/"0" default — when
the root or the CSS block can't be read at all.

Threaded through the same plumbing as every other static-scan field
this week, including the live-capture-observability fallback used on a
crash that never reaches perfSummary.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* feat(telemetry): add hf_env_overrides (names only, capped at 20)

Reports the sorted names of HF_*/HYPERFRAMES_* env vars present when
createRenderPlan resolves — never their values, since several of these
carry filesystem paths. Snapshotting at plan-resolution time (the very
first line of the render command, before any pipeline code runs)
matters: this render's own preflight later injects
HYPERFRAMES_FFMPEG_PATH/HYPERFRAMES_FFPROBE_PATH into its own
process.env, which would otherwise make those two always read back as
"operator overrides" whether or not the operator ever set them.

Reports an empty array, never an absent key, when nothing is set,
matching every other capped/bucketed field added this week. Capped at
20 names, documented alongside the cap constant.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* fix(cli): stop clobbering studio render_complete fields, fix an env leak

trackRenderComplete's explicit audio_count/image_count/root_body_mismatch/
adapters_used/etc. keys, sourced from perfSummary.drawElement and only ever
populated on the CLI's own render.ts path, always won the object spread over
the observability-capture fallback those same keys share with trackRenderError,
even when the direct value was undefined. studioRenderTelemetry.ts's
emitStudioRenderComplete never populates drawElement, only the capture
fallback, so 24 fields silently came back missing from every studio-triggered
render_complete event despite Studio having computed and sent the capture
value. Each field now falls back to its captureXxx counterpart via a new
directOrCapture() helper when the direct field is absent, with regression
tests proving both the fallback and that the direct value still wins when
both are present.

Also excludes HF_SHADER_WORKER_ENTRY from hf_env_overrides: cli.ts sets this
on every invocation to point the bundled shader worker pool at its own file,
sharing the HF_ prefix by coincidence rather than because an operator set it.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* style: remove em and en dashes from added code comments

No behavior change. Rewrites 22 comment lines this PR's own diff introduced,
using a comma or a period where the dash separated two clauses.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* test(producer): add a temporary diagnostic for a CI-only rootBodyMismatch failure

Not a real fix. The rootBodyMismatch test group fails deterministically in
CI (confirmed on two runs of the same head) but will not reproduce locally
by any method tried, including an isolated file run, the exact combined
producer:test:unit command, a worker-count-constrained run, and a fresh
clone with a clean install and build matching CI's own steps.

This dumps the intermediate regex-match state for the simplest failing
fixture to stderr so the next CI run's log says what actually differs.
To be removed once that answer lands.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* test(producer): widen the rootBodyMismatch diagnostic to compare import paths

Still not a real fix. The previous diagnostic (ca4e73983) proved the
imported HTML_BODY_CSS_WIDTH_FIRST_RE has 4 capture groups in the failing
CI run instead of the 2 the source and production code expect, but it used
a dynamic import(), which can resolve through a different codepath than
renderOrchestrator.ts's static import. This adds a static import of the
same binding, a require.resolve() of the resolved file path, and a direct
read of that file's own regex line, so the next run can rule in or out a
static/dynamic resolution split versus a genuinely different file on disk.

Locally both imports agree (2 groups, same object, correct file). To be
removed once the CI answer lands.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* fix(producer): read the merged 4-group scaffold-size regex correctly

main merged #4012 while this branch was in flight, consolidating
HTML_BODY_CSS_WIDTH_FIRST_RE/HEIGHT_FIRST_RE from 2 capture groups (digits
only) to 4 (prefix text, digits, suffix text, digits) so cli's
applyResolutionPreset could reuse the same regex for its in-place replace.
detectRootBodySizeMismatch still read groups 1/2 as the digits, now the
prefix/suffix text, so Number() on them was NaN and every render's
rootBodyMismatch/rootBodyDeltaPxBucket silently came back undefined. Fixed
to read groups 2/4, matching lint's core.ts readHtmlBodyCssSize exactly.

Removes the temporary diagnostic test (ca4e73983, e2e384c8e) now that its
CI log gave the real answer: both a static and dynamic import of the
regex resolved to the same file, and that file's own content had 4 groups
because this branch's merge-base predates #4012, not because of any
resolution or caching difference. A fresh CI run confirmed no CI-only
failure remains after merging main and fixing the indices.

Co-Authored-By: Miguel Ángel <miguel.sierra@heygen.com>

* chore: nudge CI (no workflow run triggered on previous push)

* chore: nudge CI again now that #4017 is out of draft

---------

Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
2026-09-17 02:10:27 +00:00
Vance IngallsandClaude Opus 5 6f0df2640b fix(cli,studio,core): close five R5 telemetry and canary findings
- A long-lived preview cached its telemetry posture in two places
  (readConfig and shouldTrack). Running `telemetry disable` in another
  terminal left it resolving canaries and injecting the CLI id for hours.
  Both caches are now dropped together at a request boundary.
- Studio minted and shipped a telemetry id for every render regardless of
  the browser profile's opt-out, and the server emitted the outcome under
  CLI policy, which cannot see localStorage or DNT. The browser now sends
  an explicit telemetryOptOut, distinct from an old client's omission.
- Any non-empty HYPERFRAMES_PREVIEW_HOST disabled the DNS-rebinding guard,
  so even a loopback bind accepted a hostile Host. The guard now holds for
  loopback binds and, on a LAN bind, admits only names this machine
  answers on.
- sunsetAfter had no reader of the current date. A scheduled workflow runs
  scripts/check-canary-sunset.ts weekly, so a failure lands on the
  rollout's owner rather than on an unrelated PR author.
- The install-state seed memo outlived `rm -rf ~/.hyperframes`,
  resurrecting a cleared cohort. Removed; it only saved a read on a
  readConfig cache miss.

Docs updated for the Host rule and the 100% exclusion carve-out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-01 17:21:41 -07:00
Miguel Ángel 4976b5e036 fix(producer): split capture average timing (#1657) 2026-06-22 18:10:05 -04:00
Miguel Ángel 121cdd2d9f fix(telemetry): attribute studio renders to the browser user (joinable funnel) (#1492)
Studio-triggered renders emit render_complete / render_error from the CLI
preview-server process, which stamps every event with the install's
anonymousId (client.ts drainQueueToPayload). The browser, meanwhile, fires
studio_session_start / studio_render_start under its own getAnonymousId(). So
the render outcome and the render start never share a person_id — verified in
data: of 15,125 users who started a studio render in 30d, ZERO have any
render_complete under any source, and the 898 studio-tagged completers are
disjoint server UUIDs. The studio render funnel — the product's core value
moment and strongest retention signal — is therefore unmeasurable.

Thread the browser's telemetry id through to the render-outcome events:

- client.ts: trackEvent takes an optional distinctId; drainQueueToPayload uses
  `event.distinctId ?? config.anonymousId`. CLI renders unchanged.
- events.ts: trackRenderComplete/trackRenderError forward an optional distinctId.
- studioRenderTelemetry.ts: emitStudioRender* pass opts.distinctId through.
- core studio-api (types.ts + routes/render.ts): the render route reads
  `telemetryDistinctId` from the request body (validated string) and passes it
  to the adapter's startRender, which already forwards opts to the emitters.
- studio (useRenderQueue.ts): include getAnonymousId() as telemetryDistinctId
  in the render POST — the same id studio_* events already use.

Result: studio render_complete/error now carry the browser user's id and join
studio_session_start / studio_render_start. Older clients that don't send the
field fall back to anonymousId (no regression). No new tracking surface — it's
the existing anonymous studio id.

Tests: per-event override forwarding (events), studio render distinctId
threading + older-client fallback (studioRenderTelemetry), and route body →
adapter forwarding incl. non-string rejection (core render route).
2026-06-16 02:55:22 -04:00
Miguel Ángel 29d6f1eac9 fix(render): add end-to-end observability (#1248) 2026-06-06 23:55:58 -04:00
James da38de1b12 test+fix(telemetry): address PR review — dev-mode gate, session-storage dedupe, payload tests
Addresses review comments on #982:

- studio shouldTrack(): adds VITE_HYPERFRAMES_NO_TELEMETRY (mirrors CLI's
  HYPERFRAMES_NO_TELEMETRY) and import.meta.env.DEV gates so dev / CI
  studio builds don't pollute production telemetry. shouldTrack() is now
  exported for testability.
- App.tsx session dedupe: moves the once-per-session check from a useRef
  (which resets on HMR / remount) to sessionStorage via new
  hasFiredSessionStart / markSessionStartFired helpers in config.ts.
- studioRenderTelemetry.ts: documents why `workers` is intentionally
  omitted from emitStudioRenderError (studio renders don't accept a
  user-supplied worker count, so early failures genuinely don't know one).
- client.ts flush(): documents fire-and-forget no-retry design so future
  hands don't accidentally add retry logic that double-counts.

Tests:
- studioRenderTelemetry.test.ts (8 tests): perfPayload mapping for every
  RenderPerfSummary field, undefined-perf path, missing-extract path,
  zero-elapsed edge case, error event shape.
- studio/telemetry/events.test.ts (4 tests): pin event names
  (studio_session_start, studio_render_start) and payload shape.
- studio/telemetry/client.test.ts (9 tests): shouldTrack() returns false
  for non-phc_ key, opt-out, doNotTrack, build-time env, vite dev mode;
  memoization.
2026-05-20 14:53:38 -04:00
James 50ade616a8 refactor(cli): extract studio render telemetry helpers to own file
Moves StudioRenderOpts, memSnapshot, perfPayload, stagesPayload,
extractPayload, emitStudioRenderComplete, emitStudioRenderError to
packages/cli/src/server/studioRenderTelemetry.ts. studioServer.ts now
has a single-line import diff.

Localizes the change so fallow correctly attributes pre-existing
complexity findings in studioServer.ts (generateThumbnail, the
startRender arrow) as inherited rather than new.
2026-05-20 14:53:38 -04:00