mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-09-03 04:38:33 +00:00
Step 2 of the DE engagement plan: multi-worker drawElement capture through the streaming encoder, with the full runtime self-verification net riding along — the confinement rule that kept the parallel clamp in place is now satisfied on this path. Opt-in via HF_DE_PARALLEL_STREAM=true; default routing (including the #2026 single-worker inversion) is unchanged. Mechanism: - distributeFramesInterleaved + WorkerTask.frameStride: worker i captures frames i, i+N, i+2N... — seek-based capture makes stride free and the ordered writer's reorder window shrinks from totalFrames/N to N (contiguous chunks serialize workers behind the writer). - Depth-2 pipelined worker-encode produce in the parallel worker loop (the same shape as the sequential loop; frame k's in-page encode overlaps k+stride's produce). HF_DE_PAR_DEBUG=1 traces the first frames per worker. - Drain guard extracted to createDrainFrameGuard (session-parameterized): every parallel frame gets the SAME blank-guard + PSNR self-verify as the sequential drain, against its owning worker's pre-injection ground truth (all sessions arm identical sample indices from CaptureOptions.compositionDurationSeconds). - FrameReorderBuffer.abort(err): a failed worker (e.g. verification error) rejects all parked and future waiters — without this, peers park forever in waitForFrame and the pool (which awaits ALL workers before surfacing errors) deadlocks. Found by the verify-trip test; unit-tested. - The typed DrawElementVerificationError is preserved past the pool's error-string flattening so the orchestrator's verify-retry recognizes it. - Static-dedup stride hazard fixed: lastEncodeResult reuse now requires EVERY frame in (lastEncodeResultFrame, i] to be predicted-static (sequential capture reduces to the old has(i) check). - Workers get separate browser PROCESSES under the flag: pages co-tenant in one browser starve non-active pages of BeginFrames on the paint-wait path (measured 86s vs 30s on a 3,245-frame rAF comp). Validation: - Happy path W3: verify samples pass across workers (4x inf on the 2,381f comp), output vs single-worker DE = 59.3dB (encode noise floor) — the interleave + dedup-stride produce identical pixels. - Verify-trip (marginal comp + HF_DE_VERIFY_MIN_DB=45): fails at frame 649 (32.2dB < 45), peers abort instead of deadlocking, whole render retries via parallel screenshot, RENDER_OK in 42.6s. - Canary suite 7/7 with the flag off (default paths untouched); producer orchestrator tests 99/99; engine suite 909 passed (14 pre-existing main failures, stash-A/B verified); reorder-buffer abort unit tests. Perf note: capture-only parallel speedup measured 1.38x (W2) / 1.52x (W3) over single-worker DE in the spike; end-to-end numbers on this machine are currently noisy (separate-browser init overhead + bench load) — clean benchmarks before any default routing change. The flag stays explicit opt-in; promoting it into the router replaces the #2026 W=1 pin for the same cohort. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
292 lines
9.3 KiB
TypeScript
292 lines
9.3 KiB
TypeScript
/**
|
|
* @hyperframes/engine
|
|
*
|
|
* Seekable web page to video rendering engine.
|
|
* Framework-agnostic: works with GSAP, Lottie, Three.js, CSS animations,
|
|
* or any web content that implements the window.__hf seek protocol.
|
|
*
|
|
* ## Error Convention
|
|
*
|
|
* Engine services use three error strategies depending on the operation type:
|
|
*
|
|
* - **Orchestration services throw on failure.** Browser launch, session init,
|
|
* frame capture, and CDP operations propagate errors as thrown exceptions.
|
|
* Callers are expected to catch and handle (e.g. frameCapture, browserManager,
|
|
* screenshotService, videoFrameExtractor.extractVideoFramesRange).
|
|
*
|
|
* - **FFmpeg process wrappers return `{ success, error? }` result objects.**
|
|
* Encoding, muxing, audio mixing, and streaming encode operations never reject.
|
|
* They resolve with a result that includes `success: boolean` and an optional
|
|
* `error` string (e.g. chunkEncoder, audioMixer, streamingEncoder).
|
|
*
|
|
* - **Cleanup and teardown functions never throw.** Browser close, session close,
|
|
* temp directory removal, and resource release swallow errors via `.catch(() => {})`
|
|
* to avoid masking the original failure (e.g. releaseBrowser, closeCaptureSession,
|
|
* FrameLookupTable.cleanup).
|
|
*
|
|
* - **Optional lookups return `T | undefined` or `T | null`.**
|
|
* Functions that may legitimately find nothing (resolveHeadlessShellPath,
|
|
* getFrameAtTime, detectGpuEncoder) return a nullable value instead of throwing.
|
|
*
|
|
*/
|
|
|
|
// ── Protocol types ─────────────────────────────────────────────────────────────
|
|
export type {
|
|
HfProtocol,
|
|
HfMediaElement,
|
|
HfTransitionMeta,
|
|
CaptureOptions,
|
|
CaptureVideoMetadataHint,
|
|
CaptureResult,
|
|
CaptureBufferResult,
|
|
CapturePerfSummary,
|
|
SubTimelineWaitOutcome,
|
|
} from "./types.js";
|
|
|
|
// ── Configuration ──────────────────────────────────────────────────────────────
|
|
export {
|
|
resolveConfig,
|
|
DEFAULT_CONFIG,
|
|
scaleProtocolTimeoutForComposition,
|
|
type EngineConfig,
|
|
} from "./config.js";
|
|
export {
|
|
DEFAULT_VP9_CPU_USED,
|
|
MAX_VP9_CPU_USED,
|
|
MIN_VP9_CPU_USED,
|
|
normalizeVp9CpuUsed,
|
|
} from "./services/vp9Options.js";
|
|
export {
|
|
getSystemTotalMb,
|
|
isLowMemorySystem,
|
|
LOW_MEMORY_TOTAL_MB_THRESHOLD,
|
|
} from "./services/systemMemory.js";
|
|
|
|
// ── Browser management ─────────────────────────────────────────────────────────
|
|
export {
|
|
acquireBrowser,
|
|
releaseBrowser,
|
|
drainBrowserPool,
|
|
resolveHeadlessShellPath,
|
|
resolveBrowserGpuMode,
|
|
buildChromeArgs,
|
|
ENABLE_BROWSER_POOL,
|
|
type BuildChromeArgsOptions,
|
|
type CaptureMode,
|
|
type AcquiredBrowser,
|
|
} from "./services/browserManager.js";
|
|
|
|
// ── Frame capture pipeline ──────────────────────────────────────────────────────
|
|
export {
|
|
createCaptureSession,
|
|
initializeSession,
|
|
closeCaptureSession,
|
|
captureFrame,
|
|
captureFrameToBuffer,
|
|
captureFrameToBufferPipelined,
|
|
captureFramesBatchPipelined,
|
|
DrawElementVerificationError,
|
|
isDrawElementVerificationError,
|
|
recaptureDrawElementFrameForVerify,
|
|
completeDeferredDrawElementInit,
|
|
writeCapturedFrame,
|
|
discardWarmupCapture,
|
|
getCompositionDuration,
|
|
getCapturePerfSummary,
|
|
prepareCaptureSessionForReuse,
|
|
type CaptureSession,
|
|
isTransientBrowserError,
|
|
isMemoryExhaustionError,
|
|
type BeforeCaptureHook,
|
|
type DiscardWarmupInnerCapture,
|
|
} from "./services/frameCapture.js";
|
|
|
|
// ── Screenshot (BeginFrame) ─────────────────────────────────────────────────────
|
|
export {
|
|
beginFrameCapture,
|
|
pageScreenshotCapture,
|
|
getCdpSession,
|
|
injectVideoFramesBatch,
|
|
syncVideoFrameVisibility,
|
|
cdpSessionCache,
|
|
probeBeginFrameLiveness,
|
|
initTransparentBackground,
|
|
captureAlphaPng,
|
|
applyDomLayerMask,
|
|
removeDomLayerMask,
|
|
DOM_LAYER_MASK_STYLE_ID,
|
|
type BeginFrameResult,
|
|
} from "./services/screenshotService.js";
|
|
|
|
// ── Encoding ───────────────────────────────────────────────────────────────────
|
|
export {
|
|
buildEncoderArgs,
|
|
encodeFramesFromDir,
|
|
encodeFramesChunkedConcat,
|
|
muxVideoWithAudio,
|
|
applyFaststart,
|
|
detectGpuEncoder,
|
|
ENCODER_PRESETS,
|
|
getEncoderPreset,
|
|
type GpuEncoder,
|
|
} from "./services/chunkEncoder.js";
|
|
export type { EncoderOptions, EncodeResult, MuxResult } from "./services/chunkEncoder.types.js";
|
|
|
|
export {
|
|
spawnStreamingEncoder,
|
|
createFrameReorderBuffer,
|
|
type StreamingEncoder,
|
|
type StreamingEncoderOptions,
|
|
type StreamingEncoderResult,
|
|
type FrameReorderBuffer,
|
|
} from "./services/streamingEncoder.js";
|
|
|
|
// ── Media processing ───────────────────────────────────────────────────────────
|
|
export {
|
|
parseVideoElements,
|
|
parseImageElements,
|
|
extractVideoFramesRange,
|
|
extractAllVideoFrames,
|
|
resolveProjectRelativeSrc,
|
|
getFrameAtTime,
|
|
createFrameLookupTable,
|
|
FrameLookupTable,
|
|
analyzeClipMediaFit,
|
|
type VideoElement,
|
|
type ImageElement,
|
|
type ExtractedFrames,
|
|
type ExtractionOptions,
|
|
type ExtractionResult,
|
|
type ExtractionPhaseBreakdown,
|
|
type VideoFrameFormat,
|
|
VIDEO_FRAME_FORMATS,
|
|
isVideoFrameFormat,
|
|
} from "./services/videoFrameExtractor.js";
|
|
|
|
export { createVideoFrameInjector } from "./services/videoFrameInjector.js";
|
|
|
|
export { parseAudioElements, processCompositionAudio } from "./services/audioMixer.js";
|
|
export type {
|
|
AudioElement,
|
|
AudioTrack,
|
|
AudioVolumeKeyframe,
|
|
MixResult,
|
|
} from "./services/audioMixer.types.js";
|
|
|
|
// ── Parallel rendering ─────────────────────────────────────────────────────────
|
|
export {
|
|
calculateOptimalWorkers,
|
|
distributeFrames,
|
|
distributeFramesInterleaved,
|
|
executeParallelCapture,
|
|
mergeWorkerFrames,
|
|
getSystemResources,
|
|
type WorkerTask,
|
|
type WorkerResult,
|
|
type ParallelProgress,
|
|
} from "./services/parallelCoordinator.js";
|
|
|
|
// ── File server ────────────────────────────────────────────────────────────────
|
|
export {
|
|
createFileServer,
|
|
type FileServerOptions,
|
|
type FileServerHandle,
|
|
} from "./services/fileServer.js";
|
|
|
|
// ── Utilities ──────────────────────────────────────────────────────────────────
|
|
export { quantizeTimeToFrame, MEDIA_VISUAL_STYLE_PROPERTIES } from "@hyperframes/core";
|
|
|
|
export {
|
|
assertSwiftShader,
|
|
readWebGlVendorInfo,
|
|
SwiftShaderAssertionError,
|
|
BROWSER_GPU_NOT_SOFTWARE,
|
|
} from "./utils/assertSwiftShader.js";
|
|
|
|
export { readWebGlVendorInfoFromCanvas } from "./utils/readWebGlVendorInfoFromCanvas.js";
|
|
|
|
export {
|
|
extractMediaMetadata,
|
|
extractVideoMetadata,
|
|
extractAudioMetadata,
|
|
analyzeKeyframeIntervals,
|
|
type VideoMetadata,
|
|
type AudioMetadata,
|
|
type KeyframeAnalysis,
|
|
} from "./utils/ffprobe.js";
|
|
|
|
export { assertPublicHttpsUrl, downloadToTemp, isHttpUrl } from "./utils/urlDownloader.js";
|
|
export {
|
|
runFfmpeg,
|
|
formatFfmpegError,
|
|
type RunFfmpegOptions,
|
|
type RunFfmpegResult,
|
|
} from "./utils/runFfmpeg.js";
|
|
export {
|
|
assertConfiguredFfmpegBinariesExist,
|
|
getFfmpegBinary,
|
|
getFfprobeBinary,
|
|
FFMPEG_PATH_ENV,
|
|
FFPROBE_PATH_ENV,
|
|
} from "./utils/ffmpegBinaries.js";
|
|
|
|
export { trackChildProcess, killTrackedProcesses } from "./utils/processTracker.js";
|
|
|
|
export {
|
|
decodePng,
|
|
decodePngToRgb48le,
|
|
blitRgba8OverRgb48le,
|
|
blitRgb48leRegion,
|
|
blitRgb48leAffine,
|
|
parseTransformMatrix,
|
|
roundedRectAlpha,
|
|
resampleRgb48leObjectFit,
|
|
normalizeObjectFit,
|
|
type ObjectFit,
|
|
} from "./utils/alphaBlit.js";
|
|
|
|
export { groupIntoLayers, type CompositeLayer } from "./utils/layerCompositor.js";
|
|
|
|
// ── Shader transitions ────────────────────────────────────────────────────────
|
|
export {
|
|
type TransitionFn,
|
|
TRANSITIONS,
|
|
crossfade,
|
|
sampleRgb48le,
|
|
hdrToLinear,
|
|
linearToHdr,
|
|
convertTransfer,
|
|
} from "./utils/shaderTransitions.js";
|
|
|
|
export {
|
|
initHdrReadback,
|
|
uploadAndReadbackHdrFrame,
|
|
float16ToPqRgb,
|
|
buildHdrChromeArgs,
|
|
launchHdrBrowser,
|
|
} from "./services/hdrCapture.js";
|
|
|
|
export { captureScreenshotWithAlpha } from "./services/screenshotService.js";
|
|
|
|
export {
|
|
hideVideoElements,
|
|
showVideoElements,
|
|
queryVideoElementBounds,
|
|
queryElementStacking,
|
|
type VideoElementBounds,
|
|
type ElementStackingInfo,
|
|
} from "./services/videoFrameInjector.js";
|
|
|
|
export {
|
|
isHdrColorSpace,
|
|
detectTransfer,
|
|
getHdrEncoderColorParams,
|
|
analyzeCompositionHdr,
|
|
DEFAULT_HDR10_MASTERING,
|
|
type HdrTransfer,
|
|
type HdrEncoderColorParams,
|
|
type CompositionHdrInfo,
|
|
type HdrMasteringMetadata,
|
|
} from "./utils/hdr.js";
|
|
export type { VideoColorSpace } from "./utils/ffprobe.js";
|