mirror of
https://github.com/heygen-com/hyperframes.git
synced 2026-11-15 13:58:20 +00:00
Reported: compositions with many images stutter visibly during the
first ~second of preview playback, with the only workaround being to
render the composition to mp4 and watch the output file.
Root cause: every `<img>` that has never been painted before decodes
synchronously on the main thread the first time it enters the paint
tree. With a composition of several dozen images, that's several dozen
main-thread stalls stacked inside the first paint window — visible as
jank on every seek and the first few seconds of play.
Fix: a new `packages/core/src/runtime/imageDecode.ts` module that the
runtime binds alongside `bindMediaMetadataListeners`. For every `<img>`
in the document, it:
- Sets `decoding="async"` so the browser never falls back to sync-decode
on subsequent paints.
- Calls `img.decode()` off the main thread as soon as the image has a
`currentSrc` (either immediately if already loaded, or on the next
`load` event otherwise).
- Tracks bound images in a Set so repeated polling from the runtime's
observation loop is free after the first bind.
`img.decode()` is the canonical platform primitive for this — it's a
no-op for already-decoded images, runs on the decoder thread, and
resolves when the pixels are in the bitmap cache. Decode rejections
(CORS, corrupt file, 404) are intentionally swallowed because the
runtime's existing asset-error diagnostic pipeline already surfaces
those via the `error` event.
Lazy images (`loading="lazy"`) are intentionally NOT force-loaded — if
the author opted them out, we respect that; they'll decode when they're
needed.
The module is wired in three call sites in `init.ts`:
- Initial boot (after `runAdapters("discover")`)
- Post-load of external/inline compositions
- Every 10th poll tick (catches DOM changes from author scripts,
sub-composition mounts, and dynamically-inserted content)
Tests: `imageDecode.test.ts` covers nine cases — async-flag set,
immediate vs. load-deferred decode, idempotence, torn-down no-op,
swallowed rejections, missing currentSrc, multiple imgs in one pass,
and dynamically-added imgs across bind calls. 9/9 pass.
Docs: new "Preloading and decoding" subsection in `core.mdx` under
the Runtime section, documents the invariant for future contributors.
Closes #317.
447 lines
13 KiB
Plaintext
447 lines
13 KiB
Plaintext
---
|
|
title: "@hyperframes/core"
|
|
description: "Types, HTML generation, runtime, and linter — the foundation every other package depends on."
|
|
---
|
|
|
|
The core package provides the foundational types, HTML parsing/generation, runtime, and composition linter that all other Hyperframes packages build on. If you are building tooling, writing a custom integration, or extending Hyperframes itself, this is the package you need.
|
|
|
|
```bash
|
|
npm install @hyperframes/core
|
|
```
|
|
|
|
## When to Use
|
|
|
|
<Tip>
|
|
**Most users do not need to install `@hyperframes/core` directly.** The [CLI](/packages/cli), [producer](/packages/producer), and [studio](/packages/studio) packages all depend on core internally. You only need it if you are doing one of the things listed below.
|
|
</Tip>
|
|
|
|
**Use `@hyperframes/core` when you need to:**
|
|
- Lint compositions programmatically (CI pipelines, editor plugins)
|
|
- Parse HTML compositions into structured TypeScript objects
|
|
- Generate composition HTML from data (e.g., from an API or AI agent)
|
|
- Access the Hyperframes type system for your own tooling
|
|
- Embed the Hyperframes runtime in a custom player
|
|
|
|
**Use a different package if you want to:**
|
|
- Preview compositions in the browser — use the [CLI](/packages/cli) (`npx hyperframes preview`) or [studio](/packages/studio)
|
|
- Render compositions to MP4 — use the [CLI](/packages/cli) (`npx hyperframes render`) or [producer](/packages/producer)
|
|
- Capture frames from a headless browser — use the [engine](/packages/engine)
|
|
|
|
## Package Exports
|
|
|
|
The core package has four entry points:
|
|
|
|
| Import | Description |
|
|
|--------|-------------|
|
|
| `@hyperframes/core` | Types, parsers, generators, adapters, runtime utilities |
|
|
| `@hyperframes/core/lint` | Composition linter |
|
|
| `@hyperframes/core/compiler` | Timing compiler, HTML compiler, bundler, static guard |
|
|
| `@hyperframes/core/runtime` | Pre-built IIFE runtime for browser injection |
|
|
|
|
## Types
|
|
|
|
The core type system models compositions, timeline elements, and variables:
|
|
|
|
```typescript
|
|
import type {
|
|
TimelineElement,
|
|
TimelineMediaElement,
|
|
TimelineTextElement,
|
|
TimelineCompositionElement,
|
|
TimelineElementType, // "video" | "image" | "text" | "audio" | "composition"
|
|
CompositionSpec,
|
|
CompositionVariable,
|
|
CanvasResolution, // "landscape" | "portrait"
|
|
Orientation, // "16:9" | "9:16"
|
|
FrameAdapter,
|
|
FrameAdapterContext,
|
|
} from '@hyperframes/core';
|
|
|
|
// Type guards
|
|
import {
|
|
isTextElement,
|
|
isMediaElement,
|
|
isCompositionElement,
|
|
isStringVariable,
|
|
isNumberVariable,
|
|
isColorVariable,
|
|
isBooleanVariable,
|
|
isEnumVariable,
|
|
} from '@hyperframes/core';
|
|
|
|
// Constants
|
|
import {
|
|
CANVAS_DIMENSIONS, // { landscape: { width, height }, portrait: { width, height } }
|
|
TIMELINE_COLORS,
|
|
DEFAULT_DURATIONS,
|
|
} from '@hyperframes/core';
|
|
```
|
|
|
|
### Variable Types
|
|
|
|
Compositions can expose typed variables for dynamic content:
|
|
|
|
```typescript
|
|
import type {
|
|
CompositionVariableType, // "string" | "number" | "color" | "boolean" | "enum"
|
|
StringVariable,
|
|
NumberVariable,
|
|
ColorVariable,
|
|
BooleanVariable,
|
|
EnumVariable,
|
|
} from '@hyperframes/core';
|
|
```
|
|
|
|
### Keyframe Types
|
|
|
|
```typescript
|
|
import type {
|
|
Keyframe,
|
|
KeyframeProperties,
|
|
ElementKeyframes,
|
|
StageZoom,
|
|
StageZoomKeyframe,
|
|
} from '@hyperframes/core';
|
|
|
|
import { getDefaultStageZoom } from '@hyperframes/core';
|
|
```
|
|
|
|
## Parsing and Generating HTML
|
|
|
|
Round-trip between HTML and structured data:
|
|
|
|
```typescript
|
|
import { parseHtml, generateHyperframesHtml } from '@hyperframes/core';
|
|
import type { ParsedHtml, CompositionMetadata } from '@hyperframes/core';
|
|
|
|
// Parse HTML into structured data
|
|
const parsed: ParsedHtml = parseHtml(htmlString);
|
|
// parsed.elements, parsed.gsapScript, parsed.styles, parsed.resolution, parsed.keyframes
|
|
|
|
// Extract composition metadata
|
|
import { extractCompositionMetadata } from '@hyperframes/core';
|
|
const meta: CompositionMetadata = extractCompositionMetadata(htmlString);
|
|
// meta.id, meta.duration, meta.width, meta.height, meta.variables
|
|
|
|
// Generate HTML from structured data
|
|
const html = generateHyperframesHtml(elements, {
|
|
animations,
|
|
styles,
|
|
resolution: 'landscape',
|
|
compositionId: 'my-video',
|
|
});
|
|
```
|
|
|
|
### Modifying HTML
|
|
|
|
```typescript
|
|
import {
|
|
updateElementInHtml,
|
|
addElementToHtml,
|
|
removeElementFromHtml,
|
|
validateCompositionHtml,
|
|
} from '@hyperframes/core';
|
|
|
|
// Update an element's properties
|
|
const updatedHtml = updateElementInHtml(html, 'el-1', { start: 5 });
|
|
|
|
// Add a new element
|
|
const newHtml = addElementToHtml(html, newElement);
|
|
|
|
// Remove an element
|
|
const cleanHtml = removeElementFromHtml(html, 'el-1');
|
|
|
|
// Validate HTML structure
|
|
const result = validateCompositionHtml(html);
|
|
// result.valid, result.errors
|
|
```
|
|
|
|
### GSAP Script Parsing
|
|
|
|
```typescript
|
|
import {
|
|
parseGsapScript,
|
|
serializeGsapAnimations,
|
|
updateAnimationInScript,
|
|
addAnimationToScript,
|
|
removeAnimationFromScript,
|
|
getAnimationsForElement,
|
|
validateCompositionGsap,
|
|
keyframesToGsapAnimations,
|
|
gsapAnimationsToKeyframes,
|
|
SUPPORTED_PROPS, // animatable properties
|
|
SUPPORTED_EASES, // available easing functions
|
|
} from '@hyperframes/core';
|
|
import type { GsapAnimation, GsapMethod, ParsedGsap } from '@hyperframes/core';
|
|
|
|
// Parse GSAP script into structured animations
|
|
const parsed: ParsedGsap = parseGsapScript(scriptContent);
|
|
// parsed.animations, parsed.timelineVar, parsed.preamble, parsed.postamble
|
|
|
|
// Serialize back to script
|
|
const script = serializeGsapAnimations(parsed.animations);
|
|
```
|
|
|
|
### HTML Generation
|
|
|
|
```typescript
|
|
import {
|
|
generateHyperframesHtml,
|
|
generateGsapTimelineScript,
|
|
generateHyperframesStyles,
|
|
} from '@hyperframes/core';
|
|
|
|
// Generate a complete HTML composition
|
|
const html = generateHyperframesHtml(elements, options);
|
|
|
|
// Generate just the GSAP script
|
|
const script = generateGsapTimelineScript(animations, options);
|
|
|
|
// Generate CSS styles
|
|
const { coreCss, customCss, googleFontsLink } = generateHyperframesStyles(
|
|
elements, 'landscape', customStyles
|
|
);
|
|
```
|
|
|
|
### Template Utilities
|
|
|
|
```typescript
|
|
import {
|
|
generateBaseHtml,
|
|
getStageStyles,
|
|
GSAP_CDN,
|
|
BASE_STYLES,
|
|
ELEMENT_BASE_STYLES,
|
|
MEDIA_STYLES,
|
|
TEXT_STYLES,
|
|
ZOOM_CONTAINER_STYLES,
|
|
} from '@hyperframes/core';
|
|
|
|
// Generate base HTML structure for a resolution
|
|
const baseHtml = generateBaseHtml('landscape');
|
|
const styles = getStageStyles('portrait');
|
|
```
|
|
|
|
## Linter
|
|
|
|
The composition linter checks for structural issues that would cause rendering failures or unexpected behavior. You can run it from the CLI with `npx hyperframes lint`, or call it programmatically:
|
|
|
|
```typescript
|
|
import { lintHyperframeHtml, lintMediaUrls } from '@hyperframes/core/lint';
|
|
import type {
|
|
HyperframeLintResult,
|
|
HyperframeLintFinding,
|
|
HyperframeLintSeverity, // "error" | "warning"
|
|
HyperframeLinterOptions,
|
|
} from '@hyperframes/core/lint';
|
|
|
|
const result: HyperframeLintResult = lintHyperframeHtml(html, { filePath: 'index.html' });
|
|
// result.ok, result.errorCount, result.warningCount, result.findings
|
|
|
|
for (const finding of result.findings) {
|
|
console.log(finding.severity, finding.code, finding.message);
|
|
// finding.file, finding.selector, finding.elementId, finding.fixHint, finding.snippet
|
|
}
|
|
|
|
// Additional media URL validation
|
|
const mediaFindings = lintMediaUrls(result.findings);
|
|
```
|
|
|
|
Detected issues include:
|
|
|
|
- Missing timeline registration (`window.__timelines`)
|
|
- Unmuted video elements (causes autoplay failures)
|
|
- Missing `class="clip"` on timed visible elements
|
|
- Deprecated attribute names
|
|
- Missing composition dimensions (`data-width`, `data-height`)
|
|
- Invalid `data-start` references to nonexistent clip IDs
|
|
|
|
<Info>
|
|
For a full list of what the linter catches and how to fix each issue, see [Common Mistakes](/guides/common-mistakes) and [Troubleshooting](/guides/troubleshooting).
|
|
</Info>
|
|
|
|
## Compiler
|
|
|
|
The compiler sub-package handles timing resolution, HTML compilation, and bundling:
|
|
|
|
```typescript
|
|
// Timing compiler (browser-safe — no Node.js dependencies)
|
|
import {
|
|
compileTimingAttrs,
|
|
injectDurations,
|
|
extractResolvedMedia,
|
|
clampDurations,
|
|
} from '@hyperframes/core/compiler';
|
|
import type {
|
|
UnresolvedElement,
|
|
ResolvedDuration,
|
|
ResolvedMediaElement,
|
|
CompilationResult,
|
|
} from '@hyperframes/core/compiler';
|
|
|
|
// Compile timing attributes from HTML
|
|
const compiled: CompilationResult = compileTimingAttrs(html);
|
|
|
|
// Inject resolved durations back into HTML
|
|
const updatedHtml = injectDurations(html, compiled.durations);
|
|
|
|
// Extract resolved media elements
|
|
const media: ResolvedMediaElement[] = extractResolvedMedia(html);
|
|
```
|
|
|
|
```typescript
|
|
// HTML compiler (Node.js — requires media probing)
|
|
import { compileHtml } from '@hyperframes/core/compiler';
|
|
import type { MediaDurationProber } from '@hyperframes/core/compiler';
|
|
|
|
const prober: MediaDurationProber = async (src) => getDuration(src);
|
|
const compiledHtml = await compileHtml(html, prober);
|
|
```
|
|
|
|
```typescript
|
|
// HTML bundler (Node.js — bundles to single file)
|
|
import { bundleToSingleHtml } from '@hyperframes/core/compiler';
|
|
import type { BundleOptions } from '@hyperframes/core/compiler';
|
|
|
|
const bundled = await bundleToSingleHtml({ entryPath: './index.html', inline: true });
|
|
```
|
|
|
|
```typescript
|
|
// Static guard — validate HTML contract
|
|
import { validateHyperframeHtmlContract } from '@hyperframes/core/compiler';
|
|
import type {
|
|
HyperframeStaticGuardResult,
|
|
HyperframeStaticFailureReason,
|
|
} from '@hyperframes/core/compiler';
|
|
|
|
const guard: HyperframeStaticGuardResult = validateHyperframeHtmlContract(html);
|
|
// guard.ok, guard.failures[]
|
|
// Failure reasons: "missing_composition_id" | "missing_composition_dimensions"
|
|
// | "missing_timeline_registry" | "invalid_script_syntax"
|
|
// | "invalid_static_hyperframe_contract"
|
|
```
|
|
|
|
## Runtime
|
|
|
|
The Hyperframes runtime manages playback, seeking, and clip lifecycle in the browser. The core package provides utilities for building and loading the runtime:
|
|
|
|
```typescript
|
|
import {
|
|
loadHyperframeRuntimeSource,
|
|
buildHyperframesRuntimeScript,
|
|
HYPERFRAME_RUNTIME_ARTIFACTS,
|
|
HYPERFRAME_RUNTIME_CONTRACT,
|
|
HYPERFRAME_RUNTIME_GLOBALS,
|
|
HYPERFRAME_BRIDGE_SOURCES,
|
|
HYPERFRAME_CONTROL_ACTIONS,
|
|
} from '@hyperframes/core';
|
|
import type {
|
|
HyperframeControlAction,
|
|
HyperframesRuntimeBuildOptions,
|
|
} from '@hyperframes/core';
|
|
|
|
// Load the pre-built runtime IIFE
|
|
const runtimeSource = loadHyperframeRuntimeSource();
|
|
|
|
// Build a custom runtime script
|
|
const script = buildHyperframesRuntimeScript(options);
|
|
```
|
|
|
|
The pre-built runtime IIFE is available as a direct import:
|
|
|
|
```typescript
|
|
import runtime from '@hyperframes/core/runtime';
|
|
```
|
|
|
|
### Preloading and decoding
|
|
|
|
The runtime eagerly prepares every asset in the composition so the first
|
|
paint and the first play call never stall the compositor.
|
|
|
|
- **Media** — every `<video>` and `<audio>` has `preload="auto"` set and
|
|
`.load()` kicked on bind. The first `play()` lands on buffered data
|
|
instead of triggering a network fetch + decoder spin-up.
|
|
- **Images** — every `<img>` gets `decoding="async"` and
|
|
`img.decode()` invoked off the main thread. Compositions with many
|
|
images (several dozen through several hundred) otherwise stutter
|
|
during the first ~second of preview because each image decodes
|
|
synchronously the first time it enters the paint tree (see
|
|
[GH #317](https://github.com/heygen-com/hyperframes/issues/317)).
|
|
|
|
Both binders are **idempotent** — the runtime re-runs them periodically
|
|
as sub-compositions mount and DOM changes land, and they skip elements
|
|
they've already processed.
|
|
|
|
## Frame Adapters
|
|
|
|
The core package defines the [Frame Adapter](/concepts/frame-adapters) interface and provides the built-in GSAP adapter:
|
|
|
|
```typescript
|
|
import { createGSAPFrameAdapter } from '@hyperframes/core';
|
|
import type {
|
|
FrameAdapter,
|
|
FrameAdapterContext,
|
|
GSAPTimelineLike,
|
|
CreateGSAPFrameAdapterOptions,
|
|
} from '@hyperframes/core';
|
|
|
|
// Create a GSAP frame adapter
|
|
const adapter: FrameAdapter = createGSAPFrameAdapter({
|
|
id: 'my-composition',
|
|
fps: 30,
|
|
timeline: gsapTimeline,
|
|
});
|
|
|
|
// Adapter lifecycle
|
|
await adapter.init?.(context);
|
|
const durationFrames = adapter.getDurationFrames();
|
|
await adapter.seekFrame(42);
|
|
await adapter.destroy?.();
|
|
```
|
|
|
|
## Media Utilities
|
|
|
|
```typescript
|
|
import {
|
|
MEDIA_VISUAL_STYLE_PROPERTIES,
|
|
copyMediaVisualStyles,
|
|
quantizeTimeToFrame,
|
|
} from '@hyperframes/core';
|
|
import type { MediaVisualStyleProperty } from '@hyperframes/core';
|
|
|
|
// Quantize a time value to the nearest frame boundary
|
|
const frameTime = quantizeTimeToFrame(5.033, 30); // → 5.033... snapped to frame
|
|
|
|
// Copy visual styles between media elements
|
|
copyMediaVisualStyles(fromElement, toElement);
|
|
```
|
|
|
|
## Picker API
|
|
|
|
For element selection in editor UIs:
|
|
|
|
```typescript
|
|
import type {
|
|
HyperframePickerApi,
|
|
HyperframePickerBoundingBox,
|
|
HyperframePickerElementInfo,
|
|
} from '@hyperframes/core';
|
|
```
|
|
|
|
## Related Packages
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="CLI" icon="terminal" href="/packages/cli">
|
|
The easiest way to create, preview, lint, and render compositions.
|
|
</Card>
|
|
<Card title="Engine" icon="gear" href="/packages/engine">
|
|
Low-level frame capture pipeline that uses core types and runtime.
|
|
</Card>
|
|
<Card title="Producer" icon="film" href="/packages/producer">
|
|
Full rendering pipeline built on top of core and engine.
|
|
</Card>
|
|
<Card title="Studio" icon="palette" href="/packages/studio">
|
|
Visual composition editor that embeds the core runtime for preview.
|
|
</Card>
|
|
</CardGroup>
|