Files
hyperframes/evals/editframe/reports/devbox-2026-04-30.html

526 lines
18 KiB
HTML

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>HyperFrames vs Editframe Devbox Eval</title>
<style>
:root {
color-scheme: light;
--ink: #111;
--muted: #5b6168;
--line: #d9d9d2;
--paper: #f7f5ed;
--panel: #fffdf8;
--accent: #0d6b68;
--warn: #9f5b00;
--bad: #a11;
--good: #146c2e;
}
* {
box-sizing: border-box;
}
body {
margin: 0;
font-family:
Inter,
ui-sans-serif,
system-ui,
-apple-system,
BlinkMacSystemFont,
"Segoe UI",
sans-serif;
background: var(--paper);
color: var(--ink);
line-height: 1.5;
}
main {
max-width: 1180px;
margin: 0 auto;
padding: 48px 28px 80px;
}
h1 {
font-size: clamp(38px, 6vw, 76px);
line-height: 0.95;
margin: 0 0 18px;
letter-spacing: 0;
}
h2 {
margin: 48px 0 14px;
font-size: 28px;
}
h3 {
margin: 28px 0 10px;
}
.lede {
font-size: 20px;
color: var(--muted);
max-width: 900px;
}
.grid {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 18px;
}
.card {
background: var(--panel);
border: 1px solid var(--line);
border-radius: 8px;
padding: 18px;
}
.metric {
font-size: 34px;
font-weight: 800;
}
.label {
color: var(--muted);
font-size: 14px;
text-transform: uppercase;
letter-spacing: 0.04em;
}
table {
width: 100%;
border-collapse: collapse;
background: var(--panel);
border: 1px solid var(--line);
}
th,
td {
text-align: left;
border-bottom: 1px solid var(--line);
padding: 10px 12px;
vertical-align: top;
}
th {
font-size: 13px;
text-transform: uppercase;
color: var(--muted);
}
.pill {
display: inline-block;
padding: 3px 8px;
border-radius: 999px;
font-size: 12px;
font-weight: 700;
background: #eee;
}
.HF-edge {
background: #e6f4ed;
color: var(--good);
}
.EF-conditional-edge,
.EF-source-doc-edge {
background: #fff2d8;
color: var(--warn);
}
.split {
background: #edf1ff;
color: #26408b;
}
.not_final,
.not_tested {
background: #eee;
color: #555;
}
pre {
white-space: pre-wrap;
background: #151515;
color: #eee;
padding: 14px;
border-radius: 8px;
overflow: auto;
}
a {
color: var(--accent);
}
video {
width: 100%;
border: 1px solid var(--line);
border-radius: 8px;
background: #000;
}
.shots {
display: grid;
grid-template-columns: repeat(4, minmax(0, 1fr));
gap: 10px;
}
figure {
margin: 0;
}
img {
width: 100%;
display: block;
border: 1px solid var(--line);
border-radius: 6px;
background: #111;
}
figcaption {
color: var(--muted);
font-size: 12px;
margin-top: 4px;
overflow-wrap: anywhere;
}
.callout {
border-left: 5px solid var(--accent);
background: #eef8f5;
padding: 16px 18px;
border-radius: 6px;
}
@media (max-width: 820px) {
.grid,
.shots {
grid-template-columns: 1fr;
}
}
</style>
</head>
<body>
<main>
<h1>HyperFrames vs Editframe Devbox Eval</h1>
<p class="lede">
Execution date: 2026-04-30 on <code>ip-10-0-9-220</code>. This report evaluates one
precise-timing fixture plus public/docs/API evidence. It is not a blind taste benchmark.
</p>
<section class="callout">
<strong>Bottom line:</strong> HyperFrames completed the default local H.264 MP4 render path
on this devbox. Editframe's default local CLI MP4 render failed because this
Chrome/WebCodecs build does not support the selected AVC encoder config, but the same
Editframe fixture rendered successfully through its browser API when VP9 was selected
programmatically.
</section>
<h2>Automated Render Results</h2>
<div class="grid">
<div class="card">
<div class="label">HyperFrames</div>
<div class="metric">Pass</div>
<p>
Published <code>hyperframes@latest</code> linted, validated, snapshotted, and rendered a
10s 1920x1080/30fps H.264 MP4.
</p>
<ul>
<li>Command elapsed: 5971ms</li>
<li>Duration: 10.000000s</li>
<li>Size: 700 KB</li>
<li>Codec: h264</li>
</ul>
<p>
<a href="../artifacts/devbox-2026-04-30/hyperframes/renders/render.log">render log</a> ·
<a href="../artifacts/devbox-2026-04-30/hyperframes/renders/ffprobe.json">ffprobe</a>
</p>
</div>
<div class="card">
<div class="label">Editframe</div>
<div class="metric">Mixed</div>
<p>
Published <code>@editframe/cli@0.49.7</code> default AVC render failed. Browser-driven
<code>EF_RENDER.renderStreaming({ codec: "vp9" })</code> produced a 10s 1920x1080/30fps
MP4 containing VP9 video.
</p>
<ul>
<li>Browser VP9 elapsed: 1280ms</li>
<li>Duration: 10.000000s</li>
<li>Size: 421 KB</li>
<li>Codec: vp9</li>
</ul>
<p>
<a href="../artifacts/devbox-2026-04-30/editframe/renders/render-cli-avc.log"
>failed CLI log</a
>
·
<a href="../artifacts/devbox-2026-04-30/editframe/renders/ffprobe-vp9.json"
>VP9 ffprobe</a
>
</p>
</div>
</div>
<h2>Rubric Findings</h2>
<table>
<thead>
<tr>
<th>Category</th>
<th>Weight</th>
<th>Finding</th>
<th>Evidence</th>
</tr>
</thead>
<tbody>
<tr>
<td>Visual output quality</td>
<td>18%</td>
<td><span class="pill not_final">not final</span></td>
<td>
Not blindly judged. Both controlled fixtures produced screenshots; this run should not
decide taste/video quality.
</td>
</tr>
<tr>
<td>Timing and motion control</td>
<td>14%</td>
<td><span class="pill HF-edge">HF edge</span></td>
<td>
Both can seek exact timestamps. HyperFrames validated and snapshotted the GSAP
timeline through first-party CLI; Editframe snapshotted via ef-timegroup.seek and
rendered through browser API.
</td>
</tr>
<tr>
<td>Timeline and editability</td>
<td>12%</td>
<td><span class="pill split">split</span></td>
<td>
Editframe has richer public editor primitives; HyperFrames source remains plain
HTML/data attributes and passed lint/validate.
</td>
</tr>
<tr>
<td>Agent success rate</td>
<td>12%</td>
<td><span class="pill HF-edge">HF edge</span></td>
<td>
HyperFrames fresh init plus lint/validate/snapshot/render passed. Editframe fresh init
passed, but advertised --skip-skills flag failed and default CLI render failed on AVC.
</td>
</tr>
<tr>
<td>Media handling</td>
<td>10%</td>
<td><span class="pill not_tested">not tested</span></td>
<td>This devbox run did not exercise external media, audio, captions, or alpha.</td>
</tr>
<tr>
<td>Render fidelity</td>
<td>10%</td>
<td><span class="pill HF-edge">HF edge</span></td>
<td>
HyperFrames produced default H.264 MP4 at 1920x1080/30fps/10s. Editframe CLI default
failed; browser VP9 fallback produced valid 1920x1080/30fps/10s MP4.
</td>
</tr>
<tr>
<td>Performance and cost</td>
<td>8%</td>
<td><span class="pill EF-conditional-edge">EF conditional edge</span></td>
<td>
Editframe VP9 browser render elapsed 1280ms; HyperFrames draft H.264 render command
elapsed 5971ms. Codecs/render paths differ, so this is directional only.
</td>
</tr>
<tr>
<td>Observability and debugging</td>
<td>7%</td>
<td><span class="pill HF-edge">HF edge</span></td>
<td>
HyperFrames surfaced lint/contrast checks and progress. Editframe error was clear, but
the CLI left a 28-byte output file and exposes no codec flag.
</td>
</tr>
<tr>
<td>Deployment and API ergonomics</td>
<td>5%</td>
<td><span class="pill EF-source-doc-edge">EF source-doc edge</span></td>
<td>
Editframe public docs include cloud/API render jobs, file processing, webhooks, and
CPU/GPU backend selection. This run did not authenticate to test it live.
</td>
</tr>
<tr>
<td>Human iteration workflow</td>
<td>4%</td>
<td><span class="pill not_final">not final</span></td>
<td>
Not exercised with human editors. Public docs suggest both have timeline/editor
workflows; Editframe has broader standalone editor widgets.
</td>
</tr>
</tbody>
</table>
<h2>Codec Probe</h2>
<p>
Editframe's browser renderer depends on WebCodecs encoder support. On this devbox Chrome
supports VP9/VP8/AV1 but not the AVC config selected by the CLI.
</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Codec string</th>
<th>Support</th>
</tr>
</thead>
<tbody>
<tr>
<td>avc</td>
<td>avc1.640028</td>
<td>not supported</td>
</tr>
<tr>
<td>vp9</td>
<td>vp09.00.10.08</td>
<td>supported</td>
</tr>
<tr>
<td>vp8</td>
<td>vp8</td>
<td>supported</td>
</tr>
<tr>
<td>av1</td>
<td>av01.0.05M.08</td>
<td>supported</td>
</tr>
</tbody>
</table>
<h2>Videos</h2>
<div class="grid">
<div>
<h3>HyperFrames H.264 MP4</h3>
<video
controls
src="../artifacts/devbox-2026-04-30/hyperframes/renders/hyperframes-precise.mp4"
></video>
</div>
<div>
<h3>Editframe Browser VP9 MP4</h3>
<video
controls
src="../artifacts/devbox-2026-04-30/editframe/renders/editframe-browser-vp9.mp4"
></video>
</div>
</div>
<h2>Timestamp Snapshots</h2>
<h3>HyperFrames snapshots</h3>
<div class="shots">
<figure>
<img src="../artifacts/devbox-2026-04-30/hyperframes/snapshots/frame-00-at-1.2s.png" />
<figcaption>frame-00-at-1.2s.png</figcaption>
</figure>
<figure>
<img src="../artifacts/devbox-2026-04-30/hyperframes/snapshots/frame-01-at-5.0s.png" />
<figcaption>frame-01-at-5.0s.png</figcaption>
</figure>
<figure>
<img src="../artifacts/devbox-2026-04-30/hyperframes/snapshots/frame-02-at-8.5s.png" />
<figcaption>frame-02-at-8.5s.png</figcaption>
</figure>
<figure>
<img src="../artifacts/devbox-2026-04-30/hyperframes/snapshots/frame-03-at-9.4s.png" />
<figcaption>frame-03-at-9.4s.png</figcaption>
</figure>
</div>
<h3>Editframe snapshots</h3>
<div class="shots">
<figure>
<img src="../artifacts/devbox-2026-04-30/editframe/snapshots/editframe-1200ms.png" />
<figcaption>editframe-1200ms.png</figcaption>
</figure>
<figure>
<img src="../artifacts/devbox-2026-04-30/editframe/snapshots/editframe-5000ms.png" />
<figcaption>editframe-5000ms.png</figcaption>
</figure>
<figure>
<img src="../artifacts/devbox-2026-04-30/editframe/snapshots/editframe-8500ms.png" />
<figcaption>editframe-8500ms.png</figcaption>
</figure>
<figure>
<img src="../artifacts/devbox-2026-04-30/editframe/snapshots/editframe-9400ms.png" />
<figcaption>editframe-9400ms.png</figcaption>
</figure>
</div>
<h2>Important Failure Evidence</h2>
<h3>Editframe default CLI render failure</h3>
<pre>
- Initializing...
✔ Ready
- Rendering video...
✖ Render failed
node:internal/process/promises:394
triggerUncaughtException(err, true /* fromPromise */);
^
page.evaluate: Error: This specific encoder configuration (avc1.640028, 8000000 bps, 1920x1080, hardware acceleration: no-preference) is not supported by this browser. Consider using another codec or changing your video parameters.
at http://localhost:5176/node_modules/.vite/deps/chunk-M5YSC7MS.js?v=622dd3ba:23270:17
at async VideoEncoderWrapper.processAndEncode (http://localhost:5176/node_modules/.vite/deps/chunk-M5YSC7MS.js?v=622dd3ba:23120:13)
at async VideoEncoderWrapper.add (http://localhost:5176/node_modules/.vite/deps/chunk-M5YSC7MS.js?v=622dd3ba:23073:7)
at async renderTimegroupToVideo (http://localhost:5176/node_modules/.vite/deps/chunk-67JHYUZS.js?v=622dd3ba:330:9)
at async Object.renderStreaming (http://localhost:5176/node_modules/.vite/deps/@editframe_elements.js?v=622dd3ba:34967:7)
at async eval (eval at evaluate (:302:30), &lt;anonymous&gt;:2:7)
at async &lt;anonymous&gt;:328:30
at /tmp/hf-editframe-eval-run/editframe-smoke/node_modules/@editframe/cli/dist/commands/render.js:132:17
at async withProfiling (/tmp/hf-editframe-eval-run/editframe-smoke/node_modules/@editframe/cli/dist/utils/profileRender.js:5:31)</pre
>
<h3>Editframe creator flag mismatch</h3>
<p>
The help text advertises <code>--skip-skills</code>, but the parser rejects it in this
published version.
</p>
<pre>
&gt; npx
&gt; create-editframe html -d should-fail -y --skip-skills
node:internal/util/parse_args/parse_args:102
throw new ERR_PARSE_ARGS_UNKNOWN_OPTION(
^
TypeError [ERR_PARSE_ARGS_UNKNOWN_OPTION]: Unknown option '--skip-skills'. To specify a positional argument starting with a '-', place it at the end of the command after '--', as in '-- &quot;--skip-skills&quot;
at checkOptionUsage (node:internal/util/parse_args/parse_args:102:13)
at node:internal/util/parse_args/parse_args:376:9
at Array.forEach (&lt;anonymous&gt;)
at parseArgs (node:internal/util/parse_args/parse_args:373:3)
at main (file:///tmp/hf-eval-npm-cache/_npx/a680ecfe04f5894d/node_modules/@editframe/create/dist/index.js:104:34) {
code: 'ERR_PARSE_ARGS_UNKNOWN_OPTION'
}
Node.js v22.22.2
npm error code 1
npm error path /tmp/hf-editframe-eval-run
npm error command failed
npm error command sh -c create-editframe html -d should-fail -y --skip-skills
npm error A complete log of this run can be found in: /tmp/hf-eval-npm-cache/_logs/2026-04-30T19_01_19_210Z-debug-0.log
</pre
>
<h2>Conclusions</h2>
<ol>
<li>
<strong
>For devbox-local MP4 rendering, HyperFrames is currently the safer default.</strong
>
It completed the exact CLI path users would run and produced a standard H.264 MP4.
</li>
<li>
<strong
>Editframe has a strong composition/editor architecture, but local rendering is
codec-sensitive.</strong
>
Its browser API rendered fast with VP9, but the published CLI does not expose a codec
override and failed on AVC here.
</li>
<li>
<strong>Editframe's public API/cloud story is stronger on paper.</strong> This run did not
authenticate to cloud render, so that remains source-evidence only, not live proof.
</li>
<li>
<strong>Do not use this report to claim better final video taste.</strong> The fixture was
intentionally controlled and no blind judges scored visual quality.
</li>
<li>
<strong>The eval suite should keep both tracks:</strong> default user path, and
expert/fallback path. A tool that can render only after hidden codec intervention should
not get the same reliability score as one whose default command works.
</li>
</ol>
<h2>Artifacts</h2>
<ul>
<li>
<a href="../artifacts/devbox-2026-04-30/hyperframes/index.html"
>HyperFrames fixture source</a
>
</li>
<li>
<a href="../artifacts/devbox-2026-04-30/editframe/index.html">Editframe fixture source</a>
</li>
<li><a href="devbox-2026-04-30-results.json">Machine-readable results JSON</a></li>
</ul>
</main>
</body>
</html>