From 7517f6ac86496fa422e1dd918482f10b3fe4a546 Mon Sep 17 00:00:00 2001
From: James Russo <james.russo@heygen.com>
Date: Thu, 25 Jun 2026 12:47:54 -0700
Subject: [PATCH] feat(slideshow): per-slide autoplay (manual-advance, opt-in)
 (#1708)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* feat(slideshow): per-slide autoplay (manual-advance, opt-in)

Adds an opt-in `autoplay` flag to slideshow slides: when the presenter lands
on a video slide, its `<video>` plays from the start. The slideshow still
holds and never auto-advances — the presenter clicks Next when ready. This
covers compositions whose own controls can't be clicked (the player renders
the composition pointer-events:none).

Plumbing (done, tested):
- core: `SlideRef.autoplay?: boolean`, parsed + validated in parseSlideshow
  (a non-boolean autoplay rejects the manifest); carried through resolve.
- controller: optional `PlayerPort.playSceneMedia(sceneId)`, fired only on
  forward `enterSlide` for autoplay slides (not resume/back/sync, so the
  audience — which mirrors the presenter's media events — isn't double-driven).
- component: `playSceneDocumentMedia` reaches the same-origin composition
  iframe, finds the scene's `<video>`, and asserts playback; `stopMedia`
  (already wired on slide change) resets it. An autoplay token cancels a
  pending start when the slide changes.
- tests: controller autoplay behavior + parser flag round-trip/validation
  (131 player + 22 core slideshow tests pass).

KNOWN LIMITATION — runtime media-start needs the player media model (@vance):
On current main the clip<->timeline binding from #1601 keeps every clip synced
and *paused* to the held timeline frame, which wins against playSceneMedia's
play() — so the clip does not actually start on main yet (it does on the
pre-#1601 player). The correct fix is a sanctioned "let this clip free-run
while the timeline holds" path in the player/runtime media controller. Flagging
for Vance to wire the start into the #1601 media model (or rebase onto it) when
back. The plumbing above is the stable surface that hook plugs into.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(slideshow): address code-review on per-slide autoplay

- guard playSceneDocumentMedia behind resolveMode() !== "audience": the
  audience mirrors the presenter's media events, so it must not independently
  drive its own copy of the clip.
- drop the per-enter window pointerdown/keydown "gesture retry" listeners,
  which leaked when muted autoplay succeeded without a gesture. The poll already
  re-asserts play(), so a gesture within the window is picked up next tick.
- stop polling once the clip is advancing across two ticks (was re-asserting
  play() for the full window even after playback was confirmed).
- cancel any in-flight autoplay loop on disconnectedCallback (bump the token).
- split the poll into findSceneVideo + stepAutoplay helpers (keeps each small).
- fix the enterSlide comment: autoplay fires from enterSlide (next/prev/
  goToSlide), not resumeSlide (back/backToMain/syncTo).
- parser: isOptionalBoolean type guard instead of a one-off helper; drop `as`
  assertions in the new controller test.

131 player + 22 core slideshow tests pass; lint/format/typecheck/fallow clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(slideshow): autoplay skill guidance + address review nits

Addresses review feedback on #1708:
- skill: document per-slide `autoplay` in the slideshow standalone-harness
  reference — when to use it (video is the slide's primary content, its end is
  the advance cue) vs not (background/ambient loops, footage talked over), per
  Vance's guidance, before merge.
- play() rejection is no longer blanket-swallowed: AbortError (timeline-sync
  seek interrupt) and NotAllowedError (gesture-gated autoplay) are expected and
  ignored; any other rejection is surfaced once via console.warn (Via nit 1).
- clarify in the SlideRef.autoplay doc that it plays the scene's FIRST <video>
  (Via nit 2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 .../core/src/slideshow/parseSlideshow.test.ts | 19 ++++
 packages/core/src/slideshow/parseSlideshow.ts | 15 ++-
 .../core/src/slideshow/slideshow.types.ts     |  8 ++
 .../src/slideshow/SlideshowController.test.ts | 45 +++++++++
 .../src/slideshow/SlideshowController.ts      |  9 ++
 .../src/slideshow/hyperframes-slideshow.ts    | 99 +++++++++++++++++++
 .../references/standalone-harness.md          | 14 +++
 7 files changed, 204 insertions(+), 5 deletions(-)

diff --git a/packages/core/src/slideshow/parseSlideshow.test.ts b/packages/core/src/slideshow/parseSlideshow.test.ts
index 1633d5881..8251a1640 100644
--- a/packages/core/src/slideshow/parseSlideshow.test.ts
+++ b/packages/core/src/slideshow/parseSlideshow.test.ts
@@ -196,4 +196,23 @@ describe("resolveSlideshow", () => {
     expect(resolved.slides[0].start).toBe(1);
     expect(resolved.slides[0].end).toBe(4);
   });
+
+  it("parses and carries through the per-slide autoplay flag", () => {
+    const island = `<script type="application/hyperframes-slideshow+json">
+      { "slides": [ { "sceneId": "a", "autoplay": true }, { "sceneId": "b" } ] }
+    </script>`;
+    const m = parseSlideshowManifest(island);
+    expect(m?.slides[0].autoplay).toBe(true);
+    expect(m?.slides[1].autoplay).toBeUndefined();
+    const { resolved } = resolveSlideshow(m!, SCENES);
+    expect(resolved.slides[0].autoplay).toBe(true);
+    expect(resolved.slides[1].autoplay).toBeUndefined();
+  });
+
+  it("rejects a manifest whose slide autoplay is not a boolean", () => {
+    const island = `<script type="application/hyperframes-slideshow+json">
+      { "slides": [ { "sceneId": "a", "autoplay": "yes" } ] }
+    </script>`;
+    expect(() => parseSlideshowManifest(island)).toThrow();
+  });
 });
diff --git a/packages/core/src/slideshow/parseSlideshow.ts b/packages/core/src/slideshow/parseSlideshow.ts
index a7583f840..a05537ce5 100644
--- a/packages/core/src/slideshow/parseSlideshow.ts
+++ b/packages/core/src/slideshow/parseSlideshow.ts
@@ -44,16 +44,21 @@ export function parseSlideshowManifest(html: string): SlideshowManifest | null {
   return parsed;
 }
 
+function isOptionalNumberArray(v: unknown): boolean {
+  return v === undefined || (Array.isArray(v) && v.every((n) => typeof n === "number"));
+}
+
+function isOptionalBoolean(v: unknown): v is boolean | undefined {
+  return v === undefined || typeof v === "boolean";
+}
+
 function isSlideRef(v: unknown): v is SlideRef {
   if (typeof v !== "object" || v === null) return false;
   const r = v as Record<string, unknown>;
   if (typeof r["sceneId"] !== "string") return false;
-  if (
-    r["fragments"] !== undefined &&
-    !(Array.isArray(r["fragments"]) && r["fragments"].every((n) => typeof n === "number"))
-  )
-    return false;
+  if (!isOptionalNumberArray(r["fragments"])) return false;
   if (r["hotspots"] !== undefined && !Array.isArray(r["hotspots"])) return false;
+  if (!isOptionalBoolean(r["autoplay"])) return false;
   return true;
 }
 
diff --git a/packages/core/src/slideshow/slideshow.types.ts b/packages/core/src/slideshow/slideshow.types.ts
index fdddf1d24..4d148d815 100644
--- a/packages/core/src/slideshow/slideshow.types.ts
+++ b/packages/core/src/slideshow/slideshow.types.ts
@@ -19,6 +19,14 @@ export interface SlideRef {
   notes?: string;
   fragments?: number[];
   hotspots?: SlideHotspot[];
+  /**
+   * When true, the slide's first `<video>` plays automatically on enter (the
+   * presenter lands on the slide and the clip plays). The slideshow still holds
+   * — it never auto-advances — so the presenter clicks Next when ready.
+   * Defaults to false. Use it when the video is the slide's primary content and
+   * its natural end is the cue to advance, not for background/ambient clips.
+   */
+  autoplay?: boolean;
   // Reserved — TTS deferred. Parsed and carried, never consumed.
   ttsScript?: string;
   ttsAudioUrl?: string;
diff --git a/packages/player/src/slideshow/SlideshowController.test.ts b/packages/player/src/slideshow/SlideshowController.test.ts
index 31b1bb037..0d59c35d7 100644
--- a/packages/player/src/slideshow/SlideshowController.test.ts
+++ b/packages/player/src/slideshow/SlideshowController.test.ts
@@ -13,6 +13,7 @@ function fakePlayer() {
     play: vi.fn(() => {}),
     pause: vi.fn(() => {}),
     stopMedia: vi.fn(() => {}),
+    playSceneMedia: vi.fn((_sceneId: string) => {}),
     onTimeUpdate: (fn: (t: number) => void) => {
       cb = fn;
       return () => {
@@ -683,3 +684,47 @@ describe("SlideshowController syncTo", () => {
     expect(c.position.slideIndex).toBe(0);
   });
 });
+
+describe("SlideshowController autoplay", () => {
+  // "v" autoplays; "w" does not. Both sit on the main line.
+  const AUTOPLAY_SHOW: ResolvedSlideshow = {
+    slides: [
+      { sceneId: "v", start: 0, end: 5, fragments: [], hotspots: [], autoplay: true },
+      { sceneId: "w", start: 5, end: 10, fragments: [], hotspots: [] },
+    ],
+    sequences: {},
+  };
+
+  it("plays the slide's media on enter when autoplay is set", () => {
+    const p = fakePlayer();
+    new SlideshowController(p, AUTOPLAY_SHOW); // constructs on slide "v"
+    expect(p.playSceneMedia).toHaveBeenCalledWith("v");
+    expect(p.playSceneMedia).toHaveBeenCalledTimes(1);
+  });
+
+  it("does not play media when entering a non-autoplay slide", () => {
+    const p = fakePlayer();
+    const c = new SlideshowController(p, AUTOPLAY_SHOW);
+    p.playSceneMedia.mockClear();
+    c.next(); // v → w (w is not autoplay)
+    expect(c.position.slideIndex).toBe(1);
+    expect(p.playSceneMedia).not.toHaveBeenCalled();
+  });
+
+  it("stops prior media and plays again when navigating back into an autoplay slide", () => {
+    const p = fakePlayer();
+    const c = new SlideshowController(p, AUTOPLAY_SHOW);
+    p.playSceneMedia.mockClear();
+    c.next(); // → w
+    expect(p.stopMedia).toHaveBeenCalled(); // leaving v stops its clip
+    p.playSceneMedia.mockClear();
+    c.prev(); // back into v (enterSlide) → replays
+    expect(p.playSceneMedia).toHaveBeenCalledWith("v");
+  });
+
+  it("does not require autoplay support on the port (optional hook)", () => {
+    // A port without playSceneMedia must not throw when entering an autoplay slide.
+    const { playSceneMedia: _omitted, ...port } = fakePlayer();
+    expect(() => new SlideshowController(port, AUTOPLAY_SHOW)).not.toThrow();
+  });
+});
diff --git a/packages/player/src/slideshow/SlideshowController.ts b/packages/player/src/slideshow/SlideshowController.ts
index 5abd1c19b..006be722d 100644
--- a/packages/player/src/slideshow/SlideshowController.ts
+++ b/packages/player/src/slideshow/SlideshowController.ts
@@ -5,6 +5,8 @@ export interface PlayerPort {
   play(): void;
   pause(): void;
   stopMedia?(): void;
+  /** Play the `<video>` inside the given scene (for `autoplay` slides). */
+  playSceneMedia?(sceneId: string): void;
   readonly currentTime: number;
   onTimeUpdate(cb: (t: number) => void): () => void;
 }
@@ -115,6 +117,13 @@ export class SlideshowController {
       this.frame.fragmentIndex = -1;
       this.playTo(this.restFrame(slide));
     }
+    // Opt-in: play the slide's own clip on enter (saves a click into the
+    // pointer-events:none composition). We never auto-advance — the presenter
+    // still clicks Next. Fires from enterSlide (next / prev / goToSlide), NOT
+    // from resumeSlide (back / backToMain / syncTo), which restores a saved
+    // position; the component also skips it on the audience, which mirrors the
+    // presenter's media events rather than driving its own.
+    if (slide.autoplay) this.player.playSceneMedia?.(slide.sceneId);
     this.emitChange();
   }
 
diff --git a/packages/player/src/slideshow/hyperframes-slideshow.ts b/packages/player/src/slideshow/hyperframes-slideshow.ts
index 46f8f6092..3bc00b766 100644
--- a/packages/player/src/slideshow/hyperframes-slideshow.ts
+++ b/packages/player/src/slideshow/hyperframes-slideshow.ts
@@ -44,6 +44,19 @@ interface SlideNotesTarget {
 
 type SlideshowManifest = NonNullable<ReturnType<typeof parseSlideshowManifest>>;
 
+// Autoplay re-assert poll (see playSceneDocumentMedia): the player drives clips
+// during bootstrap and on enter, so a single play() loses the race; we poll
+// briefly until the clip is advancing.
+const AUTOPLAY_STEP_MS = 150;
+const AUTOPLAY_MAX_MS = 6000;
+interface AutoplayPollState {
+  started: boolean;
+  lastTime: number;
+  advancingTicks: number;
+  waited: number;
+  warned: boolean;
+}
+
 type PlayerElement = HTMLElement & {
   seek(t: number): void;
   play(): void;
@@ -173,6 +186,9 @@ export class HyperframesSlideshow extends HTMLElement {
   private audienceMutedPlaybackKeys = new Set<string>();
   private blockedAudienceMedia = new Map<string, PresenterMediaMessage>();
   private audienceMediaUnlockButton: HTMLButtonElement | null = null;
+  // Bumped whenever autoplay starts or media is stopped (slide change), so a
+  // pending re-assert from a previous autoplay can't replay a clip we've left.
+  private autoplayToken = 0;
 
   /** Whether audio is currently muted. Reflects `data-hf-muted` attribute. */
   get muted(): boolean {
@@ -237,6 +253,7 @@ export class HyperframesSlideshow extends HTMLElement {
   disconnectedCallback(): void {
     this.disconnected = true;
     this.initGeneration += 1;
+    this.autoplayToken++; // cancel any in-flight autoplay re-assert loop
     if (this.initTimer !== null) {
       clearTimeout(this.initTimer);
       this.initTimer = null;
@@ -390,6 +407,7 @@ export class HyperframesSlideshow extends HTMLElement {
           playerEl.stopMedia?.();
           this.stopDocumentMedia();
         },
+        playSceneMedia: (sceneId) => this.playSceneDocumentMedia(sceneId),
         get currentTime() {
           return playerEl.currentTime;
         },
@@ -1062,12 +1080,93 @@ export class HyperframesSlideshow extends HTMLElement {
   }
 
   private stopDocumentMedia(): void {
+    // Invalidate any in-flight autoplay re-assert so leaving a slide can't be
+    // undone by a pending timeout replaying the clip we just paused.
+    this.autoplayToken++;
     const doc = this.ownerDocument;
     for (const el of doc.querySelectorAll("video, audio")) {
       if (el instanceof HTMLMediaElement) el.pause();
     }
   }
 
+  /**
+   * Play the `<video>` inside a given scene from its start — the runtime side of
+   * a slide's `autoplay`. Reaches into the same-origin composition iframe (which
+   * is pointer-events:none, so its own controls can't be clicked). The play()
+   * fires a "play" event that wireSlideshowMedia() mirrors to any audience
+   * window, so this runs on the presenter only — the audience drives its copy
+   * from those mirrored events, never on its own.
+   *
+   * Robust against two timing hazards: (1) the clip may not be in the iframe DOM
+   * yet at construction (first slide), and (2) the player drives clips during
+   * bootstrap and seeks the timeline on enter — both pause the clip (and reject
+   * an in-flight play() with AbortError), so a single play() loses the race. So
+   * we poll on a short timer: locate the clip, then assert play() until it is
+   * actually advancing across two ticks, then stop — leaving a later real user
+   * pause (presenter media controls) alone. A user gesture within the window
+   * (real browsers gate autoplay on one) lets the next tick's play() take.
+   * Token-guarded, so leaving the slide or disconnecting cancels it.
+   */
+  private playSceneDocumentMedia(sceneId: string): void {
+    if (this.resolveMode() === "audience") return;
+    const safeId = sceneId.replace(/["\\]/g, "\\$&");
+    const token = ++this.autoplayToken;
+    const state: AutoplayPollState = {
+      started: false,
+      lastTime: -1,
+      advancingTicks: 0,
+      waited: 0,
+      warned: false,
+    };
+    const tick = (): void => {
+      if (token !== this.autoplayToken) return; // left the slide / disconnected
+      const done = this.stepAutoplay(safeId, state);
+      state.waited += AUTOPLAY_STEP_MS;
+      if (!done && state.waited <= AUTOPLAY_MAX_MS) window.setTimeout(tick, AUTOPLAY_STEP_MS);
+    };
+    tick();
+  }
+
+  /** Locate the scene's clip in the composition iframe(s). */
+  private findSceneVideo(safeId: string): HTMLVideoElement | null {
+    for (const player of this.mediaPlayerElements()) {
+      const doc = this.playerFrameDocument(player);
+      const video = doc?.querySelector(`[data-composition-id="${safeId}"] video`) ?? null;
+      if (video instanceof HTMLVideoElement) return video;
+    }
+    return null;
+  }
+
+  /** One autoplay poll step. Returns true once the clip is confirmed playing. */
+  private stepAutoplay(safeId: string, state: AutoplayPollState): boolean {
+    const video = this.findSceneVideo(safeId);
+    if (!video) return false;
+    if (!state.started) {
+      state.started = true;
+      video.muted = this._muted || video.defaultMuted;
+      try {
+        video.currentTime = 0;
+      } catch {
+        // not seekable yet — play from wherever it is
+      }
+    }
+    const advancing = !video.paused && video.currentTime > state.lastTime;
+    state.lastTime = video.currentTime;
+    if (advancing) return ++state.advancingTicks >= 2; // confirmed playing — stop polling
+    state.advancingTicks = 0;
+    void video.play().catch((err: unknown) => {
+      // Expected during the poll: AbortError (a timeline-sync seek interrupts
+      // the play) and NotAllowedError (autoplay gated on a user gesture). Surface
+      // anything else once — a real failure (bad src, decode) shouldn't be silent.
+      const name = err instanceof DOMException ? err.name : "";
+      if (name !== "AbortError" && name !== "NotAllowedError" && !state.warned) {
+        state.warned = true;
+        console.warn("[hyperframes-slideshow] autoplay play() failed:", err);
+      }
+    });
+    return false;
+  }
+
   private presenterNotesDeckKey(): string {
     const explicit = this.getAttribute("notes-storage-key")?.trim();
     if (explicit) return explicit;
diff --git a/skills/slideshow/references/standalone-harness.md b/skills/slideshow/references/standalone-harness.md
index f42c8ce17..e24306cdd 100644
--- a/skills/slideshow/references/standalone-harness.md
+++ b/skills/slideshow/references/standalone-harness.md
@@ -109,6 +109,20 @@ For public or user-facing generated projects, make this wrapper the root `index.
 
 `interactive` is required for decks with clickable page content or media controls. Without it, iframe pointer events are disabled by the player shell and a click on the composition can be interpreted as a player play/pause toggle instead of a slide interaction.
 
+### Per-slide `autoplay`
+
+Add `"autoplay": true` to a slide in the island to play that slide's first `<video>` from the start when the presenter lands on it. The slideshow still holds — it never auto-advances — so the presenter clicks Next when ready; autoplay only saves a manual play click into the composition.
+
+```json
+{
+  "sceneId": "cold-open",
+  "autoplay": true,
+  "notes": "Promo plays on enter; click Next when it ends."
+}
+```
+
+Use `autoplay` when the video **is** the slide's primary content and its natural end is the cue to advance — a cold-open promo, a demo clip you let run and then move on from. Do **not** use it for background/ambient loops or for footage the presenter talks over; those should start on the presenter's own cue (a click on the clip's controls via `interactive`), not automatically. One autoplay clip per slide (the first `<video>` in the scene).
+
 ### Presenter media bridge for interactive media
 
 Presenter/audience mode syncs slide position through a deck-scoped `BroadcastChannel`. If the presenter is expected to play, pause, seek, mute, or change rate on media inside the composition, mirror those native media events over the same channel. Keep the media element as the source of truth; do not mirror a custom button's private state.