feat(slideshow): per-slide autoplay (manual-advance, opt-in) (#1708)

* feat(slideshow): per-slide autoplay (manual-advance, opt-in) Adds an opt-in `autoplay` flag to slideshow slides: when the presenter lands on a video slide, its `<video>` plays from the start. The slideshow still holds and never auto-advances — the presenter clicks Next when ready. This covers compositions whose own controls can't be clicked (the player renders the composition pointer-events:none). Plumbing (done, tested): - core: `SlideRef.autoplay?: boolean`, parsed + validated in parseSlideshow (a non-boolean autoplay rejects the manifest); carried through resolve. - controller: optional `PlayerPort.playSceneMedia(sceneId)`, fired only on forward `enterSlide` for autoplay slides (not resume/back/sync, so the audience — which mirrors the presenter's media events — isn't double-driven). - component: `playSceneDocumentMedia` reaches the same-origin composition iframe, finds the scene's `<video>`, and asserts playback; `stopMedia` (already wired on slide change) resets it. An autoplay token cancels a pending start when the slide changes. - tests: controller autoplay behavior + parser flag round-trip/validation (131 player + 22 core slideshow tests pass). KNOWN LIMITATION — runtime media-start needs the player media model (@vance): On current main the clip<->timeline binding from #1601 keeps every clip synced and *paused* to the held timeline frame, which wins against playSceneMedia's play() — so the clip does not actually start on main yet (it does on the pre-#1601 player). The correct fix is a sanctioned "let this clip free-run while the timeline holds" path in the player/runtime media controller. Flagging for Vance to wire the start into the #1601 media model (or rebase onto it) when back. The plumbing above is the stable surface that hook plugs into. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(slideshow): address code-review on per-slide autoplay - guard playSceneDocumentMedia behind resolveMode() !== "audience": the audience mirrors the presenter's media events, so it must not independently drive its own copy of the clip. - drop the per-enter window pointerdown/keydown "gesture retry" listeners, which leaked when muted autoplay succeeded without a gesture. The poll already re-asserts play(), so a gesture within the window is picked up next tick. - stop polling once the clip is advancing across two ticks (was re-asserting play() for the full window even after playback was confirmed). - cancel any in-flight autoplay loop on disconnectedCallback (bump the token). - split the poll into findSceneVideo + stepAutoplay helpers (keeps each small). - fix the enterSlide comment: autoplay fires from enterSlide (next/prev/ goToSlide), not resumeSlide (back/backToMain/syncTo). - parser: isOptionalBoolean type guard instead of a one-off helper; drop `as` assertions in the new controller test. 131 player + 22 core slideshow tests pass; lint/format/typecheck/fallow clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(slideshow): autoplay skill guidance + address review nits Addresses review feedback on #1708: - skill: document per-slide `autoplay` in the slideshow standalone-harness reference — when to use it (video is the slide's primary content, its end is the advance cue) vs not (background/ambient loops, footage talked over), per Vance's guidance, before merge. - play() rejection is no longer blanket-swallowed: AbortError (timeline-sync seek interrupt) and NotAllowedError (gesture-gated autoplay) are expected and ignored; any other rejection is surfaced once via console.warn (Via nit 1). - clarify in the SlideRef.autoplay doc that it plays the scene's FIRST <video> (Via nit 2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-09-03 04:38:33 +00:00 · 2026-06-25 12:47:54 -07:00
parent 041f2fa196
commit 7517f6ac86
7 changed files with 204 additions and 5 deletions
@@ -196,4 +196,23 @@ describe("resolveSlideshow", () => {
    expect(resolved.slides[0].start).toBe(1);
    expect(resolved.slides[0].end).toBe(4);
  });
+
+  it("parses and carries through the per-slide autoplay flag", () => {
+    const island = `<script type="application/hyperframes-slideshow+json">
+      { "slides": [ { "sceneId": "a", "autoplay": true }, { "sceneId": "b" } ] }
+    </script>`;
+    const m = parseSlideshowManifest(island);
+    expect(m?.slides[0].autoplay).toBe(true);
+    expect(m?.slides[1].autoplay).toBeUndefined();
+    const { resolved } = resolveSlideshow(m!, SCENES);
+    expect(resolved.slides[0].autoplay).toBe(true);
+    expect(resolved.slides[1].autoplay).toBeUndefined();
+  });
+
+  it("rejects a manifest whose slide autoplay is not a boolean", () => {
+    const island = `<script type="application/hyperframes-slideshow+json">
+      { "slides": [ { "sceneId": "a", "autoplay": "yes" } ] }
+    </script>`;
+    expect(() => parseSlideshowManifest(island)).toThrow();
+  });
 });
@@ -44,16 +44,21 @@ export function parseSlideshowManifest(html: string): SlideshowManifest | null {
  return parsed;
 }

+function isOptionalNumberArray(v: unknown): boolean {
+  return v === undefined || (Array.isArray(v) && v.every((n) => typeof n === "number"));
+}
+
+function isOptionalBoolean(v: unknown): v is boolean | undefined {
+  return v === undefined || typeof v === "boolean";
+}
+
 function isSlideRef(v: unknown): v is SlideRef {
  if (typeof v !== "object" || v === null) return false;
  const r = v as Record<string, unknown>;
  if (typeof r["sceneId"] !== "string") return false;
-  if (
-    r["fragments"] !== undefined &&
-    !(Array.isArray(r["fragments"]) && r["fragments"].every((n) => typeof n === "number"))
-  )
-    return false;
+  if (!isOptionalNumberArray(r["fragments"])) return false;
  if (r["hotspots"] !== undefined && !Array.isArray(r["hotspots"])) return false;
+  if (!isOptionalBoolean(r["autoplay"])) return false;
  return true;
 }

@@ -19,6 +19,14 @@ export interface SlideRef {
  notes?: string;
  fragments?: number[];
  hotspots?: SlideHotspot[];
+  /**
+   * When true, the slide's first `<video>` plays automatically on enter (the
+   * presenter lands on the slide and the clip plays). The slideshow still holds
+   * — it never auto-advances — so the presenter clicks Next when ready.
+   * Defaults to false. Use it when the video is the slide's primary content and
+   * its natural end is the cue to advance, not for background/ambient clips.
+   */
+  autoplay?: boolean;
  // Reserved — TTS deferred. Parsed and carried, never consumed.
  ttsScript?: string;
  ttsAudioUrl?: string;
@@ -13,6 +13,7 @@ function fakePlayer() {
    play: vi.fn(() => {}),
    pause: vi.fn(() => {}),
    stopMedia: vi.fn(() => {}),
+    playSceneMedia: vi.fn((_sceneId: string) => {}),
    onTimeUpdate: (fn: (t: number) => void) => {
      cb = fn;
      return () => {
@@ -683,3 +684,47 @@ describe("SlideshowController syncTo", () => {
    expect(c.position.slideIndex).toBe(0);
  });
 });
+
+describe("SlideshowController autoplay", () => {
+  // "v" autoplays; "w" does not. Both sit on the main line.
+  const AUTOPLAY_SHOW: ResolvedSlideshow = {
+    slides: [
+      { sceneId: "v", start: 0, end: 5, fragments: [], hotspots: [], autoplay: true },
+      { sceneId: "w", start: 5, end: 10, fragments: [], hotspots: [] },
+    ],
+    sequences: {},
+  };
+
+  it("plays the slide's media on enter when autoplay is set", () => {
+    const p = fakePlayer();
+    new SlideshowController(p, AUTOPLAY_SHOW); // constructs on slide "v"
+    expect(p.playSceneMedia).toHaveBeenCalledWith("v");
+    expect(p.playSceneMedia).toHaveBeenCalledTimes(1);
+  });
+
+  it("does not play media when entering a non-autoplay slide", () => {
+    const p = fakePlayer();
+    const c = new SlideshowController(p, AUTOPLAY_SHOW);
+    p.playSceneMedia.mockClear();
+    c.next(); // v → w (w is not autoplay)
+    expect(c.position.slideIndex).toBe(1);
+    expect(p.playSceneMedia).not.toHaveBeenCalled();
+  });
+
+  it("stops prior media and plays again when navigating back into an autoplay slide", () => {
+    const p = fakePlayer();
+    const c = new SlideshowController(p, AUTOPLAY_SHOW);
+    p.playSceneMedia.mockClear();
+    c.next(); // → w
+    expect(p.stopMedia).toHaveBeenCalled(); // leaving v stops its clip
+    p.playSceneMedia.mockClear();
+    c.prev(); // back into v (enterSlide) → replays
+    expect(p.playSceneMedia).toHaveBeenCalledWith("v");
+  });
+
+  it("does not require autoplay support on the port (optional hook)", () => {
+    // A port without playSceneMedia must not throw when entering an autoplay slide.
+    const { playSceneMedia: _omitted, ...port } = fakePlayer();
+    expect(() => new SlideshowController(port, AUTOPLAY_SHOW)).not.toThrow();
+  });
+});
@@ -5,6 +5,8 @@ export interface PlayerPort {
  play(): void;
  pause(): void;
  stopMedia?(): void;
+  /** Play the `<video>` inside the given scene (for `autoplay` slides). */
+  playSceneMedia?(sceneId: string): void;
  readonly currentTime: number;
  onTimeUpdate(cb: (t: number) => void): () => void;
 }
@@ -115,6 +117,13 @@ export class SlideshowController {
      this.frame.fragmentIndex = -1;
      this.playTo(this.restFrame(slide));
    }
+    // Opt-in: play the slide's own clip on enter (saves a click into the
+    // pointer-events:none composition). We never auto-advance — the presenter
+    // still clicks Next. Fires from enterSlide (next / prev / goToSlide), NOT
+    // from resumeSlide (back / backToMain / syncTo), which restores a saved
+    // position; the component also skips it on the audience, which mirrors the
+    // presenter's media events rather than driving its own.
+    if (slide.autoplay) this.player.playSceneMedia?.(slide.sceneId);
    this.emitChange();
  }

@@ -44,6 +44,19 @@ interface SlideNotesTarget {

 type SlideshowManifest = NonNullable<ReturnType<typeof parseSlideshowManifest>>;

+// Autoplay re-assert poll (see playSceneDocumentMedia): the player drives clips
+// during bootstrap and on enter, so a single play() loses the race; we poll
+// briefly until the clip is advancing.
+const AUTOPLAY_STEP_MS = 150;
+const AUTOPLAY_MAX_MS = 6000;
+interface AutoplayPollState {
+  started: boolean;
+  lastTime: number;
+  advancingTicks: number;
+  waited: number;
+  warned: boolean;
+}
+
 type PlayerElement = HTMLElement & {
  seek(t: number): void;
  play(): void;
@@ -173,6 +186,9 @@ export class HyperframesSlideshow extends HTMLElement {
  private audienceMutedPlaybackKeys = new Set<string>();
  private blockedAudienceMedia = new Map<string, PresenterMediaMessage>();
  private audienceMediaUnlockButton: HTMLButtonElement | null = null;
+  // Bumped whenever autoplay starts or media is stopped (slide change), so a
+  // pending re-assert from a previous autoplay can't replay a clip we've left.
+  private autoplayToken = 0;

  /** Whether audio is currently muted. Reflects `data-hf-muted` attribute. */
  get muted(): boolean {
@@ -237,6 +253,7 @@ export class HyperframesSlideshow extends HTMLElement {
  disconnectedCallback(): void {
    this.disconnected = true;
    this.initGeneration += 1;
+    this.autoplayToken++; // cancel any in-flight autoplay re-assert loop
    if (this.initTimer !== null) {
      clearTimeout(this.initTimer);
      this.initTimer = null;
@@ -390,6 +407,7 @@ export class HyperframesSlideshow extends HTMLElement {
          playerEl.stopMedia?.();
          this.stopDocumentMedia();
        },
+        playSceneMedia: (sceneId) => this.playSceneDocumentMedia(sceneId),
        get currentTime() {
          return playerEl.currentTime;
        },
@@ -1062,12 +1080,93 @@ export class HyperframesSlideshow extends HTMLElement {
  }

  private stopDocumentMedia(): void {
+    // Invalidate any in-flight autoplay re-assert so leaving a slide can't be
+    // undone by a pending timeout replaying the clip we just paused.
+    this.autoplayToken++;
    const doc = this.ownerDocument;
    for (const el of doc.querySelectorAll("video, audio")) {
      if (el instanceof HTMLMediaElement) el.pause();
    }
  }

+  /**
+   * Play the `<video>` inside a given scene from its start — the runtime side of
+   * a slide's `autoplay`. Reaches into the same-origin composition iframe (which
+   * is pointer-events:none, so its own controls can't be clicked). The play()
+   * fires a "play" event that wireSlideshowMedia() mirrors to any audience
+   * window, so this runs on the presenter only — the audience drives its copy
+   * from those mirrored events, never on its own.
+   *
+   * Robust against two timing hazards: (1) the clip may not be in the iframe DOM
+   * yet at construction (first slide), and (2) the player drives clips during
+   * bootstrap and seeks the timeline on enter — both pause the clip (and reject
+   * an in-flight play() with AbortError), so a single play() loses the race. So
+   * we poll on a short timer: locate the clip, then assert play() until it is
+   * actually advancing across two ticks, then stop — leaving a later real user
+   * pause (presenter media controls) alone. A user gesture within the window
+   * (real browsers gate autoplay on one) lets the next tick's play() take.
+   * Token-guarded, so leaving the slide or disconnecting cancels it.
+   */
+  private playSceneDocumentMedia(sceneId: string): void {
+    if (this.resolveMode() === "audience") return;
+    const safeId = sceneId.replace(/["\\]/g, "\\$&");
+    const token = ++this.autoplayToken;
+    const state: AutoplayPollState = {
+      started: false,
+      lastTime: -1,
+      advancingTicks: 0,
+      waited: 0,
+      warned: false,
+    };
+    const tick = (): void => {
+      if (token !== this.autoplayToken) return; // left the slide / disconnected
+      const done = this.stepAutoplay(safeId, state);
+      state.waited += AUTOPLAY_STEP_MS;
+      if (!done && state.waited <= AUTOPLAY_MAX_MS) window.setTimeout(tick, AUTOPLAY_STEP_MS);
+    };
+    tick();
+  }
+
+  /** Locate the scene's clip in the composition iframe(s). */
+  private findSceneVideo(safeId: string): HTMLVideoElement | null {
+    for (const player of this.mediaPlayerElements()) {
+      const doc = this.playerFrameDocument(player);
+      const video = doc?.querySelector(`[data-composition-id="${safeId}"] video`) ?? null;
+      if (video instanceof HTMLVideoElement) return video;
+    }
+    return null;
+  }
+
+  /** One autoplay poll step. Returns true once the clip is confirmed playing. */
+  private stepAutoplay(safeId: string, state: AutoplayPollState): boolean {
+    const video = this.findSceneVideo(safeId);
+    if (!video) return false;
+    if (!state.started) {
+      state.started = true;
+      video.muted = this._muted || video.defaultMuted;
+      try {
+        video.currentTime = 0;
+      } catch {
+        // not seekable yet — play from wherever it is
+      }
+    }
+    const advancing = !video.paused && video.currentTime > state.lastTime;
+    state.lastTime = video.currentTime;
+    if (advancing) return ++state.advancingTicks >= 2; // confirmed playing — stop polling
+    state.advancingTicks = 0;
+    void video.play().catch((err: unknown) => {
+      // Expected during the poll: AbortError (a timeline-sync seek interrupts
+      // the play) and NotAllowedError (autoplay gated on a user gesture). Surface
+      // anything else once — a real failure (bad src, decode) shouldn't be silent.
+      const name = err instanceof DOMException ? err.name : "";
+      if (name !== "AbortError" && name !== "NotAllowedError" && !state.warned) {
+        state.warned = true;
+        console.warn("[hyperframes-slideshow] autoplay play() failed:", err);
+      }
+    });
+    return false;
+  }
+
  private presenterNotesDeckKey(): string {
    const explicit = this.getAttribute("notes-storage-key")?.trim();
    if (explicit) return explicit;