Reference6 min read4 sections

Graphics-device recovery

docs/GRAPHICS_RECOVERY.mdSource updated 2026-10-24
Markdown
On this page 4 sections

An unexpected WebGPU device loss now pauses the player's input and audio, releases pointer capture, and shows “Rekindling the light.” The simulation worker, room connection, authority identity and local save remain alive. GPU validation errors still use the explicit failure screen; they are not retried as device loss.

#Replacement boundary

The lost renderer disposes its owned GPU resources before notifying the player. The player requests a fresh adapter/device and rebuilds the default engine pipelines and render graph. It restores the current quality setting and requests a complete presentation baseline through the existing acknowledged worker channel. Frames continue carrying simulation and shared-room state during allocation, without retaining a second message queue.

The replacement cannot be accepted until a full baseline is applied, the worker acknowledges the pause, and a rendered GPU submission completes. render() returns whether it actually submitted; a busy or unavailable renderer cannot start the acceptance wait. The scene baseline comes from the continuing canonical ECS, so shared changes received during restoration are included. Camera, player state and room identity are not recreated. Successful recovery returns gameplay to the pause menu and requires a gesture to resume; welcome and existing modal views are restored when applicable. Input stays released. The motor retains its ordinary velocity state and braking behavior after resuming.

The RendererRecovery controller permits one allocation/preparation operation at a time, two replacements per page session, and a 20-second deadline covering both allocation and preparation. Timeout, a second device loss during replacement, creation failure or failed preparation parks the attempt and presents a retry-by-reload screen. A late native factory result is disposed and cannot publish. A timed-out operation retains the allocation slot until it settles; no further allocation is admitted after failure. Page teardown aborts preparation and disposes an unaccepted candidate. Accepted resources transfer ownership to the player.

#Diagnostics and acceptance

The F2 panel records the recovery count; playtest exports include controller state and budgets. Browser-only test hooks intercept WebGPU device acquisition and the existing worker without adding a production debug endpoint. Tests destroy actual devices, delay replacement acquisition, verify visible scene pixels after recovery, repeat loss to the session limit, and inject replacement-device rejection. Shared-room coverage changes a door from a second client during restoration and verifies the original worker, admission count, self identity, host and authority epoch remain unchanged. After restoration, the original client can change the shared door again.

Headless tests verify ownership transfer, attempt limits, overlapping-allocation rejection, creation/preparation deadlines, cancellation, late completion and exactly-once candidate disposal. These tests establish bounded application behavior and software-WebGPU recovery, not a driver crash or physical-GPU recovery guarantee. JavaScript event-loop stalls can delay the deadline callback. The application cannot cancel native adapter/device acquisition, and a completely frozen main thread cannot display the recovery UI.

The development player now reinstalls its CPU-owned model catalog and original traveler rig along with default engine shaders. Restoring author-edited shader source, arbitrary live asset revisions and simulation-worker recovery remain open, as does the full renderer milestone.

#Initialization stage history

The playtest export records bounded {generation, stage, at} entries on the page's monotonic clock. A stage timestamp marks entry into an operation. Worker timestamps are translated from the worker clock; adjacent intervals include CPU work, browser scheduling and native promise completion. They are not GPU timestamp measurements. The history holds at most 64 entries, enough for all 49 entries from placement probing, the initial device and two permitted replacements.

Stages Operation being observed
probe Worker capability probe before transferring the visible canvas.
device, adapter-request, device-request Renderer creation, adapter acquisition, then native device acquisition.
resources Canvas configuration, layouts, graph and primitive GPU resources.
shader-modules, shader-validation Shader modules, compilation information and validation-scope completion.
shader-pipelines, shader-identity All required native pipelines, then the accepted bundle fingerprint.
initial-upload Initial queue completion before returning the native renderer.
catalog, model-prepare Catalog restoration and owned CPU model preparation/validation.
model-allocate Native model resources and allocation-error-scope completion.
model-upload Queue completion before publishing the candidate catalog.
model-install Instance reservation, catalog publication and rebuilding draws.
actors, ready Traveler rig restoration and graphics creation completion.

The ready entry completes graphics creation; the outer recovery controller still requires a baseline and completed verification frame. Optional backend observers run only at initialization/catalog boundaries, not per frame. Initial shader observations end with initialization and do not accumulate during later shader edits. Main-thread observations are fenced against a disposed owner or replaced generation.

Browser checks withhold actual pipeline completions and the catalog queue notification in both render placements. The report must stop at the corresponding subphase, retain paused state, and show complete recovery after release. The repeated-loss check verifies all three generation histories. A terminated worker's test-only runtime capture records the selected worker placement with its realm unavailable; it cannot substitute the page's empty probe and imply a fallback.

The preceding CI report reaches catalog restoration late in the unchanged 20-second recovery deadline, but lacks these subphases. The finer history is diagnostic work, not a proven fix for that timeout. Graphics quality, native pipeline coverage, frame completion checks and product deadlines remain unchanged.

#Frame completion accounting

The renderer already allowed at most one outstanding frame before this increment. A busy renderer skips new draws and uses the latest scene/camera after completion; it does not enqueue old frame requests. The new counters expose that existing boundary and distinguish submission, successful notification, rejected notification and disposal while waiting:

submittedFrames = completedFrames + failedFrames + cancelledFrames + pendingFrames

pendingFrames is zero or one. skippedFrames counts otherwise eligible draws skipped while busy. Disposal classifies an outstanding notification as cancelled and late callbacks cannot reclassify it. These are application ownership counts, not a claim that native GPU execution was cancelled.

The F2 inspector and exported report include the latest and longest completion-notification waits, monotonic last-submission/completion timestamps, and age of the last completion. These waits include queue execution and browser notification scheduling; they are not GPU timestamp measurements or compositor presentation times. The existing frame interval percentiles describe animation callbacks, not completed render FPS. Counts/timestamps belong to a renderer instance and restart after replacement.

Reports retain at most four graphics incident snapshots, including the disposed renderer's accounting and bounded failure reason. This preserves the previous instance's state when recovery creates new counters. No report is uploaded automatically.

The controlled browser case withholds real queue-completion notifications, verifies one outstanding frame while settings and the worker remain responsive, changes quality while busy, and releases the notification to apply the latest viewport without changing resource ownership. It then destroys a device with a withheld completion, checks cancellation accounting, recovers with the chosen quality and checks actual world pixels. This does not establish the cause of the intermittent CI capture/presentation stalls; those remain open.

The hold now records its own elapsed duration. The renderer's historical maximum must cover that duration and remain monotonic; an earlier startup submission may have taken longer. The input/capture investigation records why the previous strict-growth assertion was invalid and distinguishes the separate 107-second screenshot delay from a failed pixel value.

Source-specific acceptance includes the controlled report and before/after recovery captures. In the controlled hold, submitted/completed counts stay at 3/2 with one pending frame while skipped draws advance; release applies the selected 768-pixel viewport. No physical-GPU performance claim follows from these software-rendered measurements.

End of Graphics recovery