Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ view inside the editor.
- Real-time screen `describe` command using accessibility view tree - available in token-efficient format for agents
- Profiling built-in: CPU, memory, disk writes, network throughput, hang signals, and stack sampling
- CoreSimulator chrome asset rendering for device bezels
- iOS Simulator camera simulation from a generated pattern, local media file or stream URL, or a Mac camera source
- NativeScript, React Native, Flutter, UIKit and SwiftUI runtime inspector plugins to debug app's view hierarchy live
- `simdeck/test` for fast JS-based app tests that can query accessibility state and drive simulator controls

Expand Down Expand Up @@ -118,6 +119,11 @@ simdeck install android:<avd-name> /path/to/app.apk
simdeck uninstall com.example.App
simdeck open-url https://example.com
simdeck launch com.apple.Preferences
simdeck camera sources
simdeck camera start com.example.App --file /absolute/path/to/camera.mov
simdeck camera start com.example.App --webcam
simdeck camera switch --placeholder
simdeck camera stop
simdeck toggle-appearance
simdeck pasteboard set "hello"
simdeck pasteboard get
Expand Down Expand Up @@ -179,6 +185,14 @@ directory. Most device commands accept `[<udid>]`; when it is omitted, SimDeck
uses `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, the saved project default,
or the only booted simulator, in that order.

`camera start` asks the SimDeck daemon to publish a camera feed, injects the
SimDeck camera shim into the target iOS simulator app, and relaunches that
bundle. The source can be a generated pattern, an absolute image or video path,
an `http://`, `https://`, or `file://` video URL, or a Mac camera selected with
`--webcam [id-or-name]`.
Use the browser menu item **Camera Simulation...** for the same flow from the UI.
Camera simulation is iOS-simulator-only and requires a booted simulator.

## JS/TS Tests

```ts
Expand Down
29 changes: 29 additions & 0 deletions docs/api/rest.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,35 @@ Launch apps and open URLs through `/api/simulators/{udid}/action` with
`{ "action": "launch", "bundleId": "com.example.App" }` or
`{ "action": "openUrl", "url": "https://example.com" }`.

## Camera Simulation

| Method | Path | Purpose |
| -------- | -------------------------------------- | ----------------------------------------- |
| `GET` | `/api/camera/webcams` | List available Mac camera sources |
| `GET` | `/api/simulators/{udid}/camera` | Get daemon camera feed status |
| `POST` | `/api/simulators/{udid}/camera` | Start feed and optionally relaunch an app |
| `POST` | `/api/simulators/{udid}/camera/source` | Switch the running daemon source |
| `DELETE` | `/api/simulators/{udid}/camera` | Stop the daemon camera feed |

Start request:

```json
{
"bundleId": "com.example.App",
"mirror": "off",
"source": {
"kind": "video",
"arg": "/absolute/path/to/feed.mov"
}
}
```

Source `kind` is `placeholder`, `image`, `video`, or `webcam`. Image and video
sources require `arg`; local files must be absolute paths. Video sources also
accept `http://`, `https://`, and `file://` URLs. `webcam` can omit `arg` to use
the first available Mac camera, or pass a camera ID/name from `/api/camera/webcams`.
`mirror` is `auto`, `on`, or `off`.

## Performance

iOS simulator app processes run as host macOS processes. These endpoints expose host-process telemetry for matching simulator app PIDs.
Expand Down
19 changes: 19 additions & 0 deletions docs/cli/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,25 @@ simdeck open-url https://example.com
simdeck toggle-appearance
```

## Camera Simulation

```sh
simdeck camera sources
simdeck camera start com.example.App --file /absolute/path/to/feed.mov --mirror off
simdeck camera start com.example.App --webcam
simdeck camera switch --placeholder
simdeck camera switch --file /absolute/path/to/frame.png
simdeck camera status
simdeck camera stop
```

`camera start` is iOS-simulator-only. It starts the daemon-owned camera feed,
relaunches the target bundle with the SimDeck camera injector, and makes
`AVCaptureDevice`, `AVCaptureVideoDataOutput`, `AVCapturePhotoOutput`, and
`AVCaptureVideoPreviewLayer` consume the simulated feed. `camera switch` changes
the running daemon source without relaunching the app. Media files must use
absolute paths; URL sources are treated as video streams.

## Inspect UI

```sh
Expand Down
9 changes: 9 additions & 0 deletions docs/cli/flags.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,15 @@ button label.
| `pasteboard set` | `--stdin`, `--file` |
| `batch` | `--step`, `--file`, `--stdin`, `--continue-on-error` |

## Camera Simulation

| Command | Flags |
| --------------- | ---------------------------------------------------------------------------------- |
| `camera start` | `--file <path-or-url>`, `--webcam [id]`, `--mirror auto\|on\|off` |
| `camera switch` | `--file <path-or-url>`, `--webcam [id]`, `--placeholder`, `--mirror auto\|on\|off` |
| `camera status` | none |
| `camera stop` | none |

## Exit codes

| Code | Meaning |
Expand Down
2 changes: 2 additions & 0 deletions docs/cli/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ simdeck install /path/to/App.app
simdeck install /path/to/App.ipa
simdeck launch com.example.App
simdeck open-url https://example.com
simdeck camera start com.example.App --file /absolute/path/to/feed.mov
simdeck camera stop
simdeck tap --label "Continue" --wait-timeout-ms 5000
simdeck tap --id com.apple.settings.screenTime --expect-id BackButton
simdeck tap "Continue"
Expand Down
16 changes: 16 additions & 0 deletions docs/guide/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,21 @@ simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro

Supported commands include `launchApp`, `openLink`, `tapOn`, `inputText`, `eraseText`, `pressKey`, `assertVisible`, `assertNotVisible`, `scrollUntilVisible`, `swipe`, `takeScreenshot`, and `waitForAnimationToEnd`. Unsupported Maestro commands fail clearly so the flow can be adjusted or the compatibility layer can be expanded.

## Camera Apps

For iOS apps that use `AVFoundation`, start camera simulation before running the
camera workflow:

```sh
simdeck camera start com.example.App --file /absolute/path/to/feed.mov --mirror off
simdeck camera switch --placeholder
simdeck camera stop
```

The browser UI exposes the same controls from **Camera Simulation...** in the
simulator menu. Webcam forwarding is available with `--webcam` when macOS has an
available camera and has granted camera permission to SimDeck.

## Repository tests

Normal unit and client tests:
Expand All @@ -79,6 +94,7 @@ npm run build:cli
npm run build:client
npm run test:integration:fixture
npm run test:integration:cli
npm run test:integration:camera
```

Verbose iOS run:
Expand Down
2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
"scripts/studio-provider-bridge.mjs",
"scripts/postinstall.mjs",
"build/simdeck-bin",
"build/camera/",
"packages/client/dist/",
"packages/simdeck-test/dist/"
],
Expand Down Expand Up @@ -68,6 +69,7 @@
"test:integration:cli": "node scripts/integration/cli.mjs",
"test:integration:cli:verbose": "SIMDECK_INTEGRATION_VERBOSE=1 SIMDECK_INTEGRATION_SHOW_SIMULATOR=1 node scripts/integration/cli.mjs",
"test:integration:service": "node scripts/integration/service-restart.mjs",
"test:integration:camera": "node scripts/integration/camera.mjs",
"test:integration:fixture": "node scripts/integration/prebuild-fixture.mjs",
"test:integration:js-api": "node scripts/integration/js-api.mjs",
"test:integration:webrtc": "node scripts/integration/webrtc.mjs",
Expand Down
54 changes: 54 additions & 0 deletions packages/client/src/api/simulators.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ import { apiRequest } from "./client";
import type {
AccessibilitySourcePreference,
AccessibilityTreeResponse,
CameraStartRequest,
CameraStatusResponse,
CameraWebcamsResponse,
ChromeDevToolsTargetDiscovery,
ChromeProfile,
CreateSimulatorRequest,
Expand Down Expand Up @@ -175,6 +178,57 @@ export async function sampleSimulatorProcess(
);
}

export async function fetchCameraWebcams(
options: RequestInit = {},
): Promise<CameraWebcamsResponse> {
return apiRequest<CameraWebcamsResponse>("/api/camera/webcams", options);
}

export async function fetchCameraStatus(
udid: string,
options: RequestInit = {},
): Promise<CameraStatusResponse> {
return apiRequest<CameraStatusResponse>(
`/api/simulators/${encodeURIComponent(udid)}/camera`,
options,
);
}

export async function startCameraSimulation(
udid: string,
payload: CameraStartRequest,
): Promise<CameraStatusResponse> {
return apiRequest<CameraStatusResponse>(
`/api/simulators/${encodeURIComponent(udid)}/camera`,
{
body: JSON.stringify(payload),
method: "POST",
},
);
}

export async function switchCameraSimulationSource(
udid: string,
payload: Pick<CameraStartRequest, "source" | "mirror">,
): Promise<CameraStatusResponse> {
return apiRequest<CameraStatusResponse>(
`/api/simulators/${encodeURIComponent(udid)}/camera/source`,
{
body: JSON.stringify(payload),
method: "POST",
},
);
}

export async function stopCameraSimulation(
udid: string,
): Promise<CameraStatusResponse> {
return apiRequest<CameraStatusResponse>(
`/api/simulators/${encodeURIComponent(udid)}/camera`,
{ method: "DELETE" },
);
}

export async function fetchWebKitTargets(
udid: string,
options: RequestInit = {},
Expand Down
40 changes: 40 additions & 0 deletions packages/client/src/api/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,46 @@ export interface InstallUploadResponse {
udid: string;
}

export type CameraSourceKind = "placeholder" | "image" | "video" | "webcam";

export interface CameraSourceRequest {
kind: CameraSourceKind;
arg?: string;
}

export interface CameraStartRequest {
bundleId?: string;
source: CameraSourceRequest;
mirror?: "auto" | "on" | "off";
}

export interface CameraWebcam {
id: string;
name: string;
position?: string;
}

export interface CameraWebcamsResponse {
webcams: CameraWebcam[];
}

export interface CameraStatusResponse {
ok?: boolean;
udid?: string;
alive: boolean;
source?: CameraSourceKind | string;
arg?: string;
sourceLabel?: string;
mirror?: "auto" | "on" | "off" | string;
daemonPid?: number;
bundleIds?: string[];
width?: number;
height?: number;
sequence?: number;
appLogPath?: string;
error?: string;
}

export interface SimulatorForegroundApp {
appName?: string | null;
bundleIdentifier?: string | null;
Expand Down
14 changes: 14 additions & 0 deletions packages/client/src/app/AppShell.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ import type {
ViewMode,
} from "../features/viewport/types";
import { useViewportLayout } from "../features/viewport/useViewportLayout";
import { CameraSimulationModal } from "../features/simulators/CameraSimulationModal";
import { NewSimulatorModal } from "../features/simulators/NewSimulatorModal";
import { nextViewportWheelPanState } from "../features/viewport/viewportWheel";
import {
Expand Down Expand Up @@ -491,6 +492,7 @@ export function AppShell({
const [menuOpen, setMenuOpen] = useState(false);
const [simulatorMenuOpen, setSimulatorMenuOpen] = useState(false);
const [newSimulatorOpen, setNewSimulatorOpen] = useState(false);
const [cameraSimulationOpen, setCameraSimulationOpen] = useState(false);
const [localError, setLocalError] = useState("");
const [captureStatus, setCaptureStatus] = useState<CaptureStatus | null>(
null,
Expand Down Expand Up @@ -2872,6 +2874,10 @@ export function AppShell({
}
}}
onInstallAppPrompt={openInstallAppPicker}
onOpenCameraSimulation={() => {
setMenuOpen(false);
setCameraSimulationOpen(true);
}}
onOpenAppSwitcher={() => {
if (!selectedSimulator) {
return;
Expand Down Expand Up @@ -2991,6 +2997,14 @@ export function AppShell({
open={newSimulatorOpen && !hideSimulatorSelection}
selectedSimulator={selectedSimulator}
/>
<CameraSimulationModal
foregroundBundleId={
selectedSimulatorState?.foregroundApp?.bundleIdentifier ?? null
}
onClose={() => setCameraSimulationOpen(false)}
open={cameraSimulationOpen}
selectedSimulator={selectedSimulator}
/>
<SimulatorViewport
accessibilityHoveredId={accessibilityHoveredId}
appInstallOverlayLabel={captureOverlayLabel}
Expand Down
Loading
Loading