# Visor Development Kit: all the docs From https://visor.macedonga.com/developers/docs/. Each page starts at its own heading. # Overview The Visor Development Kit (VDK) is how other software uses Meta's glasses through Visor. Visor is an Android app that replaces the Meta AI app as the phone side of Ray-Ban Meta glasses, the Meta Ray-Ban Display and the Meta Neural Band: it holds their Bluetooth and Wi-Fi links and speaks their protocol. The VDK gives that link to your code. How that protocol works, down to the bytes, is in the [protocol reference](https://visor.macedonga.com/protocol/). ## Who it is for - **Android apps** that want the glasses' camera, their status, the band's gestures or the Display. - **Web apps on the Meta Ray-Ban Display**, which reach the VDK from JavaScript through the Display's browser. - **Scripts and tools on a computer**: automations, prototypes, a test harness, an AI agent that drives the glasses. They reach Visor through adb or over the network. - **Visor's own features**, which can be built and tried outside Visor first. ## How it works ```mermaid flowchart LR app["Your Android app"] -- "VDK library" --> visor computer["Your computer"] -- "adb, JSON-RPC" --> visor agents["Programs and AI agents"] -- "WebSocket, MCP" --> visor web["A web app on the Display"] -- "the Display's browser" --> visor visor(["Visor, on the phone"]) -- "Bluetooth, Wi-Fi Direct" --> glasses["Glasses and Neural Band"] ``` Everything the VDK does is a **method** (a request with one answer, like `camera.photo`), a **topic** (a stream of events, like `band.gestures`) or a **feed** (a stream of binary records, like the live video of `camera.live`). Each has a name, a scope, and the protocol version that added it. The [kit-protocol](https://github.com/macedonga/visor-kit/tree/main/protocol) module defines all of them, and [`protocol/api/openrpc.json`](https://github.com/macedonga/visor-kit/blob/main/protocol/api/openrpc.json) describes them for any language. There are four ways in, with the same methods and topics: - **The Android library** (`com.macedonga.visor:kit`). Your app binds to Visor's VDK service. Visor knows which app is calling from Android itself, by its package and its signing certificate, so nothing can pretend to be your app. - **The adb socket.** Visor listens on a local socket that only adb's shell user may use. The `visor` command on your computer, or any program that speaks [JSON-RPC](json-rpc.md), reaches it with `adb forward`. - **The network door.** JSON-RPC over a WebSocket, and MCP for AI agents, on the phone or the local network, for clients the person makes a token for. See [The network door](network.md). - **The Display's browser.** A web app on the Meta Ray-Ban Display connects with the TypeScript client's `Visor.connectGlasses()`. See [Web apps on the Display](web-apps.md). The person stays in charge: - **Scopes** say what each app, client or web app may do. The person grants them and can take them back in Settings. See [Scopes and consent](scopes-and-consent.md). - **Developer mode** in Visor opens the adb socket for the computer-side tools. Apps never need it. The glasses' own safeguards stay in place: the capture light turns on for every photo and video the VDK takes. ## What it can do | Area | Methods and topics | Glasses | |---|---|---| | [Glasses](glasses.md) | `glasses.status`, `glasses.pairs`, `glasses.select`, `glasses.inputs`, `glasses.motion` | All | | [Camera](camera-and-captures.md) | `camera.photo`, `camera.video.start`, `camera.video.stop`, `camera.live`, `camera.live.photo` | All | | [Captures](camera-and-captures.md) | `captures.list`, `captures.import`, `captures.available`, `captures.progress`, `captures.imported`, `captures.failed` | All | | [Neural Band](neural-band.md) | `band.status`, `band.gestures`, `band.motion`, `band.buzz` | Meta Ray-Ban Display with a Neural Band | | [Display](display.md) | `display.prompter.*`, `display.layout.*`, `display.video`, `display.cards.*` | Meta Ray-Ban Display | | [Web apps](web-apps.md) | Every method and topic, from JavaScript in a web app on the Display | Meta Ray-Ban Display | | [Voice](voice-and-assistant.md) | `voice.requests`, `voice.reply`, `voice.speak`, `voice.transcripts` | All | | [Assistant](voice-and-assistant.md) | `assistant.tools`, `assistant.tools.answer`, `assistant.turns`, `assistant.history` | All | | [Notifications](voice-and-assistant.md) | `notify.post`, `notify.remove` | Meta Ray-Ban Display | | [Settings](settings.md) | `settings.list`, `settings.get`, `settings.set`, `settings.changes` | All | ## Compared with Meta's Wearables Device Access Toolkit Meta's toolkit (DAT) works through the Meta AI app and a Meta-signed program on the glasses. The VDK works through Visor instead, so the trade-offs differ: | | Meta's toolkit | VDK | |---|---|---| | Needs | The Meta AI app, a Meta account, registration | Visor, and the person's consent | | Gating | Meta can turn features off per user; photos, speech, inputs and motion are Beta and can't be published yet | None | | Telemetry | Crash reports and analytics on by default | None | | Permissions | Allow once or always, in the Meta AI app | Allow, or only this time, on Visor's screen; taken back in Visor's settings | | Camera | A video stream up to 720p over Bluetooth; photos up to 4032 x 3024 | A live stream up to 1080p at 30 frames over Wi-Fi Direct, encoded or decoded, with the microphones; full-size photos and videos, imported over Wi-Fi Direct | | Speech | The glasses' own recognition, with partial results | On the Display, the glasses' own recognition, as DAT; on other glasses, the phone's recogniser on the glasses' microphones. Partial results and a confidence on both | | Inputs | Navigation, select, back and drags, from the touchpad and the band | Every touch and press on every model, and the Display's navigation | | Motion | Accelerometer, gyroscope, magnetometer and orientation, 5 to 60 Hz, with the toolkit app on the screen | The same samples and rates, with nothing on the screen; the band's too, on the wrist, for a band shared with the phone | | Neural Band | Navigation events | Every gesture, with the band left on the Display; vibration; with the band shared with the phone, its own events and its motion | | Voice | "Hey Meta, start {app}" opens an app | Questions that name an app go to it, answered in the glasses' own voice; a phrase also starts an app; tools for the model; lines the glasses say unasked | | Display | Layouts of boxes, text, images, icons, buttons, and a video | The same layouts, from the same toolkit app; cards of the app's own on the Display's home; the teleprompter and notifications | | Web apps on the Display | No | Every VDK method from the page's JavaScript; the page's WebMCP tools for the wearer's assistant | | Settings | No | Every setting the glasses keep, readable, and the everyday ones writable | | The assistant's history | No | Every question and answer, as they come and from the phone's history | | AI tools | A docs search over MCP; plugins and rules for coding assistants | A docs search over MCP; a plugin and guidance for coding assistants; and an MCP server on the phone, so an agent can use the glasses | | Other languages | Android and iOS | Android, JavaScript on the Display, and any language with WebSocket or HTTP, with clients for Python and TypeScript; no iOS, as Visor runs on Android | | From a computer | No | The `visor` command, through adb: call, watch, trace, simulate, record and replay | | Without the glasses | The Mock Device Kit in tests, with a camera from a video file and motion from a CSV; a Display simulator in Chrome | FakeVisor in tests, with the same feeds; `visor fake` on a computer for every language and MCP, with a Display simulator in any browser and a preview in the app; record and replay | On the glasses, layouts come from the session DAT opens with Meta's toolkit app on the Display: Visor installs that app and sets the glasses up once per pair. Motion comes from the glasses' own sensor service, and the band's and the touchpad's gestures from a session without the toolkit app, so neither puts anything on the screen. [Porting from Meta's DAT](porting-from-dat.md) maps each part of DAT to the VDK. ## Roadmap 1. **VDK 0.1:** the glasses' status and pairs, the camera and captures, the Neural Band, the Display's teleprompter, the `visor` command and FakeVisor. 2. **Voice and the assistant:** phrases routed to apps ("Hey Meta, ask My App..."), apps started by a phrase, tools for the assistant's model, answers in the glasses' own voice, lines the glasses say unasked, transcripts, notifications on the Display, the glasses' settings and the assistant's history. 3. **The network:** JSON-RPC over WebSocket and an MCP server on the phone, webhooks, Python and TypeScript clients, and raw access to the glasses' services for research. 4. **The Display:** [Briefing cards](briefing-cards.md) with details, pages, charts, clicks and live updates; the VDK in the Display's web apps, with their WebMCP tools for the assistant ([Web apps on the Display](web-apps.md)); and the Neural Band shared with the phone, so apps get its gestures, wrist motion and vibration while the Display keeps using it. 5. **The Display's toolkit session:** layouts with video, inputs, motion with nothing on the screen, and the Display's own speech recognition. Next: the first public release, on Maven Central, npm and PyPI. # Getting started This guide takes an Android app from nothing to a photo taken with the glasses. ## 1. Prepare the phone Install [Visor](https://github.com/macedonga/visor) and set up the glasses in it. Nothing else: your app asks the person for what it needs, on Visor's consent screen. ## 2. Add the library The VDK needs Android 8.0 (API 26) or later in your app. Visor itself runs on Android 12 and later. ```kotlin dependencies { implementation("com.macedonga.visor:kit:0.1.0") testImplementation("com.macedonga.visor:kit-testing:0.1.0") } ``` The library's manifest declares Visor's package in ``, so your app can see Visor on Android 11 and later without anything in your own manifest. ## 3. Connect ```kotlin if (!Visor.isInstalled(context)) { // Point the person to Visor's releases page. return } val visor = try { Visor.connect(context) } catch (e: KitException) { // e.code is VISOR_TOO_OLD, or another code: see Errors. return } ``` `connect` suspends until Visor answers. Keep one `Visor` for as long as you use the glasses, and `close()` it when you are done. The [connection](connection.md) page explains what happens when Visor restarts. ## 4. Ask for scopes Each capability needs a scope. Ask for the ones you need with `RequestScopes`, which opens Visor's consent screen over your app: ```kotlin val ask = registerForActivityResult(RequestScopes()) { held -> // held: every scope the app holds now. It can be fewer than asked for. } ask.launch(setOf(Scope.STATE, Scope.CAMERA)) ``` Register the launcher before the activity starts, as for any activity result. If your app already holds every scope it asks for, the screen does not appear and the result comes at once. `visor.scopes()` returns the scopes the app holds at any time. ## 5. Use the glasses ```kotlin lifecycleScope.launch { visor.glasses.status.collect { status -> batteryView.text = status.battery?.let { "$it%" } ?: "-" } } lifecycleScope.launch { try { val photo = visor.camera.takePhoto() imageView.setImageURI(Uri.parse(photo.uri)) } catch (e: KitException) { when (e.code) { ErrorCode.SCOPE_DENIED -> ask.launch(setOf(e.scope!!)) ErrorCode.NOT_CONNECTED -> showMessage("Put the glasses on.") else -> showMessage(e.message) } } } ``` `takePhoto()` returns once the photo is in the phone's gallery. Visor grants your app read access to its URI, so your app needs no media permission to show it. ## 6. Try it without the glasses While you build, these tools save you from wearing the glasses: - The `visor` command on your computer shows what your app sends and receives, and can stand in for the band or the glasses: `visor simulate band.gestures`, then type `index double_tap`. - `visor record` saves what the wearer does once, and `visor replay` plays it back as often as you like. - `FakeVisor` runs Visor inside your tests. - `visor fake` runs a fake Visor on your computer, for scripts, web pages and MCP clients, with no phone at all. All of them are in [Testing](testing.md). ## Next - [Scopes and consent](scopes-and-consent.md) - [Errors](errors.md): what each code means and what to do about it - The capability pages: [Glasses](glasses.md), [Camera and captures](camera-and-captures.md), [Neural Band](neural-band.md), [Display](display.md), [Briefing cards](briefing-cards.md), [Voice and the assistant](voice-and-assistant.md), [Settings](settings.md) - Building a web app for the Display instead: [Web apps on the Display](web-apps.md) # Scopes and consent An app can use only what the person allowed. Visor checks every call and every stream against the app's **scopes**, which the person grants on Visor's consent screen. Nothing else is needed: no setting to turn on, no Developer mode. ## The scopes | Scope | Lets the app | Methods and topics | |---|---|---| | `state` | See whether the glasses are connected, worn and charged | `glasses.status` | | `pairs` | See every pair set up in Visor, with names and looks, and choose which one Visor connects to | `glasses.pairs`, `glasses.select` | | `inputs` | Know when the wearer taps or swipes the touchpad or presses a button | `glasses.inputs` | | `motion` | Read how the glasses and the band move | `glasses.motion`, `band.motion` | | `microphone` | Turn what the glasses' microphones hear into text | `voice.transcripts` | | `camera` | Take photos and videos, and copy them to the phone | `camera.*`, `captures.*` | | `band` | Read the Neural Band's gestures and make it vibrate | `band.status`, `band.gestures`, `band.buzz` | | `display` | Show layouts, cards of the app's own and the teleprompter on the Display | `display.*` | | `voice` | Answer the questions the wearer asks that name the app | `voice.*` | | `assistant` | Give the assistant tools it can use while it answers | `assistant.*` | | `notify` | Show notifications on the Display | `notify.*` | | `settings` | Read the glasses' settings and change the everyday ones | `settings.*` | | `conversations` | Read what the wearer asks the assistant and what it answers | `assistant.turns`, `assistant.history` | | `raw` | Send and read the glasses' own protocol messages, for research; only with Developer mode on | `raw.*` | `kit.hello` needs no scope. Later releases add scopes; an app built against an older VDK never sees them unless it asks for them by id. ## Asking `RequestScopes` opens Visor's consent screen over your app. The screen names your app, as the launcher shows it, and lists only the scopes it does not hold yet. The person allows all of them or none: **Allow** keeps them, **Only this time** gives them until your app closes its last connection to Visor (DAT's Allow once), and **Don't allow** gives none. ```kotlin val ask = registerForActivityResult(RequestScopes()) { held -> } ask.launch(setOf(Scope.CAMERA, Scope.BAND)) ``` The result is every scope your app holds once the person is done, also when they just go back. A scope this Visor does not know is never granted, so it is missing from the result. Launching throws `ActivityNotFoundException` when Visor is not installed. Check `Visor.isInstalled(context)` first. ## When a scope is missing A call that needs a scope the app does not hold fails with `scope_denied`, and `KitException.scope` names the scope: ```kotlin catch (e: KitException) { if (e.code == ErrorCode.SCOPE_DENIED) ask.launch(setOf(e.scope!!)) } ``` A stream started without its scope ends at once with the same failure. Collect it again after the person allows it. ## Taking scopes back The person sees every app that holds scopes in Visor, under Settings, Advanced settings, Apps with access, and can remove an app's access there. When that happens: - the app's open streams end with `scope_denied`; - its next calls fail the same way; - calls already running finish. ## Programs and web apps Programs on the [network door](network.md) and [web apps on the Display](web-apps.md) do not use the consent screen: the person picks their scopes in Visor, when they make a network client, or in Settings > Web apps for a web app. Their calls fail with `scope_denied` the same way, and taking a scope back ends their streams at once. ## Who an app is Visor learns which app calls from Android, by the calling uid, and ties each grant to the app's package and to the certificate it is signed with. If a different app is later installed under the same package name with another certificate, it holds nothing. Your debug and release builds are signed with different certificates, so each one asks for its own grant. ## While you develop `visor grant` and `visor revoke` set an app's scopes from your computer, without the consent screen: ``` visor grant com.example.app camera band visor revoke com.example.app band visor apps ``` See [Testing](testing.md). # The connection A `Visor` object is one connection to Visor. It follows Visor through restarts on its own, and tells you where it stands through `state`. ## Connecting ```kotlin val visor = Visor.connect(context) ``` `connect` binds to Visor's VDK service, says hello, and returns once Visor answered. It fails with a `KitException`: | Code | Why | |---|---| | `visor_missing` | Visor is not installed. | | `visor_too_old` | This Visor has no VDK service. | It suspends while Visor's process starts. To give up after a while, wrap it: `withTimeout(10.seconds) { Visor.connect(context) }`. Keep one `Visor` for as long as you use the glasses, for example in a ViewModel or a service, and call `close()` when you are done. It uses the application context, so it holds no activity. ## State ```kotlin visor.state.collect { state -> when (state) { VisorState.Connecting -> ... // binding again after Visor restarted is VisorState.Open -> ... // state.welcome: Visor's version, its methods, your scopes is VisorState.Closed -> ... // over for good; state.error says why, null when you closed it } } ``` - **Connecting:** Visor restarted, or was updated, and Android is bringing it back. Calls made now wait for it. Streams pick up again by themselves once it is back; events sent in between are lost. - **Open:** Visor answers. `welcome` lists the methods and topics this Visor has and the scopes your app holds. - **Closed:** you closed the connection, or Visor turned it away, for example because Visor was uninstalled. Connect again for a new one. ## Calls and streams Every call is a suspending function, safe from the main thread, and fails with a `KitException`. Cancelling the coroutine cancels the call in Visor too. Every stream is a cold `Flow`: collecting it subscribes, and stopping the collection unsubscribes. Streams of state, like `glasses.status`, send the current value first. A stream ends with a `KitException` when Visor ends it, for example when a scope is taken back. ## Versions Visor and your app can be different ages. At hello, each side says which protocol version it speaks, and Visor lists what it answers. Ask before using something new: ```kotlin if (visor.supports(Methods.DISPLAY_PROMPTER_SHOW)) { ... } ``` A call that the installed Visor does not have fails with `visor_too_old` before it is sent. See [Compatibility](compatibility.md). ## Threads The library runs nothing on the main thread. Answers from Visor arrive on binder threads and resume your coroutines in their own context. # Glasses Visor can have several pairs set up, and connects to one at a time: the **active** pair. Every method acts on the active pair. The status needs the `state` scope; seeing every pair and choosing the active one needs `pairs`. ## Status ```kotlin visor.glasses.status.collect { status -> } // now, then each change val now = visor.glasses.status.first() // once ``` A `GlassesStatus`: | Field | Type | Meaning | |---|---|---| | `paired` | `Boolean` | False when Visor has no glasses set up. Every other field is then empty. | | `link` | `Link` | `disconnected`, `connecting` or `connected`. | | `model` | `String?` | The glasses' own model name, to show to people. | | `name` | `String?` | The name the person gave the pair. | | `serial` | `String?` | | | `firmware` | `String?` | The firmware build. | | `display` | `Boolean?` | True for glasses with a display. Null until Visor has seen the glasses once. | | `battery` | `Int?` | 0 to 100. | | `charging` | `Boolean?` | | | `worn` | `Boolean?` | True while the glasses sit on a face. | | `folded` | `Boolean?` | True while the temples are folded. | | `recording` | `Boolean?` | True while the glasses record a video. | | `caseBattery` | `Int?` | The charging case's battery, 0 to 100, while the glasses sit in the case. | The values come from the glasses, so most are null while they are not connected. Tell models apart by `display`, never by `model`, whose wording can change. ## Your pairs Scope: `pairs`. ```kotlin visor.glasses.pairs.collect { pairs -> pairs.forEach { println("${it.name ?: it.look?.styleName} ${if (it.active) "(active)" else ""}") } } visor.glasses.select("2Y0YBYPJ8F007Q") ``` `pairs` sends every pair set up in Visor now, then each time one is added, renamed, removed or made active. `select` makes another pair the active one: Visor lets go of the pair connected now and connects to the new one, as when the person picks it on Visor's home. It returns once Visor has switched, before the pair is connected, so watch `status` for the link. It fails with `busy` while a Live view or an import holds the glasses, and with `invalid_params` and reason `not_found` for a serial Visor has not set up. A `GlassesPair`: | Field | Type | Meaning | |---|---|---| | `serial` | `String` | | | `name` | `String?` | The name the person gave the pair. | | `active` | `Boolean` | True for the pair Visor connects to now. | | `link` | `Link` | The active pair's link; the others read `disconnected`. | | `product` | `Int?` | Meta's product id of the model, such as 266 for Ray-Ban Meta (Gen 2). | | `display` | `Boolean?` | True for glasses with a display. | | `look` | `GlassesLook?` | Frame, colour and lenses from Meta's record of the serial: `style`, `styleName`, `color`, `colorName`, `lens`, `lensName`, `size`. | | `battery` | `Int?` | The battery when Visor last heard from the pair. | | `caseBattery` | `Int?` | The case's battery at that time. | | `lastSeen` | `Long?` | When Visor last heard from the pair, in milliseconds since 1970. | | `band` | `String?` | The serial of the Neural Band paired with it. | ## Touches and presses Scope: `inputs`. All glasses. ```kotlin visor.glasses.inputs.collect { input -> if (input.source == InputSource.TOUCHPAD && input.action == InputAction.SWIPE_FORWARD) nextPage() } ``` Each `GlassesInput` is one touch of the touchpad or one press of a button, as the glasses report it to the phone. The glasses still act on it as they always do: a tap still pauses the music, and the capture button still takes a photo. The VDK cannot keep an input from them, except back on the Display (below). | `source` | Actions | |---|---| | `touchpad` | `tap`, `double_tap`, `triple_tap`, `tap_hold`, `swipe_forward`, `swipe_backward`, `slide_forward`, `slide_backward`, `touch`, `release`, `palm_tap`, `palm_hold` | | `capture_button` | `press`, `release`, `long_press`, `very_long_press`, `very_very_long_press`, `extremely_long_press`, `very_very_long_release`, `double_press` | | `action_button` | `press`, `release`, `long_release` | | `touchpad`, `band` on the Display | `nav_up`, `nav_down`, `nav_left`, `nav_right`, `select`, `back`, `drag_start`, `drag_move`, `drag_end` | A swipe goes `forward` towards the lenses. `touch` and `release` tell when a finger rests on the touchpad and when it leaves; `slide_*` is a finger that moves while it rests there. `time` is when Visor heard the input. Which actions a pair reports depends on its model and firmware: treat an action you do not know as unknown, and a source you do not know too. On the Display the band's gestures come as `tap`, `double_tap`, `triple_tap`, `tap_hold` and `nav_*` at any time. While your app shows a layout, the touchpad's come too, and both navigate, as in DAT: `nav_*`, `select` and `back`, and drags with `x`, `y`, `dx` and `dy` in the glasses' own units. A tap then comes twice, as `tap` and as `select`: act on one of them. Ask for some sources only, and keep back for your app, with an `InputsRequest`: ```kotlin visor.glasses.inputs(InputsRequest { sources = listOf(InputSource.BAND); keepBack = true }).collect { input -> } ``` `keepBack` asks the Display to hand back to your app as `back`. The band's back, a middle-finger tap, still closes the layout: the Display acts on it before the layout hears it. ## Motion ```kotlin visor.glasses.motion.collect { sample -> } // 30 samples a second visor.glasses.motion(rateHz = 60).collect { sample -> } ``` > On the glasses, Visor reads motion from their own sensor service, with nothing on the screen and no toolkit app. > The phone fuses the accelerometer, gyroscope and magnetometer into `orientation`, which settles within two > seconds of the start. `source` is `glasses`; the band's motion on the wrist is [`band.motion`](neural-band.md#motion). Motion needs the `motion` scope. The rate is one of 5, 10, 15, 24, 30 or 60 samples a second. A `MotionSample`: | Field | Type | Meaning | |---|---|---| | `time` | `Long` | When the sensors read it, in nanoseconds on the glasses' clock. Only differences mean something. | | `accelerometer` | `Vector3?` | Acceleration, gravity included, in m/s². | | `gyroscope` | `Vector3?` | Rotation speed, in rad/s. | | `magnetometer` | `Vector3?` | The magnetic field, in µT. | | `orientation` | `Quaternion?` | The sensors combined, as a unit quaternion. | | `source` | `MotionSource` | `glasses` or `band`. | The axes are Android's, in the sensor's frame. A sensor the glasses did not read for a sample is null. In a test, wait for `fake.glasses.motionRate` to be set, then send readings with `fake.glasses.motion(ax, ay, az)` or a whole `MotionSample` with `fake.glasses.motion(sample)`. ## JSON-RPC | | Name | Params | Answer | |---|---|---|---| | Method | `glasses.status` | none | a `GlassesStatus` | | Topic | `glasses.status` | none | a `GlassesStatus` now, then on each change | | Method | `glasses.pairs` | none | a list of `GlassesPair` | | Topic | `glasses.pairs` | none | the list now, then on each change | | Method | `glasses.select` | `serial` | the new active `GlassesPair` | | Topic | `glasses.inputs` | `sources`, `keepBack` (an `InputsRequest`, optional) | a `GlassesInput` for each touch or press | | Topic | `glasses.motion` | `rateHz` (default 30) | a `MotionSample` for each reading | ``` $ visor call glasses.status { "paired": true, "link": "connected", "serial": "...", "display": false, "battery": 82, "charging": false, "worn": true, "folded": false, "recording": false } ``` # Camera and captures Scope: `camera`. All glasses. The glasses keep their photos and videos until a phone imports them. Visor imports over Wi-Fi Direct, so the phone's Wi-Fi must be on; it falls back to Bluetooth only when Wi-Fi Direct fails. The glasses' capture light turns on for every photo and video, as it does for their own button. ## Photos ```kotlin val photo = visor.camera.takePhoto() // returns once it is in the gallery val onGlasses = visor.camera.takePhoto(import = false) // returns once the glasses have it ``` The result is a `Capture`. With an import, `uri` is the photo's `content://` address in the phone's gallery, and Visor grants your app read access to it: your app needs no media permission to show it. A photo fails with: | Code | Why | |---|---| | `not_connected` | The glasses are not connected and did not connect within 20 seconds. | | `busy` | The glasses are recording a video. | | `timeout` | The glasses did not take the photo, or it did not show up on them. | | `failed` | The import failed. `reason` says why: see Imports below. | The glasses take a photo for the phone only while a switch of theirs is on. Visor turns it on while any app holds the `camera` scope. The first time, it can take a minute while Visor reads the glasses' configuration, and a photo asked for meanwhile fails with `timeout`. ## Videos ```kotlin visor.camera.startVideo() // returns once the glasses record visor.camera.startVideo(VideoKind.HYPERLAPSE) visor.camera.stopVideo() // returns once they stopped ``` The glasses end a video on their own after their maximum length. The new video then appears on `captures.available`, like one taken with the button. ## The camera live ```kotlin val config = LiveConfig { quality = LiveQuality.HIGH; frameRate = 30; audio = true } LiveVideoDecoder(surfaceView.holder.surface).use { video -> visor.camera.live(config).flowOn(Dispatchers.IO).collect { sample -> video.decode(sample) } } ``` `live` streams what the glasses' camera sees, from Visor's Live view over Wi-Fi Direct, so the phone's Wi-Fi must be on. Collecting starts the Live, and the glasses' capture light stays on while it runs. When the Live already runs, for Visor's own Live view or another app, your app joins it as it is. When the last app stops collecting, the Live stops, unless the person started it in Visor. Each `LiveSample` is a `header` and its `data`: | Field | Type | Meaning | |---|---|---| | `kind` | `LiveKind` | `video` or `audio`. | | `codec` | `String` | The MIME type a `MediaCodec` takes: `video/hevc` or `video/avc` in Annex-B, `audio/mp4a-latm` as raw AAC-LC. | | `ptsUs` | `Long` | The glasses' presentation time, in microseconds. | | `config` | `Boolean` | True when `data` sets up the decoder: the video's parameter sets, or the audio's AudioSpecificConfig. | | `key` | `Boolean` | True for a video keyframe. | | `width`, `height` | `Int?` | On a video config: the picture's size. | | `sampleRate`, `channels` | `Int?` | On an audio config. | Each kind starts with its config, and the video then starts on a keyframe. A new config can come mid-stream, when the Live restarts. Visor sends the video and the audio as the glasses encode them, so you can save them, send them on, or decode them: - `LiveVideoDecoder(surface)` draws each picture on a `Surface` as soon as it is decoded. - `LiveVideoDecoder { image, ptsUs -> }` hands each picture over as a YUV_420_888 `android.media.Image`, valid until the block returns: for ML Kit, OpenCV or your own model. - `LiveAudioDecoder { samples, sampleRate, channels, ptsUs -> }` hands the audio over as 16-bit PCM, interleaved. Feed every sample to each decoder you use, on one thread; each skips the other kind. When frames are lost on the way, Visor skips ahead to the next keyframe, so a slow app sees a pause instead of a broken picture. `LiveConfig`: | Field | Values | Default | |---|---|---| | `quality` | `low` (360 x 640), `medium` (504 x 896), `high` (720 x 1280), `full_hd` (1080 x 1920) | `medium` | | `frameRate` | 2, 7, 15, 24 or 30 | 24 | | `audio` | true for the glasses' microphones too | false | The config applies only when your app starts the Live. Visor waits for an app that reads slowly, and ends the stream of an app that stops reading for 5 seconds, so that a frozen app cannot keep the glasses filming. The flow fails with: | Code | Why | |---|---| | `not_connected` | The glasses are not connected. | | `invalid_params` | The quality or the frame rate is not one of the above. | | `timeout` | The Live did not start within 90 seconds. | | `failed` | The Live did not start or ended. `reason` says why, as below. | | `visor_gone` | Visor restarted. The stream does not come back by itself: collect again. | | Reason | Meaning | |---|---| | `stopped` | The person stopped the Live in Visor. | | `in_case` | The glasses charge in their case, where they keep the camera off. | | `wifi_off` | The phone's Wi-Fi is off. | | `wifi_direct_denied` | Visor lacks Android's Nearby devices permission. | | `wifi_busy` | An import holds the Wi-Fi link to the glasses. | | `vpn` | A VPN on the phone keeps Visor off the glasses' network. | | `no_key` | Visor has no key for these glasses on this phone. | | `join_failed`, `session_failed` | The Wi-Fi link to the glasses did not come up, or the glasses opened no session on it. | | `no_video` | The Live started but no video came. | | `gave_up` | The video stopped and restarting it failed. | ### A photo while live ```kotlin val photo = visor.camera.takeLivePhoto() val heic = context.contentResolver.openInputStream(Uri.parse(photo.uri))!!.use { it.readBytes() } ``` `takeLivePhoto` takes a photo at the glasses' photo size without stopping the stream. The result is a HEIC that your app may read from `uri` for 10 minutes. It fails with `busy` when the Live is not running, and `timeout` when the glasses send no photo. Over adb or the network door, `data` holds the photo in Base64 instead of `uri`. ## Captures ```kotlin val waiting = visor.captures.list() // on the glasses, not on the phone yet val video = visor.captures.import(waiting.first().id) ``` A `Capture`: | Field | Type | Meaning | |---|---|---| | `id` | `String` | The capture's id on the glasses. `import` takes it. | | `type` | `CaptureType` | `photo`, `live` (a photo with the clip around it) or `video`. | | `time` | `Long?` | When the glasses took it, in milliseconds since 1970. | | `size` | `Long?` | Bytes on the glasses. | | `uri` | `String?` | The address in the gallery, once imported. | ### Imports `import` returns at once when the phone already has the capture. Otherwise it returns when the capture is in the gallery, and fails with `failed` and one of these as `reason`: | Reason | Meaning | |---|---| | `not_found` | The glasses have no capture with that id. | | `wifi_off` | The phone's Wi-Fi is off. | | `wifi_stuck` | Android's Wi-Fi Direct stopped answering. Only turning Wi-Fi off and on helps. | | `wifi_failed`, `glasses_did_not_join`, `no_wifi_session` | Wi-Fi Direct did not come up. | | `not_allowed` | Visor lacks Android's Nearby devices permission. | | `cut_off` | The glasses went away halfway, for example into the closed case. | | `damaged` | A file failed its checksum. | | `not_saved` | The gallery did not take the file. | | `failed` | Anything else. | The glasses send whatever else waits in their queue with an import. Each of those reaches the gallery too and comes on `imported`. ### Streams ```kotlin visor.captures.available.collect { } // each new capture on the glasses visor.captures.progress.collect { } // CaptureProgress(id, received, size), at most once per percent visor.captures.imported.collect { } // each capture that reaches the gallery visor.captures.failed.collect { } // CaptureFailure(id, failure) ``` `progress`, `imported` and `failed` cover every import, also those Visor starts on its own. ## JSON-RPC Over JSON-RPC a feed is subscribed like a topic: see [Feeds](json-rpc.md#feeds). | | Name | Params | Answer | |---|---|---|---| | Method | `camera.photo` | `import` (boolean, default true) | a `Capture` | | Method | `camera.video.start` | `kind` (`video` or `hyperlapse`, default `video`) | `{}` | | Method | `camera.video.stop` | none | `{}` | | Feed | `camera.live` | `quality`, `frameRate`, `audio` | records of `LiveHeader` and bytes | | Method | `camera.live.photo` | none | a `LivePhoto`: `mime`, and `data` in Base64 | | Method | `captures.list` | none | a list of `Capture` | | Method | `captures.import` | `id` | a `Capture` | | Topic | `captures.available` | none | `Capture` | | Topic | `captures.progress` | none | `CaptureProgress` | | Topic | `captures.imported` | none | `Capture` | | Topic | `captures.failed` | none | `CaptureFailure` | # Neural Band Scope: `band`, and `motion` for the band's motion. The Meta Ray-Ban Display with a paired Meta Neural Band. Without a band, `band.status` says `paired: false`, and the gestures, the motion and the buzz fail with `unsupported`. The band works with apps in one of two ways, which the person picks in Visor. Your code is the same for both. - **With the Display**, the usual way. The band talks to the Display, which uses its gestures to move around. Visor reads them through a session with the Display, so the Display acts on each gesture as usual and the app gets it too. A buzz borrows the band from the Display for a moment. - **Shared with the phone.** The band stays connected to the phone, and the Display reaches it through the phone, as Meta's relay lease does. Gestures come straight from the band, a buzz goes out at once, and the band's motion sensors are open to apps. The person turns it on in **Visor > Settings > Neural Band > Share with the phone**. When sharing does not work, Visor turns it off itself: the band goes back to the Display, and Visor's band settings say why. A band the person paired to the Display with Meta's app shows in `band.status` with the battery the Display reports, and `held: false`. Its gestures reach apps all the same. The buzz and sharing need Visor to hold the band's keys, which it does once the person sets the band up in Visor's band settings; until then the buzz fails with `unsupported` and the reason `not_set_up`. ## Gestures ```kotlin visor.band.gestures.collect { gesture -> when { gesture.finger == Finger.INDEX && gesture.derived == GestureAction.DOUBLE_TAP -> next() gesture.finger == Finger.MIDDLE && gesture.derived == GestureAction.HOLD -> back() } } ``` With the band on the Display, collecting `gestures` opens a session with the Display, which shows nothing on its screen. The first time, Visor asks the person once to set the glasses up for apps. The session needs the glasses worn, and the Display ends it when another app comes to the front; Visor opens it again while your app collects. Collect only while your app needs the gestures, for example while one screen is in front. With the band shared, the gestures come straight from it and no session is needed. A `Gesture`: | Field | Type | Meaning | |---|---|---| | `finger` | `Finger` | `thumb`, `index` or `middle`. | | `action` | `GestureAction` | What the finger did. With the band on the Display: `tap`, `double_tap`, `triple_tap`, `hold`, or a thumb swipe `up`, `down`, `left`, `right`. With the band shared, the band's own events: `press`, `release`, `tap`, `double_tap`, `click`, or a swipe. | | `derived` | `GestureAction?` | What the band made of the events: `tap`, `double_tap`, `hold`, `release`, or a swipe. With the band on the Display it is the same as `action`. | | `time` | `Long` | When the phone got it, in milliseconds since 1970. | Act on `derived` and your app works either way. The Display tells taps apart before it forwards them: a double tap comes as one `double_tap`, after its first `tap`. A middle-finger tap is the Display's back, so the Display goes back while your app gets the tap. Sets like `GestureAction` grow over time; treat values you do not know as nothing to act on. ## Status ```kotlin visor.band.status.collect { status -> } ``` A `BandStatus`: | Field | Type | Meaning | |---|---|---| | `paired` | `Boolean` | False when no band is paired with the glasses. | | `link` | `Link` | The phone's own link to the band: up while the phone borrows it for a buzz, and all along while the band is shared. | | `battery` | `Int?` | 0 to 100. | | `charging` | `Boolean?` | | | `leftHanded` | `Boolean?` | True when the band is set for the left wrist. | | `borrowed` | `Boolean` | True while the band sends its gestures to the phone's own link: during a lesson in Visor, or while an app reads the gestures of a shared band. | | `held` | `Boolean?` | True when Visor holds the band's keys, so the phone can buzz and share it. False for a band only the Display holds. | | `shared` | `Boolean` | True while the person shares the band with the phone. | ## Motion Scope: `motion`. A band shared with the phone. ```kotlin if (visor.band.status.first().shared) { visor.band.motion.collect { sample -> } // 30 samples a second visor.band.motion(rateHz = 60).collect { sample -> } } ``` The band's own sensors on the wrist, in the same `MotionSample` as [the glasses' motion](glasses.md#motion), with `source` `band`: `accelerometer` in m/s², `gyroscope` in rad/s, `magnetometer` as the band reports it, and the band's own fused `orientation`. The band sends each sensor on its own, so a sample carries the newest reading of each. `time` is when the phone got it, in nanoseconds; only the differences between samples mean something. Use it to point with the wrist, to know when the wearer raises the arm, or to count repetitions. Without sharing, or when sharing stops while you read, the stream ends with `unsupported` and the reason `not_shared`. Its message names the setting, so you can show it to the person. Reading the band's sensors without sharing would take the band from the Display. In a test, set `fake.band.shared = true`, wait for `fake.band.motionRate`, then send readings with `fake.band.motion(ax, ay, az)` or a whole `MotionSample`. ## Vibration ```kotlin visor.band.buzz() // 400 ms at full strength: the wearer notices it visor.band.buzz(80.milliseconds, 0.5f) // 10 ms to 1 s, strength above 0 up to 1 ``` With the band on the Display, a buzz borrows the band for a moment and gives it back; with the band shared, it goes out at once. It fails with `not_connected` when the band's link does not come up in time, and with `refused` when the band turns it down. ## JSON-RPC | | Name | Params | Answer | |---|---|---|---| | Method | `band.status` | none | a `BandStatus` | | Method | `band.buzz` | `durationMs` (10 to 1000, default 400), `strength` (1 to 255, default 255) | `{}` | | Topic | `band.status` | none | `BandStatus` now, then on each change | | Topic | `band.gestures` | none | `Gesture` | | Topic | `band.motion` | `rateHz` (5 to 60, default 30) | a `MotionSample` with `source` `band` | Without a band, `visor simulate band.gestures` makes Visor send only the gestures you type. See [Testing](testing.md). # Display Scope: `display`. The Meta Ray-Ban Display. On glasses without a display every call fails with `unsupported`. ## Teleprompter The Display has a teleprompter: one script at a time, which it scrolls or shows a page at a time. ```kotlin visor.display.prompter.show("Slide 3: the numbers") visor.display.prompter.show(script, PrompterStyle { paged = true; sizeSp = 40.0 }) visor.display.prompter.close() ``` A second `show` replaces the text on the Display. `show` returns once the Display opened the teleprompter, and fails with `refused` when the Display turns it down, `timeout` when it does not answer, and `not_connected`. `PrompterStyle`: | Field | Default | Meaning | |---|---|---| | `sizeSp` | 35 | The text size on the Display. | | `speed` | 32 | The scroll speed, in the Display's own unit. 32 is its default. | | `paged` | false | A page at a time instead of scrolling. | | `followVoice` | false | Scroll as the wearer reads aloud, with the Display's own speech recognition. | The wearer can close the teleprompter on the Display; the VDK does not hear about it. Captions or translation opening on the Display also take its place. ## Layouts > [!NOTE] > Visor shows layouts through Meta's toolkit app on the Display, as Meta's Device Access Toolkit does. The first > time an app shows one, Visor asks the person once to set the glasses up for apps, and installs the toolkit app. > A video plays alone on the screen: the glasses fetch its URL through the phone's internet, so a short clip can > take a few seconds to start. ```kotlin visor.display.layout.show(Layout { text("Garage") { style = LayoutStyle.HEADING } row { gap = 16 button("open", "Open") button("close", "Close") { style = LayoutStyle.SECONDARY } } }) visor.display.layout.clicks.collect { click -> if (click.id == "open") openGarage() } ``` A layout fills the 600 by 600 screen, and a new `show` replaces it in full: there are no partial updates. Its root is a column; boxes nest. The elements are DAT's, with the same sizes and styles: | Element | Builder | Fields | |---|---|---| | Column, row | `column { }`, `row { }` | `gap`; `padding`, or `paddingTop`, `paddingBottom`, `paddingStart`, `paddingEnd`, in pixels; `align` along the box: `start`, `center`, `end`, `space_between`; `crossAlign` across it: `start`, `center`, `end`, `stretch`; `reverse` to lay the children out from the end; `wrap`; `card` to draw it on a card; `background`, a colour behind its children; `height` in pixels; `id` to make the whole box a click target | | Text | `text("...") { }` | `style`: `heading` (40 px, bold), `title` (24 px, bold, the first line of Meta's Briefing cards), `body` (28 px, the default), `meta` (22 px); `color` (below; `secondary` is the default for `meta`); `maxLines`, cut with an ellipsis; `align`: `start`, `center`, `end` | | Image | `image(url) { }`, `image(bitmap) { }` | An `https://` URL, which the glasses fetch through the phone; a `data:` URI; or on Android a `Bitmap`, sent inside the layout. Visor scales a `data:` image larger than the Display's 600 px a side. `size`: `fill` (the width it gets, the default) or `icon` (a 32 px square); `corners`: `none`, `small`, `medium`; `height` in pixels | | Icon | `icon(LayoutIconName.HEART) { }` | One of the Display's own icons (below); `style`: `filled` (the default) or `outline`; `color`, white unless set; `size` in pixels, 32 unless set | | Button | `button(id, label) { }` | `style`: `primary` (the default), `secondary`, `outline`; `icon`; `primary = true` for the button the Display focuses first | | Button bar | `buttons { button(...) }` | The Display's own bar of buttons; `align`: `start`, `center`, `end` | | Pages | `pager { page { }; page { } }` | Pages side by side that the wearer swipes through, with dots under them, as Meta's Weather and Stocks details: each page a card of the Display's own material, inset 24 px, and padded 24 px unless it sets its padding | | Chart | `chart(values) { }` | A line through the values (the default, from the lowest to the highest, with a wash under it) or `style = LayoutStyle.BARS` (bars from zero); `color`, blue unless set; `height` in pixels, 120 unless set. Visor draws it on the phone at the width it shows | | Video | `video(url) { }` | MP4 over `https://`, at most 400 pixels a side and 70,000 pixels in all; it fills the layout alone, as on DAT; `controls` to show play and pause | Every element also takes `grow` and `shrink`, its share of the free space along its box, and `alignSelf`, its own place across the box. `pinTop`, `pinBottom`, `pinStart` and `pinEnd` pin an element that many pixels in from its box's edges, inside the box's padding, over the box's other children: the big weather icon in the corner of Meta's Weather card is one. A colour is `LayoutColor.PRIMARY` (white), `SECONDARY` (light grey), `GREEN` (the green of a rise on Meta's Stocks card), `RED`, `BLUE`, `TRACK` (the grey under a progress bar), or any colour as `LayoutColor.rgb(0xFFC83D)`, which is `"#FFC83D"` on the wire. A Visor that does not know a colour uses the default. Icons are the Display's own, by name. `LayoutIconName.dat` holds the names DAT has, `LayoutIconName.briefing` the glyphs of Meta's Briefing cards (`SMALL_TRIANGLE_UP`, `SMALL_TRIANGLE_DOWN`, `CHART_LINE`, `TROPHY` and more), and `LayoutIconName.weather` the 14 weather illustrations of Meta's Weather card (`WEATHER_SUNNY`, `WEATHER_RAINY`, `WEATHER_NIGHT_CLEAR` and the rest), which are in colour and keep it unless a `color` is set; draw them at 48 px or more. Any other name from Meta's wearables icon set works too: a Display that lacks it fetches it from Meta. On the Display the three button styles look alike: Meta's renderer keeps only an icon's outline from them. A button bar shows its labels while they fit, and only its icons when they do not. To see a layout without the glasses, open the Display simulator that `visor fake` serves: see [Testing](testing.md#the-display-simulator). ```kotlin visor.display.layout.show(Layout { padding = 24 row { crossAlign = LayoutAlign.CENTER gap = 12 icon(LayoutIconName.HOUSE) text("Garage") { style = LayoutStyle.HEADING; grow = 1.0 } } text("Closed since 18:02") { style = LayoutStyle.META } buttons { button("open", "Open") { icon = LayoutIconName.ARROW_UP_SHALLOW_U; primary = true } button("lights", "Lights") { style = LayoutStyle.OUTLINE } } }) ``` `clicks` brings a `LayoutClick` with the `id` of each button or box the wearer clicks, to the app that showed the layout only. `state` says whether the layout shows, and why it went: the wearer went `back` (as on DAT, back ends the layout), another app `replaced` it, the app `closed` it, or the glasses `failed` to draw it. `video` brings the video's `starting`, `playing`, `paused`, `ended` and `failed`, with an `error` such as `invalid_url`. `close()` empties the screen. The Display dims a layout after 20 seconds and sleeps after 25, as it does any screen. The layout stays until the wearer goes back. In a test, `fake.display.layout` holds the layout shown, `fake.display.click(id)` clicks an element, `fake.display.back()` goes back, and `fake.display.video(LayoutVideoState.ENDED, null)` moves the video along. ## Cards on the Briefing > [!TIP] > [Briefing cards](briefing-cards.md) is the guide, with recipes: weather as Meta's, a stocks watchlist, a live timer > with buttons, photos. This section is the reference. The Display's home has a Briefing panel with Meta's glanceable cards: reminders, weather, sports and stocks. An app can keep cards of its own there, after Meta's, and give each one a detail that a click opens, full screen, as Meta's cards open theirs. A card stays, whether or not the app runs, until the app removes it, so the wearer sees it whenever they look at the Briefing. Meta's toolkit has nothing like it. ```kotlin visor.display.cards.set("pasta", "Kitchen timer", LayoutIconName.CLOCK) { header(LayoutIconName.CLOCK, "Kitchen timer") stat("4:00", "Pasta") progress(0.6) detail { header("Pasta") field("Started", "19:42") field("Ends", "19:52") chart(listOf(100.0, 96.0, 90.0, 82.0)) { style = LayoutStyle.BARS } } } visor.display.cards.remove("pasta") ``` A card is a layout, drawn in the Briefing's own card style and padded 24 px as Meta's cards are unless it sets its padding. Setting a card with the same id replaces it; the Display shows the change the next time it asks, every 15 seconds while the wearer looks at the Briefing. The name and the icon show in the Display's list of Briefing cards, where the wearer can turn a card off. On the card itself buttons and videos do nothing: a click on the card opens its detail. A card and its detail take at most 256 KiB as JSON (`BriefingCard.MAX_BYTES`), images inside them included; past that `set` fails with `invalid_params`, saying the size. A few photos from `image(bitmap)` fit; for more, give `https://` URLs, which cost only their length. A chart costs nothing here: Visor draws it. `detail { }` is the screen a click opens, padded as a card. `pages { page { }; page { } }` makes the detail a set of pages the wearer swipes through, as Meta's Weather (now, the next hours, the week) and Stocks (one page a stock). ### Building blocks The pieces of Meta's own cards, sized and coloured as the Display draws them: | Block | What it draws | |---|---| | `header(icon, "Title")`, `header("Title")` | The first line: a 24 px icon beside a bold 24 px title | | `stat("18°")`, `stat("18°", "Low 9°")` | A figure in large type, and a grey caption under it | | `change("+1.8%", rising = true)` | A green triangle up and green text, or a red triangle down and red text | | `field("Wind", "12 km/h")` | A grey label and its value on one line, the value at the end | | `progress(0.6)`, `progress(0.6, color)` | A bar filled that far, from 0 to 1 | | `chart(values)` | A line or bars (see the layout elements above) | They are rows and texts of the layout above, so they mix with anything else. A card that looks like Meta's Weather: ```kotlin visor.display.cards.set("weather", "Lake weather", LayoutIconName.CLOUD) { header(LayoutIconName.CLOUD, "Lake weather") stat("18°", "Low 9°") text("Partly cloudy") { style = LayoutStyle.META; maxLines = 1 } icon(LayoutIconName.WEATHER_PARTLY_CLOUDY) { size = 104; pinTop = 0; pinEnd = 0 } pages { page { header("Next hours") chart(listOf(18.0, 19.0, 18.0, 16.0, 14.0, 13.0)) } page { header("This week") field("Mon", "9° to 21°") field("Tue", "11° to 23°") } } } ``` ### Clicks in a detail Buttons, and boxes with an `id`, work in a card's detail: each click comes to the app that owns the card, on `clicks`, whether or not the app shows a layout. A `toast` on the element tells the wearer at once that it worked. ```kotlin visor.display.cards.set("pasta", "Kitchen timer", LayoutIconName.CLOCK) { header(LayoutIconName.CLOCK, "Kitchen timer") stat("4:00", "Pasta") detail { header("Pasta") buttons { button("add", "+1 minute") { icon = LayoutIconName.PLUS; toast = "One more minute" } button("stop", "Stop") { toast = "Stopped" } } } } visor.display.cards.clicks.collect { click -> if (click.card == "pasta" && click.id == "stop") timer.stop() } ``` To show the result of a click, set the card again: the detail shows the new state the next time the wearer opens it. In a test, `fake.display.clickCard("pasta", "stop")` clicks. ### Live cards A card can follow the app's state: `keep` sets the card again for each new state of a `Flow`, as long as it runs. ```kotlin scope.launch { visor.display.cards.keep("pasta", "Kitchen timer", LayoutIconName.CLOCK, timer.remaining) { left -> header(LayoutIconName.CLOCK, "Kitchen timer") stat("%d:%02d".format(left.inWholeMinutes, left.inWholeSeconds % 60), "Pasta") progress(1 - left / total) } } ``` States closer than a second apart are merged into the newest, and a state that draws the same card sends nothing. While an app has cards on it, the Display asks for the Briefing every 15 seconds as the wearer looks at it, so a live card keeps up within seconds. When `keep` ends, the card stays as its last state drew it, until `remove`. ### From Python, TypeScript and JSON-RPC The Python and TypeScript clients have the same pieces, building the JSON the Kotlin builder makes: ```python from visor_kit.layout import card, header, stat, change, page, chart await visor.call("display.cards.set", card( "stocks", "My stocks", header("My stocks", icon="chart_line"), stat("27,012.40"), change("+481.50 (+1.8%)", rising=True), icon="chart_line", pages=[page(chart([26.1, 26.4, 26.9, 27.0]))], )) ``` ```ts import { card, header, stat, change, page, chart } from "visor-kit/layout"; await visor.call("display.cards.set", card("stocks", "My stocks", [ header("My stocks", "chart_line"), stat("27,012.40"), change("+481.50 (+1.8%)", true), ], { icon: "chart_line", pages: [page([chart([26.1, 26.4, 26.9, 27.0])])] })); ``` On the wire a card is `{"id", "name", "icon", "layout": {"root": ...}, "detail": {"root": ...}}`, each element `{"type": "text", "text": "...", "style": "title", ...}` with the field names of the tables above. ## JSON-RPC | | Name | Params | Answer | |---|---|---|---| | Method | `display.prompter.show` | `text`, `style` (a `PrompterStyle`, optional) | `{}` | | Method | `display.prompter.close` | none | `{}` | | Method | `display.layout.show` | `root`: a `LayoutNode` tree | `{}` | | Method | `display.layout.close` | none | `{}` | | Method | `display.cards.set` | `id`, `name`, `icon` (default `bell`), `layout`, `detail` (a layout, optional) | `{}` | | Method | `display.cards.remove` | `id` | `{}` | | Topic | `display.layout.clicks` | none | `LayoutClick`: `id`, `time` | | Topic | `display.cards.clicks` | none | `CardClick`: `card`, `id`, `time` | | Topic | `display.layout.state` | none | `LayoutState`: `showing`, `reason`, `time` | | Topic | `display.video` | none | `LayoutVideoEvent`: `state`, `error`, `time` | ``` visor call display.prompter.show '{"text":"Hello","style":{"paged":true}}' ``` More of the Display (cards on its home, web apps that reach the VDK) is on the [roadmap](overview.md#roadmap). # Briefing cards Scope: `display`. The Meta Ray-Ban Display. The Display's home has a Briefing panel: Meta's glanceable cards for the weather, sports, stocks and reminders. An app can keep cards of its own there, after Meta's, in the same look. Each card can open a detail, full screen, with pages the wearer swipes through and buttons that reach the app. A card stays until the app removes it, whether or not the app runs, so a delivery, a timer or a score is a glance away all day. Meta's toolkit has nothing like it. This page is the guide, with recipes. The reference (every element, field and colour) is in [Display](display.md#cards-on-the-briefing). ## Design for a glance The wearer looks at a card for a second, through a screen that adds light to the world: - Start with `header(icon, "Title")`, as Meta's cards do: it tells the wearer what the card is before they read it. - One figure in large type (`stat`), and at most two lines under it. Cut long text with `maxLines = 1`. - Put the rest in the detail: `detail { }` for one screen, `pages { }` for several, a page a topic. - Black is clear on the Display, and the card's own background is the panel's. Use colour for meaning only: `change(..., rising = ...)` is green or red, `LayoutColor.SECONDARY` for what matters less. - A weather illustration (`LayoutIconName.WEATHER_*`) needs 48 px or more; the glyphs are sharp at 20 to 32 px. A big illustration pinned in the top corner (`pinTop = 0; pinEnd = 0`), as on Meta's Weather card, leaves the left for text. - Give the card a short `name` and an icon: the wearer sees them in the Display's list of Briefing cards, where they can turn a card off. ## Recipes The recipes are in the Android sample (`samples/android`, `Cards.kt`), behind its Briefing buttons. ### Weather, as Meta's A card with today's figure and an illustration, and three pages behind it: now, the next hours, the week. ```kotlin val card = BriefingCard("weather", "Lake weather", LayoutIconName.CLOUD) { header(LayoutIconName.CLOUD, "Lake weather") stat("18°", "Low 9°") text("Partly cloudy") { style = LayoutStyle.META; maxLines = 1 } icon(LayoutIconName.WEATHER_PARTLY_CLOUDY) { size = 96; pinTop = 0; pinEnd = 0 } pages { page { gap = 8 text("Brugnera") { style = LayoutStyle.META } stat("18°", "High 21°, low 9°") text("Partly cloudy, clearing tonight") } page { gap = 16 header("Next hours") row { align = LayoutAlign.SPACE_BETWEEN for ((hour, sky, temperature) in hours) { column { gap = 8 text(hour) { style = LayoutStyle.META; align = LayoutAlign.CENTER } icon(sky) { size = 48; alignSelf = LayoutAlign.CENTER } text("$temperature°") { align = LayoutAlign.CENTER } } } } chart(hours.map { it.third.toDouble() }) { height = 90 } } page { gap = 14 header("This week") for ((day, low, high) in week) { row { gap = 12 crossAlign = LayoutAlign.CENTER text(day) { style = LayoutStyle.META } text("$low°") { style = LayoutStyle.META } // The day's range on the week's, 5 to 25 degrees. row { grow = 1.0 height = 8 crossAlign = LayoutAlign.STRETCH background = LayoutColor.TRACK row { grow = (low - 5) / 20.0 } row { grow = (high - low) / 20.0; background = LayoutColor.BLUE } row { grow = (25 - high) / 20.0 } } text("$high°") } } } } } visor.display.cards.set(card) ``` The week's bars are rows with a `background`: any proportion is three rows that share a track by `grow`. ### A stocks watchlist A line a stock on the card, and a page a stock behind it, with the week as a line in green or red. ```kotlin BriefingCard("stocks", "My stocks", LayoutIconName.CHART_LINE) { header(LayoutIconName.CHART_LINE, "My stocks") for (stock in stocks) { row { align = LayoutAlign.SPACE_BETWEEN crossAlign = LayoutAlign.CENTER text(stock.name) { maxLines = 1; shrink = 1.0 } change("%+.1f%%".format(stock.percent), rising = stock.percent >= 0) } } pages { for (stock in stocks) { page { gap = 12 header(stock.name) stat(stock.price) change("%+.1f%%".format(stock.percent), rising = stock.percent >= 0) chart(stock.week) { height = 80; color = if (stock.percent >= 0) LayoutColor.GREEN else LayoutColor.RED } field("52 week high", stock.high) } } } } ``` ### A live kitchen timer, with buttons `keep` follows a `Flow`: here the time left, ticking each second. The detail has buttons, whose clicks come back on `clicks`, and a `toast` that tells the wearer at once. ```kotlin suspend fun kitchenTimer(visor: Visor, total: Duration): Unit = coroutineScope { val left = MutableStateFlow(total) val ticking = launch { while (left.value > Duration.ZERO) { delay(1.seconds) left.update { (it - 1.seconds).coerceAtLeast(Duration.ZERO) } } } val clicks = launch { visor.display.cards.clicks.collect { click -> if (click.card != "pasta") return@collect when (click.id) { "add" -> left.update { it + 1.minutes } "stop" -> ticking.cancel() } } } val card = launch { visor.display.cards.keep("pasta", "Kitchen timer", LayoutIconName.CLOCK, left) { remaining -> header(LayoutIconName.CLOCK, "Kitchen timer") stat("%d:%02d".format(remaining.inWholeMinutes, remaining.inWholeSeconds % 60), "Pasta") progress(1 - remaining / total) detail { header("Pasta") field("Left", "%d:%02d".format(remaining.inWholeMinutes, remaining.inWholeSeconds % 60)) buttons { button("add", "+1 minute") { icon = LayoutIconName.PLUS; toast = "One more minute" } button("stop", "Stop") { toast = "Stopped" } } } } } ticking.join() clicks.cancel() card.cancel() visor.display.cards.remove("pasta") } ``` The Display asks for the Briefing every 15 seconds while it shows it, so the countdown moves in steps there. An open detail keeps what it showed until the wearer opens it again. ### A photo from the app `image(bitmap)` on Android sends a photo inside the card, scaled to the Display: ```kotlin BriefingCard("parcel", "Parcel", LayoutIconName.CART) { header(LayoutIconName.CART, "Parcel") text("At your door by 14:00") detail { image(doorstepPhoto) { height = 240; corners = LayoutCorners.MEDIUM } field("Courier", "GLS") progress(0.8) } } ``` A card and its detail take at most 256 KiB (`BriefingCard.MAX_BYTES`). A few photos fit; for more, give `https://` URLs, which the glasses fetch through the phone. ## From Python, TypeScript and JSON-RPC The Python and TypeScript clients have the same pieces. The stocks card, from Python: ```python from visor_kit.layout import card, header, row, text, change, page, stat, chart stocks = [("NAS100/USD", "27,012.40", 1.8, [26.1, 26.4, 26.7, 27.0])] await visor.call("display.cards.set", card( "stocks", "My stocks", header("My stocks", icon="chart_line"), *[row(text(name, max_lines=1, shrink=1.0), change("%+.1f%%" % pct, rising=pct >= 0), align="space_between", cross_align="center") for name, _, pct, _ in stocks], icon="chart_line", pages=[page(header(name), stat(price), chart(week, height=80)) for name, price, _, week in stocks], )) async for click in visor.subscribe("display.cards.clicks"): print(click["card"], click["id"]) ``` From TypeScript: ```ts import { card, header, stat, change, page, chart } from "visor-kit/layout"; await visor.call("display.cards.set", card("stocks", "My stocks", [ header("My stocks", "chart_line"), stat("27,012.40"), change("+1.8%", true), ], { icon: "chart_line", pages: [page([chart([26.1, 26.4, 26.7, 27.0])])] })); ``` On the wire, from any language or the `visor` command (`visor card card.json`): ```json { "id": "stocks", "name": "My stocks", "icon": "chart_line", "layout": {"root": {"type": "column", "children": [ {"type": "text", "text": "My stocks", "style": "title", "maxLines": 1}, {"type": "text", "text": "27,012.40", "style": "heading"} ]}}, "detail": {"root": {"type": "column", "children": [ {"type": "pager", "children": [ {"type": "column", "children": [{"type": "chart", "values": [26.1, 26.4, 26.7, 27.0], "height": 80}]} ]} ]}} } ``` ## Testing a card - `visor fake`, then the Display simulator's Briefing screen: the cards in the panel's look, their details, pages and buttons. `visor --fake card card.json` puts a card there from a file. - In a JVM test, `fake.display.cards` holds the app's cards and `fake.display.clickCard("pasta", "stop")` clicks. - In the app, `fake.display.briefingPreview(context)` from `kit-testing` is a view of the Briefing. - On the glasses: open the Briefing on the Display's home. A new card shows within 15 seconds. ## How it behaves - `set` with the same id replaces the card. The card stays, across Visor restarts, until `remove`. - Visor serves the cards after Meta's own, and the wearer can turn any of them off in the Display's list. - On the card itself buttons and videos do nothing: a click on the card opens its detail. - Clicks in a detail reach only the app that set the card. An element's `toast` shows once Visor has the click. - A card and its detail take at most 256 KiB; `set` fails with `invalid_params`, saying the size, past that. # Web apps on the Display Scope: any. The Meta Ray-Ban Display, with a web app the person added in Visor. The Display runs web apps in its own browser. A web app reaches the VDK through that browser and the phone's Visor, with the TypeScript client's `Visor.connectGlasses()`: the same calls, topics and errors as everywhere else, from JavaScript on the glasses. Meta's toolkit has nothing like it. ```ts import { Visor, VisorError, onGlasses } from "visor-kit"; async function start() { if (!onGlasses()) return Visor.connectFake(); // on a computer, with `visor fake` try { return await Visor.connectGlasses(); } catch (e) { if (e instanceof VisorError && e.reason === "not_allowed") { showMessage("Allow this app on your phone: Visor > Settings > Web apps > this app > Visor access."); } throw e; } } const visor = await start(); visor.subscribe("band.gestures", {}, (gesture) => console.log(gesture)); await visor.call("voice.speak", { text: "Hello from the web app" }); ``` Build the client from `clients/typescript` and serve `dist/index.js` and `dist/glasses.js` with the page, or bundle them with the page's own code. ## What the person allows In Visor, **Settings > Web apps**, the person taps the app and picks what it may do under **Visor access**, one scope at a time. That turns on **Nearby Bluetooth devices** for the app too, which is how it reaches Visor. The first time, Visor reads the glasses' configuration before it saves, which can take a minute; the sheet says so while it waits. The browser asks the wearer once more before the page's first connection. Removing the app takes its access away. ## When it fails `connectGlasses()` rejects with a `VisorError`: | Code | Reason | What happened | What to do | |---|---|---|---| | `unsupported` | | The page is not in the Display's browser. | Check `onGlasses()` first, and use `connectFake()` or `connect()` elsewhere. | | `refused` | `not_allowed` | The glasses do not let this app reach Visor. | Tell the person where to allow it, as in the example. | | `visor_gone` | | Visor did not answer within `timeoutMs` (10 s by default). | Visor is not running, or the app has no Visor access yet. | After that, a call the person did not allow fails with `scope_denied`, as anywhere else. ## How it travels The Display's browser hands a page's `navigator.bluetooth.requestLEScan` to the phone, which runs the scan and sends back what it hears. Visor reads the scan's filter instead: - A message to Visor is a scan whose filter holds manufacturer data for company `0xFFFF`, starting with `VDK1`. - Visor's answers and events come back as advertisements from a device named `Visor`, with the same kind of data. - One long scan carries everything Visor sends. Each message to Visor is a short scan of its own. The client does all of this; a page uses `call`, `subscribe` and `feed` as anywhere else. Messages go in order. A large one, such as a layout with a photo, goes in pieces of 8000 bytes. Each piece is a scan, so a call takes tens of milliseconds more than over a socket: use `subscribe` rather than polling. A web app's origin is all Visor knows about it. Every page on that origin gets the same access, so allow only origins you trust. ## Tools for the assistant A web app can give the wearer's assistant tools of its own, with the browser's WebMCP API, while it is open: ```js document.modelContext.registerTool({ name: "add_to_list", description: "Add an item to the shopping list.", inputSchema: { type: "object", properties: { item: { type: "string" } }, required: ["item"] }, execute: async ({ item }) => ({ content: [{ type: "text", text: `Added ${item}.` }] }), }); ``` The person turns this on once, in Visor: **Settings > Web apps > Web app tools for your assistant**. Visor then hands the tools to the model the person set up in Visor, with their descriptions and schemas, and runs the calls the model makes. Give each tool a name that says what it does: the model reads it first. Without the setting, `document.modelContext` is missing: check for it before you register. ## Other parts of the web-app SDK The Display's browser gives web apps more than the VDK, from the phone Visor runs on: - `navigator.geolocation`, from the phone's location. - The Generic Sensor API with `sensorId: 'phone'`: the phone's accelerometer, gyroscope, magnetometer, linear acceleration, gravity and orientation, up to 100 readings a second. - Real Bluetooth scans of the devices around the wearer, for an app with **Nearby Bluetooth devices** on. The protocol behind each is on the [Display web apps](https://visor.macedonga.com/protocol/display-web-apps/) page of the protocol reference. # Voice and the assistant Visor answers "Hey Meta" with the person's own model. An app can take part in two ways: answer the questions that name it, and give the model tools to use while it answers. ## Questions that name the app Scope: `voice`. All glasses. ```kotlin visor.voice.requests(listOf("the garage", "my lights")).collect { request -> val text = when { "close" in request.text -> garage.close().let { "Closing the garage." } else -> "I can open or close the garage." } visor.voice.reply(request, text) } ``` While the app collects `requests`, every question that holds one of its phrases comes to the app instead of the model: "Hey Meta, ask the garage to close" comes for "the garage". The glasses then say the reply in their own voice, as they would the model's answer. - **Matching.** A question matches a phrase when it holds all the phrase's words, in order and next to each other. Case, accents and punctuation do not count. When phrases of several apps match, the longest phrase wins. Pick phrases nobody says by chance: "the garage" rather than "open". - **Time.** Reply within about 20 seconds; the glasses give up on a question after that. Without a reply, the glasses tell the wearer that the app did not answer. - **Follow-ups.** `reply(request, text, listen = true)` keeps the microphone open after the reply, as after a question. The follow-up is matched like any question: it comes to the app again only if it names a phrase. - **What else comes.** `request.intent` is what the glasses understood the question as, such as `IN:QUESTION`, when they said. `request.phrase` is the phrase that matched. A `VoiceRequest`: | Field | Type | Meaning | |---|---|---| | `id` | `String` | What `reply` answers. | | `text` | `String` | The question, as the glasses heard it. | | `phrase` | `String` | The app's phrase that matched. | | `intent` | `String?` | The glasses' reading of the question. | A reply that comes too late fails with `invalid_params` and reason `expired`. ### Started by voice An app that is not running can be started by its phrase, as DAT's voice invocations start one. Declare an activity with the VDK's voice action and a string array of phrases: ```xml ``` When a question holds one of these phrases and no running app holds it, Visor starts the activity with the question. Read it with `VoiceLaunch.request(intent)` in `onCreate` and `onNewIntent`, and reply as above, within about 20 seconds: ```kotlin VoiceLaunch.request(intent)?.let { request -> lifecycleScope.launch { Visor.connect(this@GarageActivity).voice.reply(request, "The garage is closed.") } } ``` Visor starts only apps the person granted the `voice` scope, so ask for it once, at setup. While the app collects `requests`, its running offer answers instead and nothing is started. Visor reads the declared phrases again every 30 seconds, so a newly installed app is heard within half a minute. ## Saying something unasked `speak` has the glasses say a line in their own voice now, with nobody asking: a timer that ran out, a door that opened, a message worth hearing. It needs the `voice` scope. ```kotlin visor.voice.speak("Your coffee is ready.") ``` Keep it short and rare: the wearer hears it over whatever they are doing. The text holds at most 500 characters. > [!NOTE] > Visor speaks through the glasses' data collection service, which starts for it and stops after a minute of quiet. > It works on the Meta Ray-Ban Display. ## What the glasses hear Scope: `microphone`. All glasses, on Android 13 or later. ```kotlin visor.voice.transcripts(TranscriptRequest { language = "en-US" }).collect { transcript -> println(transcript.text) } ``` `transcripts` turns what the glasses' microphones hear into text, one finished utterance at a time, while you collect it. Visor records the glasses as a Bluetooth headset, never the phone's own microphone, and the phone's speech recogniser makes the text: on the phone when it holds the language, otherwise as the phone's recogniser does it. Without a `language`, Visor hears in the phone's language. While transcripts run, the glasses act as a call headset, so music pauses as in a call. Android lets Visor record only while it may use the microphone: while your app is on screen, or Visor is. The VDK lends Visor your app's access when it binds, so keep collecting from a visible screen. From a computer, keep Visor on screen. The Meta Ray-Ban Display is no Bluetooth headset to the phone. There the Display hears the wearer itself, with the speech recognition of Meta's toolkit app, as DAT's speech does: in the Display's own language, which `language` cannot change, and each transcript says. The toolkit app must be on the screen while it listens. If your app shows no layout, the Display shows a Listening screen with your app's name, so the wearer knows who listens; going back on the glasses ends the stream with `closed`. Each `Transcript` has the `text`, the `language` it was heard in, the `time` the utterance ended, and the recogniser's `confidence` from 0 to 1 when it gives one. Several apps can listen at once, in one language: an app that asks for another language meanwhile fails with `busy`. Ask for `partials` to get the text while the wearer speaks, as DAT gives it. A partial transcript has `partial` true, and the next one replaces it; the finished utterance comes last with `partial` false: ```kotlin visor.voice.transcripts(TranscriptRequest { partials = true }).collect { t -> if (t.partial) caption.text = t.text else log.append(t.text) } ``` The stream ends with: | Code | Reason | Why | |---|---|---| | `unsupported` | `no_recognizer` | Android 12, or a phone without speech recognition. | | `unsupported` | `no_permission` | Visor may not record. The person allows the microphone in Visor's settings. | | `not_connected` | | The glasses are not connected over Bluetooth. | | `busy` | | Another app hears in another language. | | `invalid_params` | | `language` is not a BCP 47 tag. | | `failed` | `no_headset` | The glasses are no headset to the phone, or the recording stopped. | | `failed` | `recognizer_failed` | The phone's recogniser stopped, for example for a language it lacks, or the Display's failed. | | `closed` | `back` | On the Display, the wearer went back from the toolkit app. | ## Tools for the model Scope: `assistant`. All glasses. ```kotlin val tools = listOf( ToolSpec { name = "set_lights" description = "Turns the living room lights on or off." schema = buildJsonObject { put("type", "object") putJsonObject("properties") { putJsonObject("on") { put("type", "boolean") } } putJsonArray("required") { add("on") } } }, ) visor.assistant.tools(tools).collect { call -> val on = call.arguments.jsonObject["on"]?.jsonPrimitive?.boolean if (on == null) visor.assistant.fail(call, "Say whether to turn them on or off.") else visor.assistant.answer(call, if (lights.set(on)) "Done." else "The lights did not answer.") } ``` While the app collects `tools`, the model can call them, alongside the person's own MCP servers, the Display's own tools and those of a [web app](web-apps.md#tools-for-the-assistant) open on it. Each call comes as a `ToolCall`; answer it with `answer` or `fail` within 15 seconds. The model reads the text and words its answer to the wearer from it, so say what happened, not how to say it. - **Names** start with a letter, then letters, digits, `_` or `-`, 40 at most. When two apps offer the same name, the second one's gets a `_` added for the model; the app still receives its own name. - **Schemas** are JSON Schema objects. Without one, the tool takes no arguments. - **Descriptions** are what the model goes by to choose a tool. Say what it does and when it applies. - Visor's diary records each call with the app's name and whether it worked. ## The conversations Visor keeps what the wearer asks the assistant, and its answers, on the phone until the person deletes them. With the `conversations` scope an app can read them, and get each new one as it is answered: a journal, a search over everything asked, a sync to the person's own notes. ```kotlin visor.assistant.turns.collect { turn -> println("${turn.question} -> ${turn.answer}") } val lastWeek = visor.assistant.history(since = System.currentTimeMillis() - 7 * 86_400_000L, limit = 500) ``` An `AssistantTurn`: | Field | Type | Meaning | |---|---|---| | `id` | `String` | Stable: a turn asked again comes again with the same id and the new answer. | | `time` | `Long` | When it was asked, in milliseconds since the epoch. | | `question`, `answer` | `String` | The answer is empty when the turn failed. | | `failure` | `String` | Why it failed, for a log; empty when it did not. | | `source` | `TurnSource` | `glasses`, `phone` (Visor's Meta AI page) or `live_ai`. | | `conversation` | `String` | Turns asked close together share one, as do the turns of one Live AI session. | | `photo` | `Boolean` | True when the glasses sent a photo with the question. The photo stays in Visor. | | `tools` | `List` | What the assistant did with the person's tools: `server`, `what`, `ok`. | `history` returns the newest turns first, at most `limit` (1 to 500, 50 by default). ## Notifications on the Display Scope: `notify`. The Meta Ray-Ban Display. ```kotlin visor.notify.post("delivery", "Package", "Your delivery is at the door.") visor.notify.remove("delivery") ``` A second post with the same id replaces the first. Glasses without a display show no notifications of apps, so there `post` fails with `unsupported`. ## JSON-RPC | | Name | Params | Answer | |---|---|---|---| | Topic | `voice.requests` | `phrases` | `VoiceRequest` | | Method | `voice.reply` | `request`, `text`, `listen` (default false) | `{}` | | Method | `voice.speak` | `text` | `{}` | | Topic | `voice.transcripts` | `language`, `partials` (optional) | `Transcript`: `text`, `language`, `time`, `partial`, `confidence` | | Topic | `assistant.tools` | `tools`: a list of `{name, description, schema}` | `ToolCall` | | Method | `assistant.tools.answer` | `call`, `text`, `error` (default false) | `{}` | | Topic | `assistant.turns` | none | `AssistantTurn` | | Method | `assistant.history` | `since` (optional), `limit` (default 50) | a list of `AssistantTurn` | | Method | `notify.post` | `id`, `title`, `text` | `{}` | | Method | `notify.remove` | `id` | `{}` | From your computer, the `visor` command can play the app's part: ``` visor watch voice.requests '{"phrases":["the garage"]}' visor call voice.reply '{"request":"","text":"Closing the garage."}' ``` # Settings The glasses keep about 370 settings: the ones the Meta AI app shows, and many it does not. Visor reads them all when the glasses connect, and an app with the `settings` scope can read them and change the everyday ones. ```kotlin val volume = visor.settings.get("notification_volume") visor.settings.set("notification_volume", 100) visor.settings.set("head_nod_controls_enabled", true) visor.settings.list("Camera").forEach { println("${it.label}: ${it.value}") } visor.settings.changes.collect { setting -> } // each change, whoever made it ``` ## A setting A `GlassesSetting`: | Field | Type | Meaning | |---|---|---| | `key` | `String` | The glasses' own name, such as `wakeword_enabled`. Stable across releases. | | `kind` | `SettingKind` | `bool`, `int`, `float`, `string`, or `other` for an encoded value. | | `value` | `String?` | The value in force, as text: what was set, else the firmware's default. Null when Visor knows neither, and for passwords. | | `default` | `String?` | The firmware's default. | | `label` | `String?` | What the Meta AI app calls it, in English. Null for a setting it does not show. | | `group` | `String?` | Visor's group, such as `Camera`, `Sound` or `Meta AI`. | | `options` | `List` | For a choice, each value with its label. | | `range` | `SettingRange?` | For a number, its `min`, `max`, `step` and `unit`. | | `writable` | `Boolean` | Whether apps may change it. | | `pending` | `Boolean` | True from a write until the glasses report the setting again. | Values are text in the form the kind gives: `"true"`, `"3"`, `"1.5"`, or one of the options' values. `accepts(value)` says whether a value fits the setting's kind, options and range. ## Changing a setting `set` sends the value and returns the setting with `pending` true. The glasses take a moment, and Visor reads the setting back: `changes` then brings it with the new value and `pending` false. If the glasses keep the old value, `changes` brings the old value. | Failure | Code and reason | |---|---| | The key is not a setting Visor knows | `invalid_params`, `unknown_key` | | The value does not fit | `invalid_params`, `invalid_value` | | Apps may not change it | `refused`, `read_only` | | The glasses are not connected | `not_connected` | Apps may change the settings the Meta AI app shows as a switch, a choice, a slider, a number, a text or a language. Visor keeps the rest to itself and the person: updates, resets, heat protection, developer and logging switches, and every encoded setting. Passwords are never shown. ## Over JSON-RPC ``` visor call settings.get '{"key":"wakeword_enabled"}' visor call settings.set '{"key":"notification_volume","value":"100"}' visor call settings.list '{"group":"Camera"}' visor watch settings.changes '{"keys":["notification_volume"]}' ``` On `visor fake`, five settings stand in for the glasses' own: `wakeword_enabled`, `head_nod_controls_enabled`, `notification_volume`, `resolution_and_framerate`, and `cloud_ota_enabled`, which apps may only read. `FakeVisor.settings.change(key, value)` changes one as the wearer would. # Errors Every failed call, and every stream that ends badly, throws a `KitException` (in JSON-RPC, an error whose `data` is a `Failure`). Act on `code`. `message` is for logs: its wording changes between releases. ```kotlin try { visor.camera.takePhoto() } catch (e: KitException) { Log.w(TAG, "photo failed: ${e.code} ${e.reason}: ${e.message}") } ``` | Code | Meaning | What to do | |---|---|---| | `scope_denied` | The app does not hold the scope in `scope`. | Ask for it with `RequestScopes`. | | `unknown_method` | Visor does not know the method or topic. | Check the name. From the library this cannot happen: it raises `visor_too_old` first. | | `invalid_params` | The params do not fit the method. | Fix the call. | | `unsupported` | These glasses cannot do it: the prompter without a display, the band without a band. | Check `GlassesStatus.display`, or `BandStatus.paired`, before offering the feature. | | `not_connected` | The glasses, or the band, are not connected and did not connect in time. | Tell the person to put the glasses on, or take them out of the case. | | `busy` | Something else holds what this needs, for example a video is recording. | Try again later. | | `refused` | The glasses or the band turned the request down. | Usually nothing; see the message. | | `timeout` | The glasses did not answer in time. | Try again; if it keeps happening, the glasses may need a restart. | | `failed` | Anything else. `reason` can say more, for example why an import failed. | See the method's page. | | `visor_missing` | Visor is not installed. Raised by the library. | Point the person to Visor. | | `visor_too_old` | Visor is older than this call needs. Raised by the library. | Ask the person to update Visor, or check `visor.supports(...)` first. | | `visor_gone` | The connection to Visor broke during the call, for example because Visor restarted. Raised by the library. | Call again; the connection comes back on its own. | | `closed` | Your app closed the connection while the call ran. Raised by the library. | Nothing. | Some failures carry a `reason` that says what the person can change: | Reason | With | What to tell the person | |---|---|---| | `not_set_up` | `unsupported` | Set the band up in Visor's band settings: only the Display holds it now. | | `not_shared` | `unsupported` | Turn on Visor > Settings > Neural Band > Share with the phone, for `band.motion`. | | `not_allowed` | `refused` | Allow the web app in Visor > Settings > Web apps > the app > Visor access. | Visor adds codes and reasons over time. Treat a code you do not know as `failed`. # Testing Wearing the glasses for every change is slow. The VDK has three tools so you mostly do not have to: - **The `visor` command**, on your computer: talk to Visor on the phone, watch what your app does, and stand in for the band or the glasses. - **`FakeVisor`**, in your tests: Visor inside your app's process, driven by your test code. - **`visor fake`**, on your computer: FakeVisor behind the same doors Visor opens on a phone, for scripts, web pages, MCP clients and the `visor` command, with no phone at all. ## The visor command ### Install The command needs Java 17 or later, and adb to reach a phone. Android Studio brings both. The fast way, one line that installs the latest release and puts `visor` on your PATH. Run it again to update. ``` curl -fsSL https://visor.macedonga.com/install.sh | sh # macOS and Linux irm https://visor.macedonga.com/install.ps1 | iex # Windows, in PowerShell ``` The script on macOS and Linux puts the command in `~/.visor` and starts it from `~/.local/bin/visor`; the one on Windows puts it in `%LOCALAPPDATA% isor` and adds its `bin` folder to your PATH. Both say so if Java is missing. The manual way: 1. Download `visor-.zip` from the [releases](https://github.com/macedonga/visor-kit/releases). 2. Unzip it where you keep tools. 3. Put its `bin` folder on your PATH. From a clone of visor-kit, `./gradlew :cli:installDist` builds it in `tools/cli/build/install/visor`. `visor --version` says which version you have. ### Set up the phone 1. Connect the phone with adb, over USB or wireless debugging: `adb devices` must list it. With more than one device, pick one with `visor -s SERIAL ...`. 2. In Visor, turn on Developer mode, in Settings, Advanced settings, Tools. It opens the socket the command uses; apps do not need it. `visor` finds adb through `ANDROID_HOME` or the PATH, forwards a free port to Visor's socket, and removes the forward when it ends. Only adb can reach the socket: Visor checks that each connection comes from adb's shell user. ### Commands | Command | What it does | |---|---| | `visor status` | Visor's version, the glasses, the band, and the apps that use the VDK. | | `visor methods` | The methods and topics this Visor answers. | | `visor call METHOD [JSON]` | Runs any method and prints its answer. | | `visor watch TOPIC...` | Prints each event of one or more topics as it comes, until Ctrl+C. | | `visor live FILE [JSON]` | Saves the camera live as raw HEVC or H.264, the sound beside it as `FILE.aac`, until Ctrl+C. | | `visor live-photo FILE` | Takes a photo while the camera is live and saves the HEIC. | | `visor trace` | Prints every call, answer and event between every app and Visor, with timings. | | `visor simulate TOPIC...` | While it runs, apps subscribed to these topics get only the events you type. | | `visor inject TOPIC EVENT` | Sends one made-up event to the apps subscribed to a topic. | | `visor record FILE [TOPIC...]` | Saves each event of the topics to a file, until Ctrl+C. | | `visor replay FILE [--loop]` | Plays a recording into the apps at its own pace. | | `visor apps` | The apps that use the VDK, their scopes, and whether they are connected. | | `visor grant PACKAGE SCOPE...` | Grants scopes without the consent screen. | | `visor revoke PACKAGE SCOPE...` | Takes scopes back. | | `visor fake` | Runs FakeVisor on this computer. See [Without a phone](#without-a-phone-visor-fake). | Add `--json` for JSON only, one value per line, to pipe into `jq` or a script. ### Watch the camera `visor live` films with the glasses until Ctrl+C. Write to a file, or to standard output for a player: ``` visor live - | ffplay -fflags nobuffer - visor live walk.hevc '{"quality":"full_hd","frameRate":30,"audio":true}' ffmpeg -i walk.hevc -i walk.hevc.aac -c copy walk.mp4 ``` In PowerShell 5, pipe through `cmd /c` or save to a file: PowerShell's pipe changes binary data. ### See what your app does Run `visor trace` in one terminal and use your app: ``` $ visor trace Tracing the VDK. Ctrl+C stops. 16:02:11.204 com.example.app call kit.hello #1 {"protocol":1,"kit":"0.1.0"} 16:02:11.209 com.example.app result kit.hello #1 4 ms {"protocol":1,"visor":"2.4.0",...} 16:02:11.240 com.example.app subscribe band.gestures #2 16:02:13.811 com.example.app event band.gestures #2 {"finger":"index","action":"tap","derived":"double_tap",...} 16:02:15.002 com.example.app call camera.photo #3 {"import":true} 16:02:18.517 com.example.app result camera.photo #3 3515 ms {"id":"...","type":"photo","uri":"content://..."} ``` ### Stand in for the band or the glasses `simulate` takes the real source out of the topics you name: apps subscribed to them get only what you type, and the band is not borrowed. It ends when you press Ctrl+C or close the input, and then the real source comes back without your app noticing. ``` $ visor simulate band.gestures Simulating ["band.gestures"]: apps get only what you type here. Ctrl+C or end of input stops. index double_tap middle hold thumb up ``` For `band.gestures`, a line is `FINGER ACTION`, or `FINGER ACTION DERIVED` when the band's reading should differ from the action. For `glasses.inputs`, a line is `SOURCE ACTION`, such as `touchpad swipe_forward`. For any other topic, a line is the event's JSON: ``` $ visor simulate glasses.status {"paired":true,"link":"connected","battery":4,"worn":true} {"paired":true,"link":"disconnected"} ``` With several topics, start each line with its topic. `visor inject` sends a single event, simulated or not, to the apps subscribed now: ``` visor inject band.gestures index tap visor inject glasses.status '{"paired":true,"link":"connected","battery":15}' ``` Visor checks every event against the topic's type before it sends it, so a typo fails here, not in your app. Made-up events go to apps only: the glasses, the band and Visor's own screens never see them. ### Script the glasses `call` runs any method, which makes the command a quick way to script the glasses: ``` visor call camera.photo '{"import":false}' visor call band.buzz '{"durationMs":120,"strength":128}' visor call display.prompter.show '{"text":"Two minutes left"}' visor --json watch band.gestures | jq -r .event.derived ``` ### Record once, replay anywhere `record` saves what the wearer does, with the state it happens in, so you can play it back while you work on your app, on the phone or on `visor fake`: ``` visor record walk.jsonl # glasses.status, glasses.inputs, band.status, band.gestures visor record taps.jsonl band.gestures # only these topics visor replay walk.jsonl # into the apps on the phone visor --fake replay walk.jsonl --loop # into visor fake, over and over ``` In a JVM test, `FakeVisor.replay` plays a recording into your app the same way, paced or all at once: ```kotlin fake.replay(File("src/test/resources/walk.jsonl").reader()) // at its recorded pace fake.replay(File("src/test/resources/taps.jsonl").reader(), paced = false) ``` Each line is one event, `{"t": 1520, "topic": "band.gestures", "event": {...}}`, with `t` in milliseconds from the start of the recording. While a replay runs, its topics are simulated: apps get the recorded events, at their recorded times, and nothing from the real glasses or band. The events keep the times they were recorded with. A recording is plain JSON, so a test can read it too, or you can write one by hand. ## FakeVisor `FakeVisor` (`com.macedonga.visor:kit-testing`) is Visor in your process. It answers every method from state your test sets, so tests run without a phone, glasses or band. It runs the same server code as Visor, so scopes, streams and errors behave the same way. ```kotlin import com.macedonga.visor.kit.testing.FakeVisor import com.macedonga.visor.kit.testing.connect @Test fun warnsOnLowBattery() = runBlocking { val fake = FakeVisor() val visor = fake.connect() fake.glasses.battery = 4 val screen = BatteryWarning(visor) screen.refresh() assertEquals("Charge the glasses", fake.display.prompter.value) assertEquals(listOf("kit.hello", "display.prompter.show"), fake.calls) fake.close() } ``` It runs as a plain JVM unit test: no emulator, no Robolectric. Calls answer from the fake's state at once, and streams of state send it first, so a test needs no waiting. Events you send while nothing collects are dropped, as with the real band: start collecting before you call `fake.band.gesture(...)`. What the fake offers: | Part of the fake | What it offers | |---|---| | `fake.granted` | The scopes the fake person granted; all at first. Taking one back ends the streams that need it. | | `fake.glasses` | `paired`, `connected`, `display`, `model`, `battery`, `charging`, `worn`, `folded`, `thermal`: each change reaches `glasses.status`. `add(type)` puts a capture on the fake glasses, and `add(type, bytes, mime)` one with those bytes. `input(source, action)` sends a touch or a press. | | `fake.band` | `paired`, `battery`, `charging`, `leftHanded`, `held`, `shared`; `gesture(...)` sends a gesture; `motion(...)` sends a reading for `band.motion` once `motionRate` is set; `buzzes` lists the buzzes asked for. | | `fake.display` | `prompter`: the script on the teleprompter, null while closed; `layout`: the layout shown; `click(id)` taps one of its elements; `back()` goes back; `video(state, error)` moves the video along; `cards`: the app's Briefing cards; `clickCard(card, id)` clicks in a card's detail. | | `fake.notify` | `shown`: the app's notifications on the Display, by id. | | `fake.settings` | `all`: five sample settings by key; `change(key, value)` changes one as the wearer would. | | `fake.assistant` turns | `turn(question, answer)` adds a turn the app gets on `assistant.turns`; `history` lists them. | | `fake.assistant` | `tools` the app offers; `call(tool, arguments)` calls one as the model would and returns the answer. | | `fake.voice` | `ask(question)` asks as the wearer would and returns the app's reply; `say(text)` is an utterance the glasses hear, for `voice.transcripts`; `listening` counts the apps that collect transcripts. | | `fake.live` | `watching` and `config` while the app collects `camera.live`; `sendVideoConfig`, `sendVideo`, `sendAudioConfig`, `sendAudio` send samples; `stop()` ends it as the person would; `photos` counts live photos, and `photo(bytes, mime)` sets what they hold. | | `fake.calls` | The name of every method the app called, in order. | In an app with `kit-testing`, the fake's imported captures and live photos open with the app's `ContentResolver`, as Visor's do: their `content://` URIs point at a provider that `kit-testing` adds to the app. Without bytes given, each holds a 1x1 PNG. In a plain JVM test the URIs open nothing. The fake films nothing: your test sends each live sample. Wait until the app watches, or the sample is lost: ```kotlin fake.live.watching.first { it } fake.live.sendVideoConfig(parameterSets) fake.live.sendVideo(keyframe, ptsUs = 0, key = true) ``` `FakeVisor` itself is plain Kotlin, in `com.macedonga.visor:kit-fake`: a JVM test or tool can drive it without Android. `kit-testing` adds `connect()`, which needs the Android library. ### Recorded feeds DAT's mock plays a video file as the camera and a CSV as the motion sensors; FakeVisor does both: ```kotlin launch { File("walk.csv").reader().use { fake.glasses.playMotion(it, loop = true) } } launch { File("street.h265").inputStream().use { fake.live.play(it, "video/hevc", 504, 896, 24, loop = true) } } ``` The CSV is DAT's: no header, 14 numbers a row, the time in nanoseconds, the accelerometer, gyroscope and magnetometer as x, y, z, and the orientation as w, x, y, z. The camera takes an Annex-B H.264 or H.265 stream; ffmpeg makes one from any clip without re-encoding: `ffmpeg -i clip.mp4 -c:v copy -bsf:v hevc_mp4toannexb street.h265` (`h264_mp4toannexb` for H.264). It starts once an app watches the live camera. ## Without a phone: visor fake `visor fake` runs FakeVisor on your computer, behind the doors Visor opens on a phone: - JSON-RPC over WebSocket on `ws://127.0.0.1:7333/rpc`, as Visor's [network door](network.md), with no token; - MCP on `http://127.0.0.1:7333/mcp`, for an assistant on your computer; - the `visor` command, with `--fake` in place of `-s SERIAL`; - the Display simulator on `http://127.0.0.1:7333/display`. Every scope is granted. Its console plays the glasses, and prints what apps do to them: ``` $ visor fake FakeVisor on ws://127.0.0.1:7333/rpc and http://127.0.0.1:7333/mcp; visor --fake on port 7334. Type help. The Display simulator: http://127.0.0.1:7333/display 00:23:47 Display: the prompter shows "Hello from the fake" 00:23:47 Band: a buzz of 400 ms gesture index double_tap battery 12 ask what is on the calendar today 00:24:02 Voice: the glasses say "Two meetings, the first at ten." ``` | Type | What happens | |---|---| | `battery N` | The glasses' battery, 0 to 100. | | `charging`, `worn`, `folded`, `connected`, `display` with `on` or `off` | That part of `glasses.status`. | | `thermal LEVEL` | `normal`, `light`, `moderate`, `severe` or `critical`. | | `input SOURCE ACTION` | A touch or a press for `glasses.inputs`, such as `touchpad tap`. | | `motion AX AY AZ` | An accelerometer reading in m/s² for `glasses.motion`, such as `motion 0 9.8 0`. | | `motion-csv FILE` | DAT's 14-column motion CSV, played over and over at the pace of its timestamps. | | `camera FILE WIDTHxHEIGHT FPS`, `camera off` | A recorded `.h264` or `.h265` stream as the live camera, looped. | | `gesture FINGER ACTION` | A band gesture for `band.gestures`, such as `index double_tap`. | | `band-shared on`, `band-shared off` | The band shared with the phone, as the person sets it in Visor. | | `band-motion AX AY AZ` | An accelerometer reading on the wrist for `band.motion`, which needs the band shared. | | `photo`, `video` | A new capture on the glasses. | | `ask QUESTION` | The wearer asks; the app whose phrase the question holds answers. | | `say TEXT` | The glasses hear TEXT, for `voice.transcripts`. | | `tool NAME [JSON]` | The assistant calls an app's tool. | | `turn QUESTION / ANSWER` | The wearer asks the assistant and it answers, for `assistant.turns`. | | `click ID` | A tap on an element of the layout on the Display. | | `back` | The wearer goes back: the layout goes. | | `video STATE [ERROR]` | The layout's video moves to `playing`, `paused`, `ended`, or `failed` with an error. | | `grant SCOPE...`, `revoke SCOPE...` | The scopes the fake person granted. | From another terminal, the command works on the fake as on a phone: `visor --fake trace`, `visor --fake watch band.gestures`, `visor --fake call display.prompter.show '{"text":"Hi"}'`. The Python client has `visor_kit.connect_fake()` and the TypeScript one `Visor.connectFake()`. `--port N` moves the doors to N and the command's port to N + 1; `--lan` opens the doors to the local network. ### The Display simulator Open `http://127.0.0.1:7333/display` while `visor fake` runs. It draws the layout an app shows on the fake Display at 600 by 600, by the same rules Visor uses to send it to the glasses, and it plays the wearer: - the arrow keys move the focus as the touchpad and the band do, Enter selects, Backspace or Esc goes back; a click on an element selects it too. The app gets the clicks and the `back` as from the glasses, and each move on `glasses.inputs` as `nav_*`, `select` and `back` from the touchpad; - the Display adds light, so its black is clear: the page draws it over a background (black, a room, a street in daylight, a night street, or your webcam) with a brightness slider; - a checkbox makes it dim and sleep after 20 and 25 seconds, as the Display does; - a video plays, and its pauses, end and failures reach the app on `display.video`; - Screenshot and Record save the display as a PNG or a WebM (Chrome crops them to the display); - Screen switches to the Briefing: the app's cards in the panel's material, one under another. Enter or a click opens a card's detail, its pages swipe with the arrows, its buttons click as on the glasses (the app gets them on `display.cards.clicks`, and an element's `toast` shows), and Backspace closes it. DAT's icons are drawn with the nearest Material Symbol; the Briefing glyphs and the weather illustrations come from Meta's own icon CDN, as the Display fetches them. Check a layout on the glasses before you ship it. `visor --fake card card.json` puts a card on the fake Briefing from a file, as `visor card` does on a phone, and `visor --fake call METHOD @params.json` takes any method's params from a file. With `visor fake --port 7400`, give the same port to the command: `visor --fake=7400 status`. In the app itself, `fake.display.preview(context)` from `kit-testing` is a view with the same renderer: it draws what the fake Display shows and clicks what you tap, as DAT's `createPreviewView()` does. `fake.display.briefingPreview(context)` draws the Briefing with the app's cards: a tap opens a card's detail, a tap there clicks, and a double tap goes back. ## Web apps on the Display `Visor.connectGlasses()` works only in the Display's browser. Elsewhere, connect the same page to `visor fake` with `Visor.connectFake()`, and pick with `onGlasses()`: see [Web apps on the Display](web-apps.md). ## The samples - `samples/android` is an app with a button for every capability. Build it with `./gradlew :sample:installDebug`, and use it with `visor trace` to see a whole exchange. - `samples/python` holds small scripts on the Python client: a teleprompter pager turned by the band, a journal of the assistant's answers, and the glasses' settings. - `samples/web` is a glasses console in one page, on the network door, with no build step. # JSON-RPC The VDK speaks JSON-RPC 2.0, so any language can use it, through two doors: Visor's adb socket, described here, and [the network door](network.md), a WebSocket with the same messages. ## Reaching the socket Visor listens on the abstract Unix socket `visor-kit` while Developer mode is on, and lets in only adb's shell user. Forward a port to it: ``` adb forward tcp:7333 localabstract:visor-kit ``` Then connect to `127.0.0.1:7333`. Remove the forward when done: `adb forward --remove tcp:7333`. If nothing answers and the connection closes at once, Visor is not running or its Developer mode is off. Developer mode opens only this socket: apps never need it. The shell holds every scope, and also has the `shell.*` methods: `shell.apps`, `shell.grant`, `shell.revoke`, `shell.simulate`, `shell.inject`, and the topic `shell.trace`. ## Framing One JSON object per line, UTF-8, ending in `\n`, both ways. Requests carry an `id` you choose; answers carry the same `id`. Several requests can be in flight at once. ``` -> {"jsonrpc":"2.0","id":1,"method":"kit.hello","params":{"protocol":1,"kit":"my-script 1.0"}} <- {"jsonrpc":"2.0","id":1,"result":{"protocol":1,"visor":"2.4.0","granted":["state","camera","band","display"],"methods":[...],"topics":[...]}} -> {"jsonrpc":"2.0","id":2,"method":"glasses.status"} <- {"jsonrpc":"2.0","id":2,"result":{"paired":true,"link":"connected","battery":82,"worn":true}} ``` Say `kit.hello` first. Params are by name, as an object; a method without params takes `{}` or none. ## Errors ``` <- {"jsonrpc":"2.0","id":3,"error":{"code":-32000,"message":"The glasses are not connected.","data":{"code":"not_connected","message":"The glasses are not connected."}}} ``` `error.data` is the VDK's `Failure`, with `code`, `message`, and sometimes `scope` and `reason`; see [Errors](errors.md). `error.code` is `-32601` for an unknown method, `-32602` for params that do not fit, `-32700` for a line that is not JSON-RPC, and `-32000` for every other failure. ## Streams JSON-RPC has no streams, so the VDK adds three methods and two notifications: | | Name | Params | Answer | |---|---|---|---| | Method | `kit.subscribe` | `topic`, `params` (optional) | `{"subscription": ID}`, where ID is the request's own id | | Method | `kit.unsubscribe` | `subscription` | `{}` | | Method | `kit.cancel` | `id` of a running call | `{}` | | Notification | `kit.event` | `subscription`, `event` | | | Notification | `kit.ended` | `subscription`, `failure` (absent when the stream simply ended) | | ``` -> {"jsonrpc":"2.0","id":4,"method":"kit.subscribe","params":{"topic":"band.gestures"}} <- {"jsonrpc":"2.0","id":4,"result":{"subscription":4}} <- {"jsonrpc":"2.0","method":"kit.event","params":{"subscription":4,"event":{"finger":"index","action":"tap","derived":"tap","time":1793111111111}}} ``` The answer to `kit.subscribe` always comes before the subscription's first event. ## Feeds A feed, such as `camera.live`, streams binary records. Over JSON-RPC it is subscribed with `kit.subscribe` like a topic, and each event is a record: its `header`, and its `payload` in Base64. ``` -> {"jsonrpc":"2.0","id":5,"method":"kit.subscribe","params":{"topic":"camera.live","params":{"quality":"low"}}} <- {"jsonrpc":"2.0","id":5,"result":{"subscription":5}} <- {"jsonrpc":"2.0","method":"kit.event","params":{"subscription":5,"event":{"header":{"kind":"video","codec":"video/hevc","ptsUs":0,"config":true,"width":360,"height":640},"payload":"AAAAAUAB..."}}} ``` A keyframe in Base64 runs to hundreds of kilobytes at `full_hd`: allow lines that long. The Python and TypeScript clients have `feed`, which decodes the payload. ## The contract [`protocol/api/openrpc.json`](https://github.com/macedonga/visor-kit/blob/main/protocol/api/openrpc.json) is an [OpenRPC](https://open-rpc.org) document of every method, its params and its answer, with every type under `components.schemas`. The VDK's own additions: - `x-scope`: the scope a method or topic needs. - `x-since`: the protocol version that added it. - `x-shell`: answered only to the adb shell. - `x-topics`: the topics, each with its `params` and its `event`. - `x-feeds`: the feeds, each with its `params` and its record's `header`. Generate a client from it, or read it as the reference. It is written by the build from the Kotlin contract, so it is always exact. ## A client in Python ```python import json, socket, subprocess subprocess.run(["adb", "forward", "tcp:7333", "localabstract:visor-kit"], check=True) sock = socket.create_connection(("127.0.0.1", 7333)) lines = sock.makefile("r", encoding="utf-8") def call(id, method, params=None): sock.sendall((json.dumps({"jsonrpc": "2.0", "id": id, "method": method, "params": params or {}}) + "\n").encode()) while True: message = json.loads(lines.readline()) if message.get("id") == id: return message call(1, "kit.hello", {"protocol": 1, "kit": "python-example"}) print(call(2, "glasses.status")["result"]) call(3, "kit.subscribe", {"topic": "band.gestures"}) for line in lines: message = json.loads(line) if message.get("method") == "kit.event": print(message["params"]["event"]) ``` # The network door Visor can serve the kit over the network: JSON-RPC over a WebSocket for programs, and MCP for AI agents. It is off until the person turns it on, and every client needs a token the person made for it. ## Turning it on In Visor, open Settings, Advanced settings, Network access: 1. Turn on **Open to clients**. Visor listens on `127.0.0.1:7333`, so only programs on the phone itself reach it (Termux, for example). 2. Turn on **From the local network** to let computers on the same Wi-Fi in too. The row shows the phone's address. 3. **Add a client**: give it a name and its scopes. Visor shows the client's token once; copy it then. Visor keeps only its hash. Remove a client to take its access back: its streams end at once. To reach the door from a computer without the local network, forward the port over adb: `adb forward tcp:7333 tcp:7333`. ## JSON-RPC over WebSocket `ws://
:7333/rpc`, with the token as `Authorization: Bearer `, or as `?token=` where a client cannot set headers, as in a browser. The messages are the same as on the adb socket: see [JSON-RPC](json-rpc.md). The client holds only its own scopes, and the `shell.*` methods are not there. ## MCP for AI agents `http://
:7333/mcp` is an MCP server over Streamable HTTP, with the same token. Each method the client may call is a tool, named with `_` for `.`: `glasses_status`, `camera_photo`, `display_prompter_show`, and so on. A tool's answer is the method's result as JSON text. Streams are not tools; use JSON-RPC for them. With Claude Code: ``` claude mcp add --transport http visor http://127.0.0.1:7333/mcp --header "Authorization: Bearer " ``` To try an agent with no phone, run `visor fake` and point the agent at it, with no token: ``` claude mcp add --transport http visor-fake http://127.0.0.1:7333/mcp ``` Give an agent's client only the scopes it needs. An agent with `camera` takes photos when it decides to; the glasses' capture light still turns on for each one. ## Webhooks For a service that cannot hold a connection open, such as a home automation hub, Visor can post events instead. When you add a client, give it a **webhook URL** and pick its topics: those its scopes allow, without params. While the network door is open, Visor subscribes as that client and posts each event, in order, as JSON: ```json {"client": "home", "topic": "glasses.status", "event": {"paired": true, "link": "connected"}, "time": 1794072397747} ``` Each post carries `X-Visor-Signature: sha256=`, the HMAC-SHA256 of the body. The key is the hex SHA-256 of the client's token, which Visor keeps instead of the token: ```python key = hashlib.sha256(token.encode()).hexdigest().encode() valid = hmac.compare_digest("sha256=" + hmac.new(key, body, hashlib.sha256).hexdigest(), signature) ``` Visor waits 5 seconds for an answer and does not retry. When the receiver keeps failing, Visor's event log says so once a minute. ## Clients for other languages - **Python:** [`clients/python`](https://github.com/macedonga/visor-kit/tree/main/clients/python). `connect_adb()` or `connect_network(url, token)`, then `await visor.call(...)` and `async for event in visor.subscribe(...)`. - **TypeScript:** [`clients/typescript`](https://github.com/macedonga/visor-kit/tree/main/clients/typescript), for browsers and Node 22 or later. `await Visor.connect(url, token)`, then `visor.call(...)` and `visor.subscribe(...)`. ```typescript const visor = await Visor.connect("ws://192.168.1.20:7333/rpc", token); visor.subscribe("glasses.status", {}, (status) => console.log(status)); ``` ## Raw access Scope: `raw`. For research into the glasses' protocol: send a message to any service of the glasses and read what a service sends, as they are. | | Name | Params | Answer | |---|---|---|---| | Method | `raw.send` | `service`, `type`, `payload` (Base64) | `{}` | | Topic | `raw.messages` | `service` | `{service, type, payload, time}` | Visor answers only while its Developer mode is on, and refuses the services it keeps to itself: identity and ownership (36), the link's authentication (54 and 79) and firmware updates (29). A wrong message can still upset the glasses until they restart. # Compatibility People update Visor and your app at different times, so the VDK promises that either one can be the older side. ## The promise - An app built against any VDK release works with every later Visor. - An app built against a newer VDK works with an older Visor, for everything that Visor has. `visor.supports(...)` tells you what it has, and anything newer fails with `visor_too_old` instead of misbehaving. - Upgrading the VDK in your app never breaks code that compiled against an earlier release of the same major version, at the source level or the binary level. - A method keeps doing what it did. New behavior comes as a new method, or as an optional field that asks for it. ## Versions - **Library versions** (`kit`, `kit-protocol`, `kit-testing`) follow semantic versioning. - **The protocol version** (`KitProtocol.VERSION`) is one number that each release adding a method, topic, feed or field raises by one. It only goes up. Each method, topic and feed records the version that added it in `since`. ## What changes, and how | Change | Allowed? | |---|---| | A new method, topic, feed, type or scope | Yes, with a higher protocol version. | | A new optional field in params, a result or an event | Yes. | | A new value in an open set (error codes, scopes, gesture actions, capture types) | Yes. Your code must treat values it does not know as unknown. | | Removing or renaming a method, topic, feed, type or field | No. | | Changing a field's type, or whether it is required | No. | | Removing public Kotlin API | Only in a major release, after a deprecation in two minor releases. | ## Writing code that lasts - Use `else` in every `when` over `ErrorCode`, `GestureAction`, `CaptureType`, `Finger` or `Link`. They are value classes, not enums, because they grow. - Treat a `null` field as "not known". Most fields are null when the glasses are not connected. - Check `visor.supports(...)` before using a method added after the VDK version you require of Visor. ## How it is enforced The build checks every change against the last release before it can merge: - **The wire contract.** `protocol/api/openrpc.json` describes every method, topic and type. The build compares it with the one in the last release and fails on anything removed or changed, on a new required field, and on anything added without a higher protocol version. - **The Kotlin API.** Each library's public API is dumped to `api/*.api`. The build compares it with the one in the last release and fails on any line that disappeared. - **The binder interface** between apps and Visor is frozen: new methods only ever take new transaction codes. The full rules for contributors are in [CLAUDE.md](https://github.com/macedonga/visor-kit/blob/main/CLAUDE.md). # Porting from Meta's DAT This page maps each part of Meta's Wearables Device Access Toolkit for Android (DAT 1.0.1) to the Visor Development Kit. It is for an app that already works with DAT and should also work, or only work, with Visor. The two kits differ in shape more than in reach: - **No registration.** DAT registers the app through the Meta AI app, and Meta approves some features per app. With the VDK, the person grants scopes on Visor's consent screen, and that is all. - **One connection, no capabilities to attach.** DAT creates a `DeviceSession`, starts it, and attaches a camera, a display or inputs to it. The VDK has one `Visor` connection; each capability is a call or a flow. Collecting a flow starts it, and cancelling it stops it. - **Exceptions, not `DatResult`.** Every VDK call fails with a `KitException` whose `code` is an `ErrorCode`. - **The active pair.** DAT selects a device per session. The VDK acts on the pair Visor connects to; an app can change that pair for everyone with `glasses.select`. ## Setup | DAT | VDK | |---|---| | `com.meta.wearable:mwdat-core`, `mwdat-camera`, `mwdat-inputs`, `mwdat-speech`, ... | `com.macedonga.visor:kit`, one artifact | | `com.meta.wearable:mwdat-mockdevice` | `com.macedonga.visor:kit-testing` | | Manifest `com.meta.wearable.mwdat.APPLICATION_ID`, `CLIENT_TOKEN` | Nothing | | `ANALYTICS_OPT_OUT`, `CRASH_REPORTING_OPT_OUT` | Nothing: the VDK sends nothing anywhere | | `BLUETOOTH_CONNECT` permission, and the URL scheme for registration | Nothing: Visor holds the Bluetooth link | | `Wearables.initialize(context)` | `Visor.connect(context)`, which suspends until Visor answers | | `Wearables.startRegistration(activity)`, `registrationState` | `RequestScopes()`, an activity result contract | | `Wearables.RequestPermissionContract()`, `Permission.CAMERA` | `RequestScopes()` with `Scope.CAMERA` | | `Permission.MICROPHONE` | `Scope.MICROPHONE` for transcripts; the camera's live audio needs `Scope.CAMERA` only | | `Wearables.checkPermissionStatus(permission)` | `visor.scopes()` | | "Allow once" and "Allow always" | "Only this time" and "Allow" on Visor's consent screen | | `Wearables.openFirmwareUpdate(activity)` | No equivalent | ```kotlin // DAT Wearables.initialize(context) val launcher = registerForActivityResult(Wearables.RequestPermissionContract()) { result -> } launcher.launch(Permission.CAMERA) // VDK val visor = Visor.connect(context) val ask = registerForActivityResult(RequestScopes()) { held -> } ask.launch(setOf(Scope.CAMERA)) ``` ## Devices and sessions | DAT | VDK | |---|---| | `Wearables.devices`, `Wearables.devicesMetadata[id]` | `visor.glasses.pairs`, scope `pairs` | | `DeviceSession.deviceInfo` | `visor.glasses.status`, scope `state` | | `AutoDeviceSelector()` | The active pair, which every call acts on | | `SpecificDeviceSelector(id)` | `visor.glasses.select(serial)`: it changes Visor's active pair for every app | | `Wearables.createSession(selector)`, `session.start()` | `Visor.connect(context)` | | `session.stop()` | `visor.close()` | | `DeviceSessionState` `STARTING`, `STARTED`, `STOPPED` | `VisorState.Connecting`, `Open`, `Closed` | | `session.errors` | Each call's or flow's `KitException` | | `session.addCamera(...)`, `addInputs`, `addSpeech`, ... | Nothing: call or collect what you need | `Device` fields: | DAT `Device` | VDK | |---|---| | `name` | `GlassesPair.name`, `GlassesStatus.name` | | `deviceType.isDisplayCapable`, `isDisplayCapable()` | `display` | | `linkState`: `CONNECTING`, `CONNECTED`, `DISCONNECTED` | `link`: `connecting`, `connected`, `disconnected` | | `batteryLevel` | `battery`, null while unknown instead of 0 | | `chargingState` | `charging`, a `Boolean?` | | `donState`: `DONNED`, `DOFFED`, `UNKNOWN` | `worn`: true, false, null | | `hingeState` | `folded` | | `thermalLevel`: `NONE`, `LIGHT`, ..., `SHUTDOWN` | `thermal`: `normal`, `light`, ..., `shutdown`, an open set; and `throttling` | | `compatibility` | Nothing: Visor deals with the glasses' firmware | | | `model`, `serial`, `firmware`, `recording`, `caseBattery` | ## Camera | DAT | VDK | |---|---| | `session.addCamera(StreamConfiguration(...))`, `camera.stream.start()` | `visor.camera.live(LiveConfig { ... })`, collected | | `VideoQuality.LOW`, `MEDIUM`, `HIGH` | `LiveQuality.LOW`, `MEDIUM`, `HIGH`, and `FULL_HD` (1080 x 1920) | | `frameRate`: 2, 7, 15, 24, 30 | `frameRate`: the same | | `compressVideo = true`: HEVC | The default: each `LiveSample` is HEVC or H.264, Annex-B | | `compressVideo = false`: decoded frames | `LiveVideoDecoder { image, ptsUs -> }`: each picture as a YUV_420_888 `Image` | | | `LiveVideoDecoder(surface)`: straight onto a `Surface` | | `VideoFrame.buffer`, `width`, `height`, `presentationTimeUs` | `Image.planes`, `width`, `height`, and `ptsUs` | | `VideoFrame.isCodecConfig` | `LiveHeader.config` | | `stream.state`: `STREAMING` | The flow sends samples | | `stream.errorStream`, `StreamError` | The flow fails with a `KitException`; `reason` says why | | `camera.stop()` | Cancel the collection | ```kotlin // DAT val camera = session.addCamera(StreamConfiguration(videoQuality = VideoQuality.HIGH, frameRate = 30)).getOrThrow() camera.stream.start() camera.stream.videoStream.collect { frame -> analyse(frame.buffer, frame.width, frame.height) } // VDK LiveVideoDecoder { image, _ -> analyse(image) }.use { video -> visor.camera.live(LiveConfig { quality = LiveQuality.HIGH; frameRate = 30 }) .flowOn(Dispatchers.IO) .collect { video.decode(it) } } ``` Differences to know: - DAT streams over Bluetooth; the VDK streams over Wi-Fi Direct, so the phone's Wi-Fi must be on. It reaches 1080p. - A tap on the touchpad pauses DAT's stream. The VDK's stream does not pause; the person stops the Live in Visor. - When the Live already runs, for Visor's own view or another app, your app joins it as it is, whatever it asked. ### Photos | DAT | VDK | |---|---| | `stream.capturePhoto()` while streaming | `visor.camera.takeLivePhoto()` while `live` is collected | | `PhotoData.HEIC(data)` | `LivePhoto.uri`, a HEIC your app may read for 10 minutes | | `PhotoData.Bitmap(bitmap)` | Decode the HEIC with `ImageDecoder` | | `camera.photo.capturePhoto(PhotoResolution.FULL, PhotoQuality.HIGH)` | `visor.camera.takePhoto()`: the glasses' own photo at full size, in the gallery | | `photo.transferProgressStream` | `visor.captures.progress` | | | `visor.camera.startVideo()`, `stopVideo()`, and every capture on the glasses with `visor.captures` | ### Audio | DAT | VDK | |---|---| | `StreamConfiguration(audioCodec = AudioCodec.PCM(AudioSampleRate.RATE_16000, 1))` | `LiveConfig { audio = true }` | | `stream.audioStream`, `AudioFrame.buffer` | `LiveAudioDecoder { samples, sampleRate, channels, ptsUs -> }` | The glasses encode their microphones as 48 kHz stereo AAC, and `LiveAudioDecoder` hands that over as 16-bit PCM. Resample it if your code expects 16 kHz mono. ## Speech | DAT | VDK | |---|---| | `session.addSpeech()`, `speech.start()` | `visor.voice.transcripts(TranscriptRequest { language = "en-US" })`, collected | | `speech.transcriptions`: `TranscriptionResult(text, isFinal, confidence)` | `Transcript(text, language, time, partial, confidence)` | | Partial results, replaced until `isFinal` | `TranscriptRequest { partials = true }`; `partial` is `!isFinal` | | `speech.locale` | `TranscriptRequest.language`, or the phone's language | | `speech.stop()` | Cancel the collection | | `SpeechError.UNAVAILABLE` | `unsupported` with `no_recognizer` | On the Meta Ray-Ban Display the VDK recognises on the glasses as DAT does, with the toolkit app's speech, in the Display's language. On Ray-Ban Meta glasses, which DAT's speech does not cover, the text comes from the phone's speech recogniser, fed by the glasses' microphones as a Bluetooth headset, so music pauses while transcripts run. A confidence comes when the recogniser gives one. ## Voice invocations | DAT | VDK | |---|---| | `Wearables.startVoiceInvocationsStream(selector)` | `visor.voice.requests(phrases)`, collected | | `LaunchApp` | `VoiceRequest(id, text, phrase, intent)` | | `invocation.responseHandle.sendSuccess(...)` | `visor.voice.reply(request, text)`: the glasses say `text` | | `isVoiceInvocationsIntent(intent)` | `VoiceLaunch.request(intent)`, for an activity that declares its phrases | | The app name as the keyword, set in the developer centre | Phrases in the activity's `com.macedonga.visor.kit.VOICE_PHRASES` meta-data | DAT opens the app when the wearer names it. The VDK does too, for an activity that declares its phrases, and sends the question itself to the app, which answers in the glasses' voice. Beyond DAT, `visor.assistant.tools(...)` gives the assistant's model tools to call while it answers. ## Inputs | DAT | VDK | |---|---| | `session.addInputs(InputsConfiguration(sources, consumeBack))` | `visor.glasses.inputs(InputsRequest { sources = ...; keepBack = true })`, collected, scope `inputs` | | `InputEvent.Select` | `select` on the Display; on other glasses `touchpad` `double_tap`, as DAT reads it | | `InputEvent.Nav(direction)` | `nav_up`, `nav_down`, `nav_left`, `nav_right` on the Display; on other glasses `touchpad` `swipe_forward` or `swipe_backward` | | `InputEvent.Drag(action, x, y, dx, dy)` | `drag_start`, `drag_move`, `drag_end` with `x`, `y`, `dx`, `dy` on the Display; on other glasses `slide_forward` or `slide_backward`, between `touch` and `release` | | `InputEvent.Capture(CapturePressType.SHORT_PRESS)`, `HOLD`, `DOUBLE_PRESS` | `capture_button` `press`, `long_press`, `double_press` | | `InputEvent.Button(ButtonType.ACTION)` | `action_button` `press`, `release`, `long_release` | | `InputEvent.Back` | `back` on the Display; no gesture means back on other glasses | | `InputSource.CAPTOUCH`, `CAPTURE_BUTTON`, `ACTION_BUTTON` | `InputSource.TOUCHPAD`, `CAPTURE_BUTTON`, `ACTION_BUTTON` | | `InputSource.NEURAL_BAND`, `NEURAL_BAND_DRAG` | `InputSource.BAND`; every band gesture is also on `visor.band.gestures`, scope `band` | | `consumeBack = true` | `keepBack = true`, on the Display only: other glasses still act on every input. The band's back still closes the layout | ## Motion | DAT | VDK | |---|---| | `session.addMotion(MotionConfiguration(samplingRate))` and `motion.samples` | `visor.glasses.motion(rateHz)`, scope `motion` | | `MotionSamplingRate` 5, 10, 15, 24, 30, 60 Hz | `rateHz`, the same six values | | `MotionSample` (timestamp, accelerometer, gyroscope, magnetometer, orientation, source) | `MotionSample` with the same fields and units | | `MotionSource.GLASSES`, `NEURAL_BAND` | `MotionSource.GLASSES`, `BAND` | | `MotionError.SENSOR_UNAVAILABLE` | the stream ends with `unsupported` | > On the glasses, Visor reads motion from their own sensor service, with nothing on the screen, where DAT opens > its toolkit app. The phone fuses the sensors into `orientation`. The band's motion is a stream of its own, > `visor.band.motion`, for a band the person shares with the phone. FakeVisor serves all the fields: > `fake.glasses.motion(...)`, DAT's motion CSV with `fake.glasses.playMotion(...)`, and `fake.band.motion(...)`. ## Display | DAT | VDK | |---|---| | `display.sendContent { flexBox { ... } }` | `visor.display.layout.show(Layout { ... })` | | `flexBox(direction = Direction.COLUMN)`, `ROW`, `COLUMN_REVERSE`, `ROW_REVERSE` | `column { }`, `row { }`, with `reverse = true` | | `flexBox(gap, alignment, crossAlignment, wrap, padding, paddingTop, ...)` | `gap`, `align`, `crossAlign`, `wrap`, `padding`, `paddingTop`, ... | | `flexBox(background = FlexBoxBackground.CARD, onClick = ...)` | `card = true`, `id = "..."` | | `flexGrow`, `flexShrink`, `alignSelf` on any element | `grow`, `shrink`, `alignSelf` | | `text(..., style = TextStyle.HEADING, color = TextColor.SECONDARY)` | `text("...") { style = LayoutStyle.HEADING; color = LayoutColor.SECONDARY }` | | `image(uri = ..., size = ImageSize.ICON, cornerRadius = CornerRadius.SMALL)` | `image(url) { size = LayoutImageSize.ICON; corners = LayoutCorners.SMALL }`: an `https://` URL or a `data:` URI | | `image(bitmap = ...)` | `image("data:image/png;base64,...")` | | `icon(IconName.HEART, IconStyle.OUTLINE)` | `icon(LayoutIconName.HEART) { style = LayoutStyle.OUTLINE }`, the same names | | `button(label, style, iconName, onClick, actionRole = ActionRole.PRIMARY)` | `button(id, label) { style = ...; icon = ...; primary = true }`, and its clicks on `visor.display.layout.clicks` | | `buttonGroup(alignment) { button(...) }` | `buttons { align = ...; button(...) }` | | `video(VideoPlayer(VideoSource.Url(url), VideoCodec.MP4))`, `player.state` | `video(url) { controls = true }`, alone in its layout, and `visor.display.layout.video` | | `display.state`, back ending the display | `visor.display.layout.state`, with `reason` `back` | | `display.clearDisplay()` | `visor.display.layout.close()` | On the glasses, Visor shows layouts through the Display's toolkit session, as DAT does, and draws them with the same rules as DAT's own builder, videos included. FakeVisor, `visor fake` and its Display simulator serve everything. The VDK also shows text on the teleprompter (`visor.display.prompter`) and notifications (`visor.notify`), and keeps an app's cards on the Display's home, beside Meta's weather and stocks, with details, pages, charts and clicks: see [Briefing cards](briefing-cards.md). DAT has nothing like them. The VDK's layouts go beyond DAT's elements with what Meta's own cards use: the `title` text style, any colour, `maxLines` and text `align`, icon `color` and `size`, box `background` and `height`, pins, a `pager` and a `chart`. ## Testing | DAT `MockDeviceKit` | `FakeVisor` | |---|---| | `MockDeviceKit.getInstance(context).enable()` | `val fake = FakeVisor()`, then `fake.connect()`, in a plain JVM test | | `pairGlasses(GlassesModel.META_RAYBAN_DISPLAY)` | `fake.glasses.display = true`; more pairs with `fake.pairs.add(...)` | | `powerOn()`, `powerOff()` | `fake.glasses.connected` | | `don()`, `doff()` | `fake.glasses.worn` | | `fold()`, `unfold()` | `fake.glasses.folded` | | `setBatteryLevel`, `setChargingState`, `setThermalLevel` | `fake.glasses.battery`, `charging`, `thermal` | | `permissions.set(Permission.CAMERA, PermissionStatus.Denied)` | `fake.granted = fake.granted - Scope.CAMERA` | | `services.camera.setCameraFeed(uri)` | `fake.live.play(stream, codec, width, height, fps, loop)`, or `sendVideoConfig(...)` and `sendVideo(...)` | | `services.motion.setMotionFeed(csv)` | `fake.glasses.playMotion(reader, loop)`, DAT's CSV as it is | | `services.captouch.tap()`, `services.input.select(...)` | `fake.glasses.input(InputSource.TOUCHPAD, InputAction.TAP)`, `InputAction.SELECT` | | `services.display` and the Display simulator in Chrome | `fake.display.layout`, `click(id)`, `back()`, and the simulator `visor fake` serves at `/display` | | `createPreviewView()` | `fake.display.preview(context)` | | `services.speech.simulateTranscription(text, ...)` | `fake.voice.say(text)`, or `say(text, partial, confidence)` | | `services.voiceInvocation.simulateLaunchAppAction()` | `fake.voice.ask(question)` | | `startTestServer(port)` | `visor simulate` and `visor inject` from a computer, against the real Visor | See [Testing](testing.md) for the rest. ## Errors | DAT | VDK `ErrorCode` | |---|---| | `PermissionStatus.Denied`, `PermissionError` | `scope_denied`, with the missing `scope` | | `DeviceSessionError.NO_ELIGIBLE_DEVICE`, `DEVICE_DISCONNECTED` | `not_connected` | | `CaptureError.NotStreaming` | `busy` | | `StreamError.TIMEOUT` | `timeout` | | `StreamError.CRITICAL_STREAM_ERROR` | `failed`, with a `reason` | | `DeviceSessionError.INSUFFICIENT_SDK_VERSION` | `visor_too_old` | The [Errors](errors.md) page lists every code. # AI-assisted development A coding agent can set up the VDK, write the code and test it on fake glasses. Give it the prompts on this page, and the VDK gives it the docs, the rules and the glasses themselves. ## Set up, by your agent Paste this into Claude Code, Codex, Cursor or any agent that runs commands, in your project: ```text Set up this project for the Visor Development Kit (VDK), which lets software use Meta glasses through Visor. 1. If `visor --version` fails, install the visor command. It needs Java 17 or later. On macOS or Linux: `curl -fsSL https://visor.macedonga.com/install.sh | sh`. On Windows: `irm https://visor.macedonga.com/install.ps1 | iex`. 2. Add the VDK's docs as an MCP server, `visor docs-mcp`. In Claude Code: `claude mcp add vdk-docs -- visor docs-mcp`. 3. Save the VDK's rules for agents with `visor agents > AGENTS.md`, and follow them. 4. Read https://visor.macedonga.com/llms.txt, then add the VDK to this project as the getting started guide says. Check every method name in the docs before you use it. ``` ## Prompts for common tasks Each works once the agent is set up. Change the details to your app. ```text Show a Briefing card on the Meta Ray-Ban Display with the next three events from the phone's calendar, updated when they change. Test it with `visor fake` and its Display simulator before the glasses. ``` ```text When the wearer double-taps with the index finger on the Neural Band, take a photo with the glasses and save it to the phone's gallery. Ask for the scopes it needs, and handle the person saying no. ``` ```text Answer the questions that name this app, such as "ask Recipes what to cook tonight", in the glasses' own voice. ``` ```text Port this app from Meta's Wearables Device Access Toolkit (DAT) to the VDK, with the porting guide: https://visor.macedonga.com/developers/docs/porting-from-dat.md ``` ```text Write unit tests for the glasses features with FakeVisor from kit-testing, then run them. ``` ## What agents can read | What | Where | |---|---| | An index of every page | [llms.txt](https://visor.macedonga.com/llms.txt) | | All the VDK docs in one file | [llms-full.txt](https://visor.macedonga.com/llms-full.txt) | | Any page as Markdown | The page's address with `.md` in place of the last `/`, such as [neural-band.md](https://visor.macedonga.com/developers/docs/neural-band.md) | | Every method, topic and type, for any language | [`protocol/api/openrpc.json`](https://github.com/macedonga/visor-kit/blob/main/protocol/api/openrpc.json), an OpenRPC document | | The glasses' own protocol, for work below the VDK | [The protocol reference](https://visor.macedonga.com/protocol/) | Each page of the docs also has **Copy as Markdown**, to paste it into a chat. ## The docs as an MCP server The `visor` command carries the docs, so an agent can search them with no network: ``` visor docs layout button click # the sections that answer best visor docs # the pages ``` `visor docs-mcp` serves them as an MCP server over standard input and output, with two tools: `search_vdk_docs` (a query, and `max_results` up to 10) and `read_vdk_doc` (a page, such as `display`). Add it to Claude Code with: ``` claude mcp add vdk-docs -- visor docs-mcp ``` Any assistant that runs MCP servers as commands takes the same command. ## Rules for agents `visor agents` prints the rules for code that uses the VDK, the same as [for-agents.md](for-agents.md). Save them as `AGENTS.md` in your app, which Codex, Cursor, GitHub Copilot and Claude Code read: ``` visor agents > AGENTS.md ``` ## The Claude Code plugin The VDK's repo is a Claude Code plugin marketplace. The `vdk` plugin adds skills for getting started, the VDK's conventions, the Display, voice, testing and porting from DAT, and sets up the docs MCP server: ``` /plugin marketplace add macedonga/visor-kit /plugin install vdk@visor-kit ``` ## The glasses, for agents An agent can use the glasses as well as write code for them. Visor's [network door](network.md) serves MCP with a token the person makes for the agent, and each method the token allows becomes a tool: ``` claude mcp add --transport http visor http://127.0.0.1:7333/mcp --header "Authorization: Bearer " ``` To let an agent try its code with no glasses, run `visor fake` and give it the fake's MCP, with no token. It calls the same methods, and the Display simulator shows what it puts on the Display: ``` claude mcp add --transport http visor-fake http://127.0.0.1:7333/mcp ``` `visor --fake simulate band.gestures` then plays gestures to the app, as a band would. # VDK guidance for coding assistants Save this as `AGENTS.md` in an app that uses the Visor Development Kit (`visor agents > AGENTS.md`), or point a rules file at it. It tells a coding assistant how to build on the VDK. For any detail, search the docs: run `visor docs QUERY`, or add the docs MCP server with `claude mcp add vdk-docs -- visor docs-mcp` and call `search_vdk_docs` and `read_vdk_doc`. ## What the VDK is The VDK lets an Android app, or a program on a computer, use Ray-Ban Meta glasses, the Meta Ray-Ban Display and the Meta Neural Band through Visor, the Android app that runs them. It is an alternative to Meta's Wearables Device Access Toolkit (DAT): no Meta AI app, no Meta account, no developer centre. Porting from DAT: read `porting-from-dat`. - Android: `implementation("com.macedonga.visor:kit:")`, then `val visor = Visor.connect(context)`. - Tests: `com.macedonga.visor:kit-testing` gives `FakeVisor` and `fake.connect()` in a plain JVM test. - Other languages: JSON-RPC over adb or Visor's network door; Python `visor_kit` and TypeScript clients. - A web app on the Display: the TypeScript client's `Visor.connectGlasses()`, guarded by `onGlasses()`. On a `refused` with reason `not_allowed`, tell the person to allow the app in Visor > Settings > Web apps. ## Rules 1. Ask for scopes before using them: `registerForActivityResult(RequestScopes())`, then `launch(setOf(...))`. A call without its scope fails with `KitException` code `scope_denied`, naming the missing `scope`. 2. Every call is a `suspend fun`; every stream is a cold `Flow` that runs while collected. Collect streams in a lifecycle-aware scope and cancel them to stop: there are no start and stop pairs. 3. Handle `KitException` by its `code` (`unsupported`, `not_connected`, `busy`, `invalid_params`, `failed`, `scope_denied`) and `reason`. Treat an unknown code as `failed` and unknown enum-like values as unknown: the value sets (scopes, input actions, gesture actions, layout types) grow. 4. Layouts for the Display are whole screens: build one with `Layout { }` and show it with `visor.display.layout.show(...)`; a new show replaces it. Clicks come on `display.layout.clicks` by the element's `id`; `display.layout.state` says when the wearer went back. For something the wearer should see all day (a timer, a delivery, a score), prefer a Briefing card: `visor.display.cards.set(id, name, icon) { header(...); stat(...); detail { } }`, kept with `cards.keep(id, name, icon, flow) { }`, with clicks in its detail on `display.cards.clicks`. A card takes at most 256 KiB. 5. A voice phrase reaches the app while it collects `visor.voice.requests(phrases)`, or starts it when its activity declares the phrases (`VoiceLaunch`). Reply within about 20 seconds with `visor.voice.reply`. 6. `band.motion` needs a band the person shares with the phone: check `band.status` `shared` first, and on `unsupported` with reason `not_shared`, show the person the exception's message, which names the setting. 7. Do not invent methods. The methods and topics are in `kit-protocol` (`Methods`, `Topics`) and in the docs' JSON-RPC tables; `visor methods` lists what a Visor answers. ## Testing without glasses - `FakeVisor` in unit tests: set `fake.glasses.*` and `fake.band.shared`, send `fake.glasses.input(...)`, `fake.band.gesture(...)`, `fake.band.motion(...)`, `fake.voice.say(...)`, `fake.display.click(id)`, `fake.display.back()`, `fake.display.clickCard(card, id)`. - `visor fake` on a computer: JSON-RPC on `ws://127.0.0.1:7333/rpc`, MCP on `/mcp`, the Display simulator on `http://127.0.0.1:7333/display`, and a console that plays the glasses. - `visor record FILE` and `visor replay FILE` capture and play back what the wearer did. - `FakeGlasses.playMotion` plays DAT's motion CSV; `FakeLive.play` plays a recorded H.264 or H.265 stream. # Developer mode > [!NOTE] > New apps use the [VDK](getting-started.md): it does all of this and more, and the person grants each app its own > scopes. Developer mode's intents are for automation apps, such as Tasker or MacroDroid, that can only send > intents, and they keep working. The switch also opens the socket the [`visor` command](testing.md#install) uses. Developer mode lets automations and the launcher work the glasses' camera through Visor. They can take a photo, start and stop a video, import a capture, hear about new captures, and get the Live view's picture. ## Turn it on Open Settings, Advanced, and turn on **Developer mode** under Tools. The glasses take a photo for the phone only when a Meta switch, `wearables_robusta.demo_enabled`, is on. Developer mode turns it on, and turns it off again when you turn the mode off. If Visor has never read the glasses' configuration, it reads it first, which can take a minute over Bluetooth. While the mode is on: - Visor answers the actions below. - The launcher offers two shortcuts on Visor's icon: Take photo and Record video. - Visor broadcasts the events below. When the mode is off, Android does not deliver the actions to Visor at all. ## Token With the mode on, Advanced shows a **Token** under the switch. Tap it to copy it. The arrow next to it makes a new one, and the old one stops working at once. An app needs the token for the Live frames. For the actions, it needs the token only when **Actions need the token** is on. Then Visor ignores an action without the right `token` extra and says so in the event log. The launcher's shortcuts carry the token on their own. ## Actions Send each action as a broadcast to the package `com.macedonga.visor`, or start it as an activity. The activity shows nothing and closes at once. | Action | Extras | What it does | |---|---|---| | `com.macedonga.visor.action.TAKE_PHOTO` | none | Takes a photo. | | `com.macedonga.visor.action.START_VIDEO` | `capture_type` (optional): `video` or `hyperlapse` | Starts a video. | | `com.macedonga.visor.action.STOP_VIDEO` | none | Stops the video. | | `com.macedonga.visor.action.TOGGLE_VIDEO` | `capture_type` (optional) | Stops the video if one runs, else starts one. | | `com.macedonga.visor.action.IMPORT_MEDIA` | `id`: the id from `MEDIA_AVAILABLE` | Imports that capture to the phone. | With **Actions need the token** on, add `token` to every action. If the glasses are not connected, Visor tries to connect and waits up to 20 seconds before it acts. From a computer: ``` adb shell am broadcast -p com.macedonga.visor -a com.macedonga.visor.action.TAKE_PHOTO adb shell am broadcast -p com.macedonga.visor -a com.macedonga.visor.action.IMPORT_MEDIA --es id ``` From an app: ```kotlin context.sendBroadcast(Intent("com.macedonga.visor.action.TAKE_PHOTO").setPackage("com.macedonga.visor").putExtra("token", token)) ``` Set the package. Android does not deliver a broadcast without one to Visor. In Tasker, use Send Intent with the package `com.macedonga.visor` and the target Broadcast Receiver. ## Events Visor sends these as broadcasts while developer mode is on. | Event | When | Extras | |---|---|---| | `com.macedonga.visor.event.MEDIA_AVAILABLE` | The glasses have a capture that this phone does not have yet. | `id`, `type`, `time`, `size` | | `com.macedonga.visor.event.IMPORT_PROGRESS` | The photo or video of a capture is on its way, once per whole percent. | `id`, `received`, `size`, `percent` | | `com.macedonga.visor.event.NEW_MEDIA` | A capture is in the gallery. | `id`, `type`, `time`, `uri`, `path` | | `com.macedonga.visor.event.IMPORT_FAILED` | An import did not finish. | `id`, `reason` | | `com.macedonga.visor.event.VIDEO_STARTED` | The glasses started a video. | `time` | | `com.macedonga.visor.event.VIDEO_STOPPED` | The glasses stopped a video. | `time` | The extras: - `id`: the capture's id on the glasses. `IMPORT_MEDIA` takes it. - `type`: `photo`, `live` (a photo with its clip) or `video`. - `time`: milliseconds since 1970. For a capture, when the glasses took it. For a video event, when Visor saw it. - `size` and `received`: bytes, as longs. - `percent`: 0 to 100, as an int. - `uri`: the capture's `content://media/...` address in the gallery. - `path`: the file, for example `/storage/emulated/0/Pictures/Visor/Visor_20261021_103433.jpg`. Photos go to `Pictures/Visor` and videos to `Movies/Visor`. - `reason`: why the import failed. `NOT_FOUND` means the glasses have no capture with that id. `WIFI_OFF`, `NOT_ALLOWED`, `CUT_OFF` and the others come from Visor's import. To receive the events, register a receiver while your app runs: ```kotlin val filter = IntentFilter().apply { addAction("com.macedonga.visor.event.MEDIA_AVAILABLE") addAction("com.macedonga.visor.event.NEW_MEDIA") } ContextCompat.registerReceiver(context, receiver, filter, ContextCompat.RECEIVER_EXPORTED) ``` A receiver declared in your manifest does not get them, because Android delivers broadcasts without a package only to receivers registered at run time. ## Live frames While a Live runs on Visor, an app that holds the token can get its picture as JPEG stills. Visor decodes each still from the stream on its own, so it asks for one every 200 ms at most, and how many it manages depends on the phone. 1. Declare Visor in your manifest, or Android hides it from your app: ``. 2. Bind the service with the action `com.macedonga.visor.action.LIVE_FRAMES` and the package `com.macedonga.visor`. 3. Send message 1 (subscribe) with `token` and, if you want, `interval_ms` (default 1000, at least 200). Set `replyTo` to your own `Messenger`. 4. Read message 3 (frame): `jpeg` is the picture as a byte array, `time` is when Visor took it, in milliseconds since 1970. 5. Send message 2 (unsubscribe), or unbind, to stop. Message 4 (refused) comes instead when the token is wrong or the mode is off. Its `reason` is `TOKEN` or `OFF`. It also comes later if someone makes a new token or turns the mode off. ```kotlin val frames = Messenger(Handler(Looper.getMainLooper()) { msg -> when (msg.what) { 3 -> msg.data.getByteArray("jpeg")?.let { show(BitmapFactory.decodeByteArray(it, 0, it.size)) } 4 -> Log.w("frames", "refused: ${msg.data.getString("reason")}") } true }) val connection = object : ServiceConnection { override fun onServiceConnected(name: ComponentName, binder: IBinder) { val subscribe = Message.obtain(null, 1).apply { data = bundleOf("token" to token, "interval_ms" to 500L) replyTo = frames } Messenger(binder).send(subscribe) } override fun onServiceDisconnected(name: ComponentName) {} } context.bindService( Intent("com.macedonga.visor.action.LIVE_FRAMES").setPackage("com.macedonga.visor"), connection, Context.BIND_AUTO_CREATE, ) ``` Nothing comes while no Live runs. Frames start again by themselves when the next Live starts. ## Things to know - A typical flow: take a photo, wait for its `MEDIA_AVAILABLE`, send `IMPORT_MEDIA` with that id, and read the file from `NEW_MEDIA`. In a test, `MEDIA_AVAILABLE` came less than a second after the photo. - An import goes over Wi-Fi Direct, so the phone's Wi-Fi must be on. - The glasses also send captures still waiting in their own queue from an earlier import. Each one gets its own `NEW_MEDIA`, so check the `id`. - `IMPORT_MEDIA` for a capture already on the phone imports nothing and sends `NEW_MEDIA` again. - After Visor restarts, it sends `MEDIA_AVAILABLE` again for every capture still waiting on the glasses. - Visor's own imports, from the gallery or on their own, send `IMPORT_PROGRESS`, `NEW_MEDIA` and `IMPORT_FAILED` too. - To read the file at `uri` or `path`, your app needs Android's permission to read photos and videos. - While the mode is on and **Actions need the token** is off, any app on the phone can send these actions. The events go to every app that listens. The glasses' capture light still turns on for every photo and video. - The glasses' call app also refers to `wearables_robusta.demo_enabled`. What the switch does there is not known. - Visor's event log (Settings, Advanced, Event log) shows each event it sends, with its extras.