Voice and the assistant

Visor answers "Hey Meta" with the person's own model. An app can take part in two ways: answer the questions that name it, and give the model tools to use while it answers.

Questions that name the app

Scope: voice. All glasses.

visor.voice.requests(listOf("the garage", "my lights")).collect { request ->
    val text = when {
        "close" in request.text -> garage.close().let { "Closing the garage." }
        else -> "I can open or close the garage."
    }
    visor.voice.reply(request, text)
}

While the app collects requests, every question that holds one of its phrases comes to the app instead of the model: "Hey Meta, ask the garage to close" comes for "the garage". The glasses then say the reply in their own voice, as they would the model's answer.

• Matching. A question matches a phrase when it holds all the phrase's words, in order and next to each other. Case, accents and punctuation do not count. When phrases of several apps match, the longest phrase wins. Pick phrases nobody says by chance: "the garage" rather than "open".
• Time. Reply within about 20 seconds; the glasses give up on a question after that. Without a reply, the glasses tell the wearer that the app did not answer.
• Follow-ups. reply(request, text, listen = true) keeps the microphone open after the reply, as after a question. The follow-up is matched like any question: it comes to the app again only if it names a phrase.
• What else comes. request.intent is what the glasses understood the question as, such as IN:QUESTION, when they said. request.phrase is the phrase that matched.

A VoiceRequest:

A reply that comes too late fails with invalid_params and reason expired.

Started by voice

An app that is not running can be started by its phrase, as DAT's voice invocations start one. Declare an activity with the VDK's voice action and a string array of phrases:

<activity android:name=".GarageActivity" android:exported="true">
    <intent-filter>
        <action android:name="com.macedonga.visor.kit.action.VOICE" />
        <category android:name="android.intent.category.DEFAULT" />
    </intent-filter>
    <meta-data android:name="com.macedonga.visor.kit.VOICE_PHRASES" android:resource="@array/garage_phrases" />
</activity>

When a question holds one of these phrases and no running app holds it, Visor starts the activity with the question. Read it with VoiceLaunch.request(intent) in onCreate and onNewIntent, and reply as above, within about 20 seconds:

VoiceLaunch.request(intent)?.let { request ->
    lifecycleScope.launch { Visor.connect(this@GarageActivity).voice.reply(request, "The garage is closed.") }
}

Visor starts only apps the person granted the voice scope, so ask for it once, at setup. While the app collects requests, its running offer answers instead and nothing is started. Visor reads the declared phrases again every 30 seconds, so a newly installed app is heard within half a minute.

Saying something unasked

speak has the glasses say a line in their own voice now, with nobody asking: a timer that ran out, a door that opened, a message worth hearing. It needs the voice scope.

visor.voice.speak("Your coffee is ready.")

Keep it short and rare: the wearer hears it over whatever they are doing. The text holds at most 500 characters.

What the glasses hear

Scope: microphone. All glasses, on Android 13 or later.

visor.voice.transcripts(TranscriptRequest { language = "en-US" }).collect { transcript ->
    println(transcript.text)
}

transcripts turns what the glasses' microphones hear into text, one finished utterance at a time, while you collect it. Visor records the glasses as a Bluetooth headset, never the phone's own microphone, and the phone's speech recogniser makes the text: on the phone when it holds the language, otherwise as the phone's recogniser does it. Without a language, Visor hears in the phone's language.

While transcripts run, the glasses act as a call headset, so music pauses as in a call. Android lets Visor record only while it may use the microphone: while your app is on screen, or Visor is. The VDK lends Visor your app's access when it binds, so keep collecting from a visible screen. From a computer, keep Visor on screen.

The Meta Ray-Ban Display is no Bluetooth headset to the phone. There the Display hears the wearer itself, with the speech recognition of Meta's toolkit app, as DAT's speech does: in the Display's own language, which language cannot change, and each transcript says. The toolkit app must be on the screen while it listens. If your app shows no layout, the Display shows a Listening screen with your app's name, so the wearer knows who listens; going back on the glasses ends the stream with closed.

Each Transcript has the text, the language it was heard in, the time the utterance ended, and the recogniser's confidence from 0 to 1 when it gives one. Several apps can listen at once, in one language: an app that asks for another language meanwhile fails with busy.

Ask for partials to get the text while the wearer speaks, as DAT gives it. A partial transcript has partial true, and the next one replaces it; the finished utterance comes last with partial false:

visor.voice.transcripts(TranscriptRequest { partials = true }).collect { t ->
    if (t.partial) caption.text = t.text else log.append(t.text)
}

The stream ends with:

Tools for the model

Scope: assistant. All glasses.

val tools = listOf(
    ToolSpec {
        name = "set_lights"
        description = "Turns the living room lights on or off."
        schema = buildJsonObject {
            put("type", "object")
            putJsonObject("properties") { putJsonObject("on") { put("type", "boolean") } }
            putJsonArray("required") { add("on") }
        }
    },
)
visor.assistant.tools(tools).collect { call ->
    val on = call.arguments.jsonObject["on"]?.jsonPrimitive?.boolean
    if (on == null) visor.assistant.fail(call, "Say whether to turn them on or off.")
    else visor.assistant.answer(call, if (lights.set(on)) "Done." else "The lights did not answer.")
}

While the app collects tools, the model can call them, alongside the person's own MCP servers, the Display's own tools and those of a web app open on it. Each call comes as a ToolCall; answer it with answer or fail within 15 seconds. The model reads the text and words its answer to the wearer from it, so say what happened, not how to say it.

• Names start with a letter, then letters, digits, _ or -, 40 at most. When two apps offer the same name, the second one's gets a _ added for the model; the app still receives its own name.
• Schemas are JSON Schema objects. Without one, the tool takes no arguments.
• Descriptions are what the model goes by to choose a tool. Say what it does and when it applies.
• Visor's diary records each call with the app's name and whether it worked.

The conversations

Visor keeps what the wearer asks the assistant, and its answers, on the phone until the person deletes them. With the conversations scope an app can read them, and get each new one as it is answered: a journal, a search over everything asked, a sync to the person's own notes.

visor.assistant.turns.collect { turn -> println("${turn.question} -> ${turn.answer}") }
val lastWeek = visor.assistant.history(since = System.currentTimeMillis() - 7 * 86_400_000L, limit = 500)

An AssistantTurn:

history returns the newest turns first, at most limit (1 to 500, 50 by default).

Notifications on the Display

Scope: notify. The Meta Ray-Ban Display.

visor.notify.post("delivery", "Package", "Your delivery is at the door.")
visor.notify.remove("delivery")

A second post with the same id replaces the first. Glasses without a display show no notifications of apps, so there post fails with unsupported.

JSON-RPC

From your computer, the visor command can play the app's part:

visor watch voice.requests '{"phrases":["the garage"]}'
visor call voice.reply '{"request":"<id from the event>","text":"Closing the garage."}'