Ask Tappy to open an app and complete a multi-step task, stopping before a sensitive final action.
Get step by step on-screen teaching with highlighted boxes and arrows instead of the app acting for you.
Summon a voice assistant from anywhere on the phone using a floating handle or Quick Settings tile.
Interrupt the assistant mid-response the way you would in a real conversation.
| haz3-jolt/tappy | dw2lam/openlauncher | thatjoshguy67/blur-widget-demo | |
|---|---|---|---|
| Stars | 18 | 18 | 18 |
| Language | Kotlin | Kotlin | Kotlin |
| Setup difficulty | hard | moderate | moderate |
| Complexity | 4/5 | 2/5 | 2/5 |
| Audience | general | general | developer |
Figures from each repo's GitHub metadata at analysis time.
Requires enabling Android's Accessibility service, granting microphone and screen share permissions, and running a separate Cloud Run token broker.
Tappy Live is an Android app, built for a hackathon, that acts as a real time voice assistant able to both teach you how to use your phone and actually operate apps on your behalf. You can summon it from anywhere on the phone, speak a request in plain language, and it either walks you through the steps with on screen highlights and arrows, or takes over and performs the steps itself, stopping to ask for confirmation before any sensitive final action like completing a purchase. It combines two pieces. The voice side uses Google's Gemini Live API for a natural, interruptible conversation, meaning you can talk over the assistant while it is mid-sentence and it will adapt rather than finish its scripted response first. The control side uses Android's Accessibility service, which lets the app see a structured, numbered list of the buttons, text fields, and other elements on whatever screen is currently open, and then tap, type, scroll, or swipe on the user's behalf using that same list. A small floating handle can summon Tappy over any app, and it can also register itself as the phone's default voice assistant or as a Quick Settings tile. The README lists a working demo loop confirmed on a Samsung S23 Ultra: the app can connect to Gemini Live, stream microphone audio, play back spoken responses, capture and index the current screen, and complete a real multi-step task such as opening Uber, tapping the destination field, typing in a location, and stopping right before the ride is actually booked. When teaching rather than acting, Tappy can draw numbered boxes, circles, and arrows over parts of the screen and wait for the user to complete each step themselves. Setting it up requires granting microphone and notification permissions, turning on Tappy's Accessibility service, and approving screen sharing each time a new session starts, since Android does not let apps silently obtain those permissions on their own. A separate small cloud service issues short lived Gemini Live access tokens so the mobile app never holds a long lived credential directly. The project is licensed under AGPL-3.0.
An Android voice assistant that watches your screen through Accessibility services and either teaches you how to use an app step by step or operates it for you.
Mainly Kotlin. The stack also includes Kotlin, Android, Gemini Live API.
You can use and modify this freely, but if you distribute it or run a modified version as a network service, you must share your source code under the same license.
Setup difficulty is rated hard, with roughly 1h+ to a first successful run.
Mainly general.
This repo across BitVibe Labs
Verify against the repo before relying on details.