Customizing user interfaces of binary applications

This invention describes a system that lets you control existing phone apps with your voice or gestures, without changing the app's original code. When an app starts, the system identifies its on-screen elements like buttons and menus. It then creates custom labels or overlays for these elements, allowing you to interact with them hands-free, for example, by speaking a command that the system translates into a touch action. The system specifically uses a customizable overlay template and a label index, which can be a simple file like XML or text.

Why it matters: Filed before advanced AI models could easily interpret natural language commands and dynamically map them to arbitrary UI elements. The voice-to-UI mapping, complex in 2022, is now more achievable with LLMs.

Status: Not in forceListed as no longer active. The specific reason is not in the record we hold.
How hard to build: SpecializedHeavy software system, OS-level UI interaction

3 AI reinvention directions for this patent

See all three

AI gives you a few directions you could take this. Pick one, and we check whether your version is different enough to patent, then write the filing.

Reinvent this with AI

Where this one came from

Every morning we go through millions of abandoned patents and pick out 15 worth your time. This was one of today’s.

This one

+10