OpenAI DevDay Exchange · Bengaluru 2026

Ghost Hands

Ask how. Watch it happen on your screen. Then do it yourself.

Say "how do I freeze the top row?" A ghost cursor does it on your real screen at human pace and narrates every step. Then it hands you the mouse and coaches you through it.

Watch the demo Run it yourself
Built onGPT-6.1 Sol computer use·gpt-transcribe·gpt-4o-mini-tts

Demo

Ninety seconds on one real screen.

Unedited screen recording. The ghost freezes the top row of a real Google Sheet, then gets asked to send it to a manager and stops at Send.

Video unavailable

Watch it on GitHub instead.

3 steps to freeze a row ~1 s from step to spoken caption 0 clicks on Send

How it works

It shows you. Then it makes you do it.

Tutorials happen on someone else's screen. Agents that just do the task leave you no better at it. Ghost Hands does both halves.

  1. 01

    Ask

    Hold F8 and say it, or press F7 and type it. "How do I make a bar chart of sales by month?"

  2. 02

    The ghost does it

    A cursor glides to each real target at human pace, clicks and types there, and narrates every step with a caption and a calm voice.

  3. 03

    Your turn

    Press F9. A ring marks each step and waits for your real click. Hits turn green; after two misses the ghost shows you again.

  4. 04

    It never does the scary part

    At Send, Pay, Delete or Submit it stops with a lock and says "That one is yours." Grab the mouse at any point to take over.

Built on OpenAI

It reads pixels, not APIs.

So it works in any app on your screen. The loop runs on GPT-6.1 Sol with the computer-use tool in the Responses API.

Actgpt-6.1-sol · computerResponses API with ordered action batches. Its commentary becomes the captions.
Listengpt-transcribePush-to-talk speech to text with keyword hints for a noisy room.
Speakgpt-4o-mini-tts · marinStreamed, and steered toward a calm, unhurried teacher. Every line is cached, so replays need no network.
  • Invisible to the modelThe overlay sets WDA_EXCLUDEFROMCAPTURE, so the AI never sees the ghost. That flag fails on layered windows, so the overlay composites through OpenGL instead.
  • Never steals focusA click-through, no-activate tool window. Your app keeps the keyboard throughout.
  • 60 fps on a laptop iGPUPyQt6 with prebaked glow, eased arcs and dirty-region repaints.
  • Clicks land exactlyPer-monitor DPI aware, so coordinates match at 125% and 150% scaling.
  • A crash can't take it downUI Automation runs in a helper process that restarts itself.
  • Works offline when it has toF10 replays the last run exactly, with no AI calls.

Built in a day with Codex and Claude Code working in parallel: one on the brain, one on the overlay.

For judges

Try it yourself.

About five minutes on any Windows 10 or 11 machine with Python 3.11+ and an OpenAI API key.

Set up

> git clone https://github.com/Arham-Begani/ghost-hands
> cd ghost-hands
> pip install -r requirements.txt
# .env
OPENAI_API_KEY=sk-...
> python main.py

No key? python main.py --fake plays a scripted run of the whole overlay with no API calls.

Hotkeys

F8 holdTalk. Release to run.
F7Type the task. For noisy rooms.
F9Your turn. Coaches the last run.
F10Replay the last run. No AI calls.
EscStop everything.
Move mouseTake over mid-run.

Safety

The irreversible click is always yours.

The stop lives in code, not in a prompt the model could talk its way around.

  • Hard stop before every clickIt reads the accessibility name under the pointer and the step's own caption, and refuses Send, Pay, Delete, Submit, Post, Buy and Transfer.
  • Grab the mouse and it lets goA deliberate move or click stops the run on the spot.
  • Human pace on purposeYou can't learn from a blur, and you always see what it's about to do.
  • Safety checks are never auto-acknowledgedIf the model raises one, control comes back to you and the run ends.
Stopped before Send. That one is yours.

Refuses send · pay · delete · submit · post · buy · transfer · checkout · place order · sign out