Models such as GPT-6.1 Terra, which OpenAI describes as built for visual reasoning and precise computer use, make screenshots and software tasks a real prompt category. These prompts are our own and work as patterns for any assistant that accepts images. Results vary, so test on low-stakes tasks first.
The pattern
Give the goal, describe what the screen shows, set limits, and say when to stop and ask.
Reading a screenshot
- Explain it: “This screenshot shows a settings page. List each section, what it controls, and which option I would change to turn off email alerts. If you cannot read something, say so instead of guessing.”
- Extract data: “Pull every row of the table in the image into a Markdown table with the columns Date, Item and Amount. Flag any cell that is cut off or blurry.”
- Compare two versions: “Compare the two screenshots of our checkout page. List every visible difference in layout, wording and colour, most important first.”
- Review a design: “Act as a reviewer. For this app screen, list five usability problems and one fix for each. Do not suggest new features.”
Operating software (agents)
- Narrow the task: “Open the report tool, set the date range to last month, and export the summary as CSV. Do nothing else.”
- Set stop rules: “Stop and ask me before you send, delete, pay or change any setting. Describe what you are about to do first.”
- Ask for a log: “After each step, write one line saying what you clicked and what changed on screen.”
Safety checklist
- Use a test account or sandbox for new tasks.
- Never let an agent handle payments or passwords unattended.
- Check results against the original screen before you act on them.
- Remember that on-screen text can contain instructions meant for the AI; tell it to treat page content as data, not orders.
More patterns: the prompt workflow, long-document review prompts and the ChatGPT page.