ChatGPT Can Now Read Your Screen to Help Build Apps Faster

Here's how you can use ChatGPT's latest "Work with Apps" feature to help you build apps faster on programming tools like Xcode, VS Code, and even terminal apps.

The next big trend in AI is here—AI chatbots controlling your PC.

Just a little over a month ago, Anthropic introduced a feature on Claude called Computer Use, allowing it to control desktops with text-based commands. Soon after, Microsoft revealed Omniparser, a research preview for an AI agent that can parse your screen—likely a hint on an upcoming feature to control a user’s desktop, most likely through Copilot.

While Apple and Google have been quiet about similar tools, it’s safe to assume they’re working on something behind the scenes.

Today, OpenAI released a new feature called “Work with Apps” on the ChatGPT desktop app for Mac. This feature lets ChatGPT control programming tools like Xcode, VS Code, and even terminal apps.

As a developer and an avid user of AI programming assistants, this is huge.

In this article, I’ll explain what the new ChatGPT capability is all about, how it works, and how to use it to build an iOS application similar to the screenshot below.

Let’s get started.