OpenAI says newest AI model can ‘think with images,’ understanding diagrams and sketches

OpenAI has released its latest artificial intelligence model, which it said is capable of “thinking with images,” meaning it can understand and analyze a user’s sketches and diagrams, even if they’re low quality.

The main new reasoning model from OpenAI is called o3, and the company simultaneously released a smaller model dubbed o4-mini. The rollout follows the September debut of OpenAI’s first reasoning model, o1, which focused on solving complex problems and deliberating over its answers in multiple steps.

With o3, users can upload whiteboards, sketches and other images and have the AI analyze and discuss them. The models can also rotate, zoom and use other image-editing tools.

Since introducing its viral ChatGPT chatbot in late 2022, OpenAI has been rapidly upgrading its models to go well beyond text into images, voice and videos. The company is racing to stay ahead in generative AI, where it faces heated competition from rivals including Google, Anthropic and Elon Musk’s xAI.

“For the first time, our reasoning models can independently use all ChatGPT tools — web browsing, Python, image understanding, and image generation,” OpenAI wrote. “This helps them solve complex, multi-step problems more effectively and take real steps toward acting independently.”

The company, valued at $300 billion in a funding round last month, said that o3 and o4-mini are its first AI models that can “think with images.” That means “they don’t just see an image, they can integrate visual information directly into the reasoning chain,” according to OpenAI.

Last month, OpenAI released a native image-generation feature that went viral online for its ability to produce Studio Ghibli-style anime images.

OpenAI said its o3 model is especially tuned for math, coding, science and understanding images, while o4-mini operates faster and at a lower cost. Both models were available starting Wednesday to ChatGPT Plus, Pro and Team customers.

OpenAI’s user community has long joked about its strange or confusing AI model names. CEO Sam Altman joined the banter this week, writing in a post on X, “how about we fix our model naming by this summer and everyone gets a few more months to make fun of us (which we very much deserve) until then?”

The company also said that both models had been “stress-tested under our most rigorous safety program to date” and linked to its “Preparedness framework” updated earlier this week.

OpenAI has come under fire recently for changes to its safety precautions and processes. It said this week that it reserved the right to “change its safety requirements if ‘another frontier AI developer releases a high-risk system without comparable safeguards.'”

In changing its policies this week, OpenAI wrote it would no longer require safety tests for certain fine-tuned models. The company also avoided releasing a “model card” — or a report containing information about safety tests done before a model is released — for its GPT-4.1 model. In February, OpenAI launched AI agent tool Deep Research weeks before publishing its system card.

OpenAI didn’t immediately respond to a request for comment.

WATCH: OpenAI considering building social network