Why OpenAI Still Needs Human Engineers Even With Advanced AI Agents
OpenAI Built an AI System That Investigates 17 Software Problems Every Day Without Waiting for Engineers
Software companies have a new problem. AI can write code faster than ever, but faster development also means more changes, more opportunities for mistakes, and more pressure on engineering teams.
Why OpenAI Still Needs Human Engineers Even With Advanced AI Agents
OpenAI has found a way to address part of this problem. Its internal system, called Performance Factory, uses Codex-powered AI agents to investigate software performance problems, identify their causes, propose solutions, and check whether those solutions actually work.
The numbers are interesting. In just over three months, the system handled more than 1,700 investigations and identified the root causes of over 600 cases.
That is approximately 17 automated investigations every day.
But the bigger story is not that AI can find software problems. It is that OpenAI is building an engineering system where multiple AI agents can work together, challenge each other's conclusions, and bring human engineers into the process when important decisions are required.
OpenAI Is Turning Software Debugging Into an Automated Process
Previously, when ChatGPT became slower, engineers had to investigate the problem manually.
They examined logs, studied performance measurements, followed requests across different services, and reviewed recent code changes.
OpenAI had already developed more than 30 specialized tools to help engineers perform these investigations.
Codex introduced a different approach. Engineers could ask the AI why a service had become slower, allowing it to inspect code and technical information.
However, engineers still needed to guide the investigation.
OpenAI eventually developed a reusable Codex skill called perf-forensics. It taught the AI how to follow the same investigation procedures used by experienced performance engineers.
The team then connected performance alerts directly to cloud-based Codex sessions.
This removed the need for engineers to manually copy alerts into AI conversations and transfer investigation results into Slack.
The next step was more important. OpenAI realized that every performance problem could become an independent task managed by AI agents.
That idea became Performance Factory.
The Four Components Behind Performance Factory
Performance Factory is not simply a chatbot connected to a codebase.
It is an internal system that organizes engineering work into tasks and predefined workflows.
Its architecture has four main components.
Task service
This stores each investigation, its current progress, findings, and important information. Engineers define workflows using configuration files and Markdown instructions.
Dispatch loop
This component checks which tasks are ready for the next stage and starts a Codex session to handle them.
Cloud Codex API
AI agents operate in controlled cloud environments with access to OpenAI's code repositories and observability systems. Permissions and agent actions can be monitored.
Event subscriptions
Tasks can wait for external events, such as an engineer reviewing a pull request on GitHub, before continuing.
The important idea is that agents do not need to solve everything in one conversation.
One agent investigates. Another reviews. A later agent proposes a fix. The task maintains its history so work can continue without repeatedly starting from zero.
Two AI Agents Are Better Than One
One of the most interesting lessons came from a weakness in AI reasoning.
OpenAI discovered that Codex sometimes stopped investigating after finding the first explanation that appeared reasonable.
But a reasonable explanation is not necessarily the correct explanation.
To address this, OpenAI introduced adversarial review.
A second Codex session receives the investigation record with a fresh context. Its job is to challenge the original diagnosis, search for missing evidence, and test alternative explanations.
In one example, an agent blamed failed ChatGPT conversations on image download errors.
The reviewing agent noticed that the internal service responsible for locating files had already recovered, while image requests continued failing.
That meant the original explanation was incomplete.
The investigation had to continue.
Currently, approximately 25% of diagnoses are returned for further investigation through this review process.
This is an important design lesson. AI systems should not automatically trust their own first answers, particularly when those answers influence production software.
The Numbers Reveal Both Progress and Limitations
OpenAI's reported results show meaningful progress, but they also reveal how much work remains.
| Performance measurement | Reported result |
|---|---|
| Investigations | More than 1,700 |
| Root causes identified | More than 600 |
| Cases reaching proposed fixes | Around 400 |
| Shipped fixes influenced by or matching agent suggestions | Around 50 |
| Exact Codex-authored pull requests shipped | Around 15 |
| Diagnoses returned for additional investigation | Around 25% |
The difference between identifying a problem and delivering a reliable solution is significant.
Codex can investigate many cases, but relatively few fixes have been shipped as exact AI-authored pull requests.
That does not make the system unsuccessful.
Finding the cause, understanding the impact, tracking the problem, and verifying recovery are valuable engineering activities. Automating these tasks can save substantial human attention.
However, the figures do not establish how much engineering time or infrastructure cost OpenAI has saved. The original report does not provide those measurements.
That is an important limitation when evaluating the business value.
OpenAI Is Building an Internal Engineering Platform
Performance Factory creates value by reducing repetitive engineering work.
Consider a software company with hundreds of services. Every performance problem can consume time from developers, infrastructure specialists, and technical managers.
If AI can investigate routine incidents automatically, engineers can spend more time building products and improving architecture.
This is particularly valuable for OpenAI because ChatGPT depends on complex infrastructure. A slowdown in one service can affect the experience of millions of users.
The system currently focuses on performance regressions, but OpenAI plans to expand it.
Potential applications include comparing CPU usage, memory consumption, and request traces across software releases.
The team is also exploring service rightsizing, which means identifying inefficient infrastructure configurations and improving resource usage.
Eventually, OpenAI wants some low-risk fixes to be deployed without human approval.
This could turn Performance Factory from a reactive maintenance tool into a continuous optimization platform.
The Competitive Advantage Is the Workflow, Not Just the AI Model
Many companies can access AI coding assistants.
The harder challenge is connecting AI to real business operations.
Performance Factory combines several capabilities:
- Access to internal engineering information.
- Specialized investigation instructions.
- Task management across multiple agents.
- Independent review of AI conclusions.
- Human approval for important changes.
- Automated verification after deployment.
This creates a system that is more useful than an AI assistant working independently.
The competitive advantage comes from connecting AI to company knowledge, operational data, and existing engineering processes.
Competitors can build similar systems, but reproducing years of internal engineering knowledge, reliable workflows, and operational discipline takes time.
The Biggest Risks Are Trust, Security, and Incorrect Decisions
Automation creates new risks.
An AI agent with access to production-related systems could misunderstand an incident, propose an unsafe change, or waste resources investigating the wrong problem.
OpenAI's adversarial review helps reduce incorrect conclusions, but it does not eliminate them.
Another challenge is information overload. Engineers initially received too many investigation results and had to determine which ones mattered.
OpenAI responded by grouping related incidents, ranking estimated impact, and generating shorter reports with graphs and metrics.
Security is equally important. AI agents need access to internal repositories and observability systems, but they should not receive unlimited permissions.
For businesses adopting similar systems, access controls, activity monitoring, approval processes, and rollback procedures are essential.
The Legal and Business Weakness Behind AI Engineering
From a competition and consumer-protection perspective, the most important question is not simply whether AI makes software development faster.
It is who controls the infrastructure, operational data, and knowledge required to make these systems work.
Companies with large engineering teams and extensive internal data can build sophisticated AI workflows. Smaller competitors may depend on external AI providers for similar capabilities.
This creates a potential imbalance.
There is also a transparency problem. OpenAI reports impressive investigation numbers, but the available figures do not establish the financial savings, average resolution time, reduction in customer-facing incidents, or long-term accuracy of the system.
The weakest part of the evidence is the gap between diagnosis and implementation.
More than 600 investigations reached a root cause, but only around 15 shipped fixes were exact pull requests authored by Codex.
This suggests that AI engineering is still more reliable at investigating problems than independently delivering production-ready solutions.
Businesses should distinguish between an AI system that produces convincing answers and one that consistently creates measurable business results.
What Small and Medium Businesses Should Learn
Small businesses do not need to build something as complicated as Performance Factory.
They can start with simpler workflows.
Automate repetitive technical investigations. Use AI to examine error reports, summarize logs, and identify possible causes.
Create reusable company knowledge. Record troubleshooting procedures so employees and AI tools can follow consistent steps.
Use independent verification. Ask a separate review process to challenge important AI-generated recommendations.
Keep humans responsible for important decisions. AI can propose changes, but people should approve risky deployments.
Measure business results. Track investigation time, downtime, recurring errors, repair costs, and customer impact.
The goal is not to replace every developer. It is to allow a small team to handle more work without increasing its workload at the same rate.
For an SME struggling with limited budgets and technical staff, that can become a meaningful competitive advantage.
The Next Stage of AI Is Managing Work, Not Just Writing Code
Performance Factory offers a glimpse of a different future for software development.
The first generation of AI coding tools helped developers write code faster.
The next generation is beginning to manage entire workflows, investigate problems, coordinate specialized agents, and verify results.
OpenAI has not yet achieved fully autonomous software maintenance. Its own numbers show that human judgment remains essential.
But the direction is clear.
The companies that benefit most from AI may not simply be those that generate the most code. They may be the ones that build the best systems for turning AI-generated work into reliable business outcomes.
For OpenAI, keeping ChatGPT fast is the immediate goal.
The larger opportunity is building an engineering operation where software can increasingly monitor, investigate, and improve itself.
Why OpenAI Still Needs Human Engineers Even With Advanced AI Agents
OpenAI Performance Factory, Codex AI agents, AI software engineering, automated debugging, software performance optimization, AI coding automation, ChatGPT infrastructure, AI incident management, agentic workflows, software regression testing, AI engineering tools, automated code review, enterprise AI automation, small business AI
Editorial note: The supplied article does not contain actual Reddit, X, or Product Hunt community reactions, nor does it provide financial savings or independent customer evaluations. I have therefore avoided inventing community consensus or presenting unverified financial benefits as facts. The legal and competitive analysis is an independent analytical perspective, not a statement attributed to any named lawyer.
Share: