IT Brief US - Technology news for CIOs & IT decision-makers
United States
Google launches engineering benchmark for frontier AI

Google launches engineering benchmark for frontier AI

Mon, 16th Nov 2026 (Today)
Sofiah Nichole Salivio
SOFIAH NICHOLE SALIVIO News Editor

Google AI & Infrastructure has launched Physical-Engineering's Last Exam, a benchmark and open competition for testing AI models on engineering work. The project was developed with Mercor.

The benchmark is intended to assess whether frontier AI models can complete tasks in mechanical, electrical, civil and structural engineering. Google is releasing 29 public tasks, grading tools and an open leaderboard so researchers can test and compare systems.

Benchmarks are already widely used to measure AI progress in specialist fields, but the team behind the new test argues that physical engineering has lacked a comparable standard. It focuses on work tied to the design of real-world systems and structures rather than textbook questions or short coding exercises.

Engineering tasks

Physical-Engineering's Last Exam, or PLE, includes 85 long-horizon tasks across four disciplines: 30 in mechanical engineering, 27 in electrical engineering, 18 in civil engineering and 10 in structural engineering.

Rather than respond to a single prompt, an AI agent operates inside a containerised project folder designed to resemble a professional engineering assignment. According to the published details, each task contains an average of 61 files, for a total of 5,170 across the benchmark, including PDFs, spreadsheets, emails, 2D CAD drawings, 3D BIM models and document revisions.

The agent is expected to review those materials, identify relevant building codes and physical constraints, run calculations and produce deliverables for checking by a licensed engineer. The benchmark was built with experts from Mercor and licensed practising engineers from industry.

Of the 85 tasks, 44 were authored directly by practising engineers, who created the project folders, prompts and grading rubrics based on professional work. The remaining 41 were co-authored: engineers designed the initial brief, an agent pipeline assembled the environment and grader from engineering standards, and engineers then validated the tasks and increased the difficulty.

Benchmark gap

The team said existing tests do not show whether an AI agent can produce safe, code-compliant and cost-effective engineering designs under the review and control of a licensed professional. It argued that many current academic benchmarks measure narrower skills, such as answering textbook-style questions, making a single tool call or solving a small, self-contained design problem.

That distinction matters because physical engineering tasks often involve multiple files, revisions, regulatory rules and trade-offs around feasibility and cost. PLE therefore awards points only when a design passes every required hard safety check before other factors are counted.

Early results

Google tested 14 frontier models on the benchmark using seven publicly available agent harnesses: Antigravity, Claude Code, Codex, Hermes, OpenCode, OpenHands and Pi. On its private test set, all models scored below 50%.

On its native harness, the highest-performing model achieved a mean score of 48.3%, according to the published results. The top-scoring agent configuration met 91% of individual constraints on co-authored tasks, but only 41.3% of its deliverables passed every required hard safety and feasibility check.

The results also suggest that the choice of underlying model has a much larger effect on performance than the choice of harness. Google reported up to a 50-point spread across models, compared with a median 3-point spread across harnesses for the same model, though it added that harnesses can still significantly affect run cost.

The launch adds to a growing set of AI evaluations aimed at measuring progress beyond general language tasks. Google positioned PLE alongside other specialist exams that test general knowledge or digital professional work, saying the new benchmark is intended to measure work involved in building physical infrastructure and industrial systems.

The public release covers 29 tasks rather than the full benchmark, along with grading scripts and an evaluation container. Researchers can also submit their own agent setups for evaluation on the private test set through the benchmark site, while 13 sponsored models are available to run on Google's servers at no cost.

In the baseline data published with the launch, the strongest model still fell short of a passing level on a benchmark designed to reflect the safety, feasibility and cost demands of engineering work, with a mean score of 48.3%.