Do engineers even use generative AI tools?

While generative AI for CAD as a market is predicted to grow to the high tens of billions, the successes we've seen are pretty much just academic benchmarks or company claims measured against each other and there is still a lack of clear, measurable success in industry.

Berger et al. tested four frontier models: Claude Opus 4.5, Claude Sonnet 4.5, Gemini 3 Pro, Gemini 3 Flash.^[1]

Their test involved a four agent pipeline for designing and evaluating designs, including an agent for planning, an agent for engineering, and VLMs for reviewing geometry and physical robustness.

They then tested how well each model made it through different stages of evaluation:

R1: Execution Success - Did the generated code run without a geometry error?
R2: Meshing Success - Could the resulting shape be meshed at all?
R3: FEA Success - Did the physics simulation run without erroring?
Target safety factor - Did the result of that simulation land in the safe range?

What they found was counter-intuitive: the model with the lowest failure rates was the smallest model. All models succeeded at syntax but failed consistently at one thing: geometric validity.

Gemini 3 Flash (the smallest model) produced geometrically or physically invalid designs only 8.7% of the time, while Sonnet 4.5 failed 85.7% of the time. And for the average model, 4 of every 10 designs produced would be invalid, even with a VLM in the loop.

The most common failure mode by a wide margin was violating the design space and producing disconnected geometry, a problem consistent across every model.

So, even though models could generate correct syntax and successfully mesh their design and run it through FEA, the result would still be physically and geometrically invalid most of the time.

This would be like if Claude Code spent 5 minutes building an app, it ended without testing the code, AND the result was wrong 4 out of 10 times. It would be a very expensive, near-pointless tool to use.

To produce more correct results more often, the problem comes down to how a frontier model can accurately test its output and iterate on test results.

When SWE-bench was first introduced, models acting in a zero-shot, single-turn capacity scored less than 4%, while being able to verify and iterate on their outputs, on average, tripled that score.^[2] Similar boosts to performance can be observed in almost any software engineering benchmark.

Another, potentially much more powerful consequence is that more reliable verification leads to better RL.

Reinforcement learning from verifiable rewards (RLVR) improvements are dramatically larger than any improvements that come from being able to verify and iterate in an agent loop.

And the performance improvements we are seeing are specifically high in verifiable domains: frontier models are now consistently achieving gold in the IMO and top scores in programming competitions.^[3] This was not conceivable more than 2 years ago.

If CAD can be made more verifiable, we will see both better inference-time performance of text-to-CAD agents, increasingly better improvements in models through RL. As a result: we'll get more reliable, synthetic CAD datasets (needed in areas such as robotics) which is a market, in itself, predicted to grow to $7B by 2030.^[4]

This is taken from my own essay on The State of AI-Generated CAD in Industry.