We’ve found that the most useful applications involve building purpose-built tools that engineers can verify and reuse.

Where the real gains are in 2026

Caliber’s engineers work embedded in client teams across a wide slice of New Zealand engineering, which gives us a useful vantage point on what’s actually working. Our approach to AI has been engineer-led: make sure the team understands the risks, give them access to good tools, then let them figure out how best to integrate it into their workflows. Every quarter we survey the team for their best use cases so we can share the best practices with the team and our clients (making sure we don’t release sensitive info in the process). We score each use case on how trustworthy the output is and how much impact it has on the engineering process.

When the results came in, the standout use cases clustered in the high-trust, high-impact corner. The more striking finding was that every one of them worked the same way. None of the best use cases put AI in the live workflow. Instead, someone used AI to build a tool (a script, a macro, a purpose-built calculator) and then used that tool to do the work.

Picture1

Where the standout use cases sat in our team survey.

That distinction matters because a tool can be checked. It goes through the same verification and validation processes we already use for everything else: hand calcs, reference cases, comparison against the old spreadsheet. You don’t need to trust what the model produces on any given day, because the model isn’t doing the work. So our advice right now is not to ask AI to produce the answer. Instead, use AI to build tools, then check the tools.

This approach takes care of much of the IP problem. If the AI only builds the tool, only the build data needs to go anywhere near it. The model can build and test against dummy or sanitised data, and the finished tool then runs locally on the real project data. Sensitive client information never needs to reach the model provider.

A macro that checks drawings or a script that automates file handling isn’t particularly exciting. But these are the kinds of applications we are seeing deliver real value in clients’ businesses.

We walked through real examples and two live-built demos in our Engineering New Zealand webinar, so if you want the detail, that’s the place to go. This article looks at what that shift does to the nature of engineering work.

This change is happening unevenly across the spread of tasks that makes up an engineer’s day. AI might be extremely effective for drafting spreadsheets or calculators but next to useless for packaging in a machine design. This unevenness has been called the jagged frontier[1]: tasks that look similar from a human perspective can sit on opposite sides of the line. The discussion below relates to those tasks AI is currently well suited to. We expect the frontier to keep moving quickly, and many of the tasks that sit outside AI’s capabilities today to move inside them. Where that line sits for a given task is worth re-testing every few months.

Engineering is moving up the stack

So, what does an engineer working effectively with AI look like in 2026? It increasingly resembles what an engineering lead did a few years ago.

Because generating output has become so cheap, producing the output is no longer where most of the effort goes. What’s left is the work that used to sit with the lead or principal engineer: specifying what a good outcome looks like, working out what the project really needs, and verifying and validating work done by others. The difference now is that “others” includes the machine, and this work is landing on far more engineers, much earlier in their careers.

The value was hiding in plain sight

Most value in engineering was always in the critical thinking applied, not in the output artefact produced. It was unnecessary to say so explicitly, because the output was evidence of the thinking. You couldn’t produce a drawing package or a calc pack without thinking your way through it, so the artefact stood in for the thought behind it.

With AI, output is cheap and no longer proves anyone really thought through a design. That’s why the work is moving up the stack. The thinking has to be done explicitly, because

convincing-looking output is no longer a reliable indicator that someone has thought thoroughly about the problem.

Shopify CEO Tobi Lütke framed this well[1] as a change in failure mode. Lazy work used to show up as missing output (the unfinished report, the calc that never got done), which was easy to spot. Now it shows up as lots of output, unchecked and without good thinking behind it, which is much harder to spot because it looks like what we’ve always recognised as work.

The clearest example is the slop grenade. The term describes a big pile of AI-generated output that gets lobbed at someone else to check. Slop grenades are cheap to produce and expensive to receive, because the cost of checking lands on whoever catches them. A useful test for any AI workflow is to ask whether it’s speeding the project up, or just making me feel faster while slowing everyone else down.

Both demos from our webinar produced slick-looking tools, and both would be slop grenades if I handed them to someone without proper verification.

The same trap shows up at a personal scale. Some uses of AI feel like productivity but remove the moments where thinking happens. An agent-written summary waiting for you each morning works fine, but those fifteen minutes spent writing out your task list for the day were actually critical thinking masquerading as admin overhead.

Doing the thinking on purpose

So the thinking has to be done on purpose, and mostly up front.

In practice, that starts with outcomes rather than the build. It’s very easy to start building a tool, run down a track feeling like you’re making great progress, and discover a couple of hours later that you didn’t actually need to know what that tool tells you. A few minutes deciding what success looks like saves a lot of wasted effort. Then decide how you’ll check something before it exists: hand calcs, a reference case from a source you trust, an old tool to compare against, counts in and counts out.

The bottleneck has moved to judgement. Is this the right problem? Is the result plausible? Would I put my name to it?

Your checking process isn’t broken

Much of the hype and the panic assumes our existing checking procedures can’t cope with AI. In our experience they cope fine, and engineers should be leaning on them harder, especially when building tools. Document control, version control, independent checks and sign-off work for people plus AI just as they do for teams of people.

The question we got most at the webinar was some version of “how much should I check AI output?” The answer is the same as for anything else. How much rigour you apply comes down to two things we already understand well. First is consequence: if you’re exploring concepts early in a project, you don’t need to chase down every reference, but high-consequence work gets thorough review regardless of who performed it. The other is professional integrity. You don’t put your name to work you aren’t willing to stand behind. Neither of those questions hinges on whether the work came from a model or a graduate.

Part of good checking is also knowing when to stop. Ask a second model to review the first model’s work (or indeed your own) and it will invariably find something wrong. However, not every criticism identifies a meaningful defect. Part of good engineering judgement is distinguishing a genuine technical problem from an immaterial issue and knowing when the work is sufficiently verified for its intended purpose.

The worry that AI will soon be smarter than us, and therefore impossible to check, doesn’t hold up that well either. I’m already normally not the smartest person on a given project. Engineering leads routinely sign off work from specialists who are far better in their niche than the lead is, and they do it with independent checks such as hand calcs, sanity bounds and physical testing. We already have processes for checking work from potentially unreliable humans, and they can be extended to cover machines.

The catch is that those processes only work as well as the people running them. In 1983, Lisanne Bainbridge described the “ironies of automation”[1] in process control and aviation, observing that the more reliable the automation, the worse people get at monitoring it, and the more they lose the hands-on skill needed to step in when it fails. Engineering checks are no different. A hand calc catches an error because the person doing it has done hundreds of them and knows what a wrong answer looks like. If AI takes on more of the doing, that instinct must be kept sharp deliberately.

The open questions

We don’t have good answers to the next three questions yet.

1. The thinking hidden in the doing

If the thinking is the value, where does it happen once the doing goes away?

Take CAD. After ten thousand-odd hours in SolidWorks, the modelling itself becomes pretty mechanical: sketch, extrude, loft, repeat. But while my hands are busy, an awful lot of design thinking is going on in the background. How will this part be manufactured? Can someone get a spanner on this fastener? Does it clash with the frame? Is that load path a bit awkward? The clicking became a mechanical skill, leaving time and headspace for much of the design thinking.

Hand calcs, drawing markups and report writing all host the same hidden thinking. Our background thinking has always happened while our hands were busy.

This sits in some tension with moving up the stack, which assumes the thinking can be front-loaded into a good spec. Some of it can. But some of it only shows up while you’re doing the work. How do we capture that when the point and click is done by a machine? Much of that thinking was tacit and incidental; nobody wrote it down or planned for it, because it just happened while we worked. How do we make it deliberate?

This is also a moving target. Models will keep getting better, and at some point they may well be trustworthy enough to do much of the design thinking that used to hide in the doing. How far that goes, and how much needs to stay with a human, is genuinely unknown. In the meantime, part of the answer will be deliberately making time for that thinking, which raises the question of who pays for that time.

2. The commercial reality

Every engineering business in New Zealand is going to have to decide how to use the gains from AI. Think of it as a sliding scale. At one end, you do the same work in half the time. At the other, you take the same time and deliver double the quality. Every firm will land somewhere along that scale.

Competition will often push hard towards half the time. Quoting is competitive, clients notice price, and output is now cheap. But if output is cheap and the thinking is the value, then racing to the cheapest output starts to look like slop grenades at an industrial scale.

The most impressive firms we have seen in industry are landing somewhere in the middle. They are delivering projects that they otherwise would not have the manpower to deliver and on shorter timescales. But they are also being much more rigorous in investigating the problem space to deliver higher quality solutions.

Where firms put those gains matters beyond the bottom line, too. Developing young engineers takes time, and whether that time exists depends on the choice above.

3. Young engineers

This one is a big concern for juniors, for Engineering New Zealand, and for us.

Long division by hand hasn’t been a useful skill since calculators arrived, and nobody mourns it. Long term, we think AI-enabled engineering will be fine (and it will eventually just be called engineering). We’ve had the same panics with typewriters, word processors, calculators and Excel, and we still do good engineering. We’ll develop a slightly different skill set, and good pathways to build it.

The danger zone is the next two to five years, maybe a little longer. A lot of early-career learning happens by osmosis and by doing grunt work. The tasks that are easiest to automate are often the tasks that taught you the job. Under commercial pressure, it would be very easy for that learning to quietly disappear.

This is a close cousin of the ironies of automation from earlier. Seniors can check AI output today because they’ve spent years developing mastery by doing the work themselves. If the doing goes away, seniors risk having those skills atrophy, but a graduate may never build them at all.

Aviation’s answer to the ironies of automation was deliberate manual practice. Ours will probably look similar, which means actively making space for young engineers to do the work and build the skills, rather than leaving it to chance. The questions we’re working through are which skills are dropping out of the toolkit for good, what tacit knowledge those skills were carrying, and how we pass that knowledge on when the skill itself is no longer needed.

Engineering New Zealand is tackling exactly this through its Engineering and AI programme, and I’m part of the working group looking at it. If you’ve got thoughts, we’d love to hear them.


This article was provided by Aaron Johnson (Caliber Design), a member of the Engineering New Zealand Te Ao Rangahau AI Advisory Committee. The views expressed are those of the author.


References

  • [1] Dell’Acqua, F. et al. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. Harvard Business School Working Paper 24-013.
  • [2] The Knowledge Project with Shane Parrish, “Tobi Lütke: AI Agents, Better Decisions, and the Future of Work.” https://fs.blog/knowledge-project-podcast/tobi-lutke-3/
  • [3] Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775-779.