Getting Past the Pilot Often Starts With Selecting the Right Technology

Many pilots of AI and other advanced technologies begin with real enthusiasm and then lose momentum without reaching a clear conclusion.

Various industry studies suggest that between 70 and 80 percent of technology transformations stall before reaching operational maturity, and the current pressure to be seen adopting AI has arguably made this more common.

It is such a widespread issue that I dedicated a large part of my book, Industrial Safetytech, to why it happens and how it can be addressed.

A client once put it to me in a way that has stayed with me:

“I would love to use AI for X, but the tech I have seen is not quite right for us yet.”

It brings to mind that saying: “water, water, every where, nor any drop to drink”. Many leaders today find themselves in a similar position, surrounded by an ocean of new technology and still unable to find something that fits.

That hesitation is understandable, and it often says as much about how technology is found and assessed as it does about the technology itself.

What tends to surprise people is how seldom the technology is the main cause. According to research by the World Economic Forum and McKinsey on industrial sites adopting advanced technology at scale, the barriers are more often organisational than technical. In many cases the difficulty can be traced back to how the technology was chosen.

Choosing well involves a good deal more than the capabilities of the technology itself. It also means looking at how the technology fits with existing processes and systems, the people who will use it, the conditions it will operate in, the regulatory and data governance requirements that apply, and the supplier’s ability to support it over time.

The good news is that there is a solid body of established practice to draw on, and I would like to share a few of the points that, in my experience, tend to make a real difference.

Focus on the problem, not the tech

Understanding the problem thoroughly before looking at any technology means being clear on several things: where the stakes are highest in the operation, what outcome is needed, the context in which a solution would have to work, the constraints it would have to respect, such as site conditions, regulation and the workforce, and how things stand today compared with where they need to be. Two of these are worth singling out.

The first is to resist starting from picking the technology. A team that starts from “we need a computer vision solution like this” has in effect already made its decision. A team that starts from “we need to spot risks before they reach the shop floor” still has an open question, and the answer may turn out to be a different technology or supplier.

The second is the baseline, which means measuring how things stand today and where they need to be. For instance, a site that spots, say, six in ten risks before they reach the shop floor and wants to reach nine in ten has a clear gap to close, and every candidate technology can be judged on how far it closes that gap. Without such a baseline it becomes difficult to select the right tech, let alone show any improvement later on.

Look also beyond the usual suppliers

Most organisations turn first to vendors they already know. For mature technology that is often sensible. This market moves quickly, though, and a strong candidate may come from a neighbouring sector or from a company the team has not yet encountered. Organisations short of the time or contacts for a wide search may be better served by outside help than by narrowing the field too early.

Be systemic in checking fit and vendor claims

Vendors naturally present their technology at its best, and performance in a controlled demonstration does not always carry over to a working site. With several suppliers each making strong claims, the challenge is to compare them on equal terms.

A balanced scorecard is a simple way to do that. Each candidate is scored against the same short list of criteria, weighted by how much each one matters and built around the particular problem, organisation and infrastructure involved, since the more specific the criteria, the more useful they tend to be. They go beyond technical performance to cover fit with existing processes and systems, the people who will use the technology, the conditions on site and the regulation that applies.

A common mistake is to focus on what would make a good pilot, with little thought for deployment and the full lifecycle, and the more forward looking the criteria are, the more robust they tend to be. Not everything can be worked out in advance, though, and where important questions cannot be settled on paper, a short feasibility study before the pilot is usually a modest cost for the reassurance it brings.

Luckily for everyone, AI can now speed up and deepen much of this due diligence. Large language models can gather and summarise technical documentation, case studies and published reports in a fraction of the time it once took. The harder part remains defining what really matters to the organisation and how to measure it, and that judgement still rests with the people who know the operation.

It is also important to bear in mind that a systemic assessment looks beyond the vendor. It will inevitably reach inside the organisation as well, covering matters such as the quality of its data, the availability of its assets and the maturity of its people and processes.

What is the pilot for?

A pilot exists to answer the questions that can only be settled by trying the technology in real working conditions, such as whether it performs on site as claimed and whether people will use it.

There is always a temptation to do too much, or too little, but best practice suggest that a good scope of work needs to be focus on what answering those questions require.

The site and the operational setting chosen for the pilot matters as well. One that resembles what real deployment would look like is likely to give results that can be relied on to decide on live deployment, or applicable elsewhere (another ship, another factory, etc).

The right pilot design also makes it easy to measure whether those questions, such as how much performance has improved, have been answered. Real working conditions involve many factors at once, so the design needs to separate the effect of the technology from everything else going on around it.

Success criteria are far more useful when agreed before the pilot starts. Written afterwards, they are easily shaped around what the pilot achieved, a tendency that has been widely reported in the management literature. A fixed decision date helps too, along with a named person who has the authority to make the call.

Governance deserves the same care. A pilot should in principle start, stop and lead to a decision, yet more often than not it either drifts into limbo or ends without a decision being taken. Solid governance from the outset, with a clear end point and agreement on who decides and on what evidence, goes a long way towards avoiding both.

Deployment starts at selection

Good governance leads to a decision, and sometimes that decision will be a no. When it is a yes, the moment of truth arrives and the technology goes live. This is often when an organisation discovers the things nobody had thought of, and it is the stage at which many successful pilots struggle to deliver.

Much of the explanation lies in how change is managed. A pilot runs inside a carefully created bubble, with a dedicated team, close attention and willing participants. Deployment removes that bubble, and the technology has to earn its place among people who had no part in choosing it and alongside routines that worked well enough before. John Kotter’s work on leading change remains one of the most cited references on this human side of transformation.

This matters for technology selection because the amount of change required depends heavily on the technology itself, including how intuitive its interface is, how it handles data and how easily it connects with existing standard operating procedures. It is therefore worth asking, while the choice is still open, how much change each option would demand, who would lead it and where resistance is likely to come from. Not everything can be known in advance, as noted earlier, but options that ask less of the organisation, or ask it in ways the organisation is ready for, tend to have a smoother path into daily use.