AI could leave us living among digital equivalents of Roman aqueducts – systems that remain impressive and useful, but which the people depending on them can no longer confidently explain, repair or recreate.
What the Romans Left Behind
After the Western Roman Empire declined, people continued to live among roads, bridges, walls and aqueducts built by an earlier civilisation. These structures did not disappear overnight. Some remained useful for generations, while others were adapted, dismantled or gradually allowed to fail. What weakened was not only the physical infrastructure, but also the network of knowledge, institutions and skilled labour that had made it possible.
An aqueduct was never simply a line of arches. It depended on surveying, mathematics, materials, administration, maintenance and people who understood how the entire system worked. Once enough of that support disappeared, the aqueduct became something people could still use and admire, but not necessarily reproduce. The water might continue to flow for years, yet the ability to rebuild the system was already fading.
The Modern Parallel
That is what makes the comparison with AI uncomfortable. We are becoming able to produce software at extraordinary speed. A person can generate an application, connect it to a database, add authentication and deploy it without fully understanding what has been created. The interface may look polished, the code may appear professional and the system may work perfectly well.
The weakness only becomes visible when something changes. A service is withdrawn, a dependency breaks, a security flaw appears or the data behaves in an unexpected way. The original developer may have moved on, the documentation may be incomplete and nobody may be certain why parts of the system were designed as they were.
At that point, the organisation still owns the software, but ownership is not the same as understanding. It has inherited the aqueduct, but it may no longer have the engineers.
The Warning in Foundation
Isaac Asimov explored a similar idea in Foundation, now familiar to a much wider audience through the television series. The story was shaped by the decline of the Roman Empire, but it moved the pattern into a distant galactic future. As the Galactic Empire begins to collapse, its technology does not immediately disappear. The machines remain, but the scientific culture needed to understand, repair and reproduce them begins to weaken.
This is what makes Foundation so relevant to the age of AI. The frightening future is not necessarily one in which the machines suddenly stop working. It is one in which they continue working for long enough that nobody notices the builders have disappeared.
Advanced technology can survive for a time after the understanding behind it has weakened. People may still know how to operate a system without knowing how to recreate it. They may be able to follow procedures without understanding the principles. The technology remains visible, but the culture that produced it has started to disappear.
Working Is Not the Same as Understood
Software has always been complicated. No single person understands every part of a modern operating system, aircraft or banking network. That is not the problem. Complex systems are built through specialisation, with different people understanding different layers.
The problem begins when no group of people understands enough of the system to challenge it, maintain it or rebuild it. AI-generated code can create the appearance of understanding without the substance. It may contain sensible variable names, neat comments, convincing documentation and even tests. Yet it can still hide weak assumptions, security flaws, unnecessary complexity or decisions that nobody has properly examined.
A program can work and still be badly understood. It can produce the correct answer for normal inputs and fail at the edges. It can survive for years while the people around it become steadily less capable of repairing it.
How Skills Are Lost
The loss of skill would not arrive dramatically. It would look efficient. A student stops tracing code because an AI can explain it. A developer stops debugging because it is quicker to generate another version. A company stops recording why decisions were made because the software appears to work.
Each shortcut saves time, but each one also removes a little more human understanding from the system. Over time, the ability to produce software may grow while the ability to explain it weakens. Society could end up with more code than ever before and fewer people who can confidently read, test and repair it.
That matters because software is no longer confined to websites and office tools. It controls education, healthcare, transport, finance, communication and public services. The more important the system, the less acceptable it is for nobody to understand it.
A hospital cannot respond to a failure by saying that the AI wrote the code. A school cannot protect student data by saying that the system seemed reliable. A government cannot justify an automated decision by pointing to a model. At some point, someone must be able to explain what the system is doing, why it is doing it and what happens when it goes wrong.
Why Coding Still Matters
This is why learning to read and write code still matters. The point is not that humans need to compete with AI at typing syntax, and it is not that every command must be memorised. The point is to preserve understanding.
Reading code matters because generated code can be wrong. Writing code matters because it exposes whether someone really understands the problem. Debugging matters because real systems fail in ways that prompts do not predict. Explaining code matters because important systems cannot be allowed to become technical folklore.
AI can still support this learning. It can help beginners get started, explain unfamiliar ideas, suggest tests and speed up development. It can make programming more accessible and allow people to attempt projects that once felt out of reach.
The risk begins when AI replaces the thinking rather than supporting it. Acceleration without understanding is not progress. It is dependency.
Keeping the Builders
The real test of an AI-generated system is not whether it works today. It is whether people can still understand it tomorrow. Can they explain the important decisions? Can they identify what could go wrong? Can they change it safely? Can they rebuild the damaged section?
The Romans left behind aqueducts. Asimov imagined an empire leaving behind machines. We may leave behind millions of lines of generated code.
The question is whether we also leave behind people who can understand them.
We should build with AI. We should use it to move faster, create more and attempt more ambitious ideas. But we should not build a world in which only AI knows how the world works.
Otherwise, we may keep the aqueducts and lose the builders.
