RAG isn't about AI; it's about engineering

(work in progress)

RAG is an engineering challenge

While browsing the AI & Machine Learning Community group, I came across a post [1] and a presentation by Hastika Cheddy that hit the nail on the head. Her thesis on AI deployments was brutally honest:

“I audited 10+ enterprise RAG deployments. Every single one collapsed in production. Not because of the LLM. Not because of the embedding model. Because of broken architecture.”

These words resonated with me instantly. Since the foundations fail more often than the models themselves, it’s time for me to add my two cents to the discussion on building effective RAG systems.

RAG: 5 minutes in a tutorial, 5 months in production

Analyzing commonly available examples of RAG systems, it’s hard to escape the impression that we are stuck in a phase of naive and rushed implementations. This technology, while it promises the world, is in reality extremely demanding and temperamental. The temptation to quickly fire up a simple sequence – “slice text into pieces, throw it into a vector database, search” – is so strong that key engineering foundations often take a back seat.

Aspects such as reliable data quality, process auditability, or resistance to attacks are given dangerously low priority. To make matters worse, fundamental design decisions are made on the fly. An example is the mass use of fixed-size chunking, which, when faced with real-world business documents, almost always loses to more sophisticated, semantic segmentation techniques.

Of course, there are more advanced RAG systems that have solid architectural foundations and work quite efficiently in production environments. However, they are not commonly discussed in tutorials – perhaps because they don't seem as appealing to those eager for quick results. They are associated with the long and meticulous building of a complex, multi-layered system.

In my view, it is precisely this complexity that is the core of the issue and the most interesting part of it. Therefore, I encourage you, the reader, to look at RAG as an engineering challenge. Here, we are not looking for "low-hanging fruit," but approaching the subject with senior-level seriousness – just as one approaches the design of any critical and complex system.

From three tiles on a diagram to a hundred-headed monster

Most diagrams we encounter online present RAG as an elegant and almost maintenance-free mechanism. The image is tempting: a document enters the machine, turns into vectors, passes through "three blocks total," and generates the ideal answer. Unfortunately, this vision of the “happy path” rarely survives its first collision with reality.

Places where these schematic simplifications mask the actual complexity of the process are plenty:
(TBD: list of examples goes here)

In the real world, your data is not sterile text files, but "dirty" PDFs with multi-level tables, footers, and non-obvious structures. On top of that, there are users who rarely ask questions in a predictable and orderly manner. For the system to stop hallucinating and start actually solving problems, we must abandon the illusion of simplicity in favor of solid engineering that secures the process at every stage.

Why a production system needs 15+ modules, not just three?

The true anatomy of a professional RAG system does not boil down to just the three simple steps suggested by the acronym itself: Retrieve, Augment, and Generate. That is only the tip of the iceberg. If you are building a solution that is meant to be resilient to messy data, confused users, and even malicious manipulation attempts, you must look much deeper.

Below you will find a list of 20 components that should be considered – and in business conditions, usually simply implemented – for RAG to stop being just a technical curiosity and become a reliable enterprise-grade tool.

Only an architecture that takes these aspects into account allows for the creation of a system ready for crisis situations, audits, or potential legal disputes. It is worth remembering one principle: in RAG, the greatest engineering effort is not at all related to the choice of the language model itself, but to the construction of the “pipes” through which the data flows. It is precisely in this invisible-at-first-glance infrastructure that it is decided whether your system will be real support or a spectacular deployment failure.

RAG is not a ready-made "out of the box" product, but a continuous process. Success does not depend on buying a subscription to the most expensive LLMs, but on the painstaking work on information quality and flow. Cognitive dissonance will disappear only when we stop treating AI like magic and start seeing it as another complex element of modern software architecture.

...work in progress...