Schrödinger's AI Part 9: Inside RAG

Schrödinger's AI is your invitation to look inside. Right now, AI feels like a mystery , wired like a brain, yet running on pure math.
Each article is a new layer of the box. We start with the first spark of an idea and move all the way to the models reshaping everything we thought we knew .

Part 9: Inside RAG

Back in 2023, talking to an AI felt like talking to a genius who’d been stuck in a cave for two years. They were incredibly smart, but they had no idea what was happening in the world.

When an AI gets stuck in the past or ignores your private environment, that’s the "knowledge cutoff." It’s a wall they can’t see past. And when an AI hits that wall, it starts to hallucinate, it just starts making things up because it's lost its way.

And that is where RAG (Retrieval-Augmented Generation) comes in as a fix. Instead of guessing based on old info or spinning wild stories, the AI can actually "look" at your data. It keeps the AI grounded in reality so you get an answer that’s real, current, and actually useful.

so we are trying to solve 3 problems,

First, knowledge can be outdated. Second, models may hallucinate incorrect facts. Third, they cannot access private or domain specific information.

What Exactly is RAG?

RAG is a framework that gives an LLM a search engine. Instead of relying solely on its internal memory (training data), the model looks up relevant information from an external source before generating an answer.

How RAG Works

A typical RAG pipeline has three main stages.

Retrieval: When a user asks a question, the system converts the query into a vector representation and searches a database of documents for the most relevant matches.
Augmentation: The retrieved passages are inserted into the prompt as context. This gives the language model fresh information related to the question.
Generation: The language model reads the question plus the retrieved context and produces a final answer that is informed by the external data.

Because the model sees real documents at runtime, it is less likely to invent facts and more likely to cite accurate details.

The Architecture

Rikam Palkar AI For Dummies Part 9 - RAG

1. Data Ingestion & Embedding Pipeline (The Preparation)

Before the AI can answer questions, it must first "digest" the knowledge.

Data Sources: This could be your private files, you want AI to use for augmentation. The system pulls information from various formats, such as PDFs, Wikis, Databases (DBs), and Website URLs.
Document Parsing & Chunking: Large files are broken down into smaller, manageable "chunks" of text so the system can pinpoint specific information later.
Embeddings Model: These text chunks are passed through an AI model (like text-embedding-ada-002) that converts words into numerical "vectors" representing their meaning.
Vector Database: These vectors are stored in a specialized database where they are indexed with metadata, making them searchable by concept rather than just keywords.

2. Retrieval & Generation Flow (The Live Answer)

When a user asks a question, the system follows this logical loop to find and deliver the answer:

User Query & Rewriting: The user's question (e.g., "How do I optimize our cloud spending?") this probably needs a database search.
Vector Similarity Search: The system looks into the Vector Database to find chunks that are mathematically similar to the user's query.
Contextual Compression & Reranking: The retrieved documents are sorted and refined to ensure only the most relevant "context" is kept.
Augmented Prompt: The system bundles the original user question with the retrieved chunks into one large prompt.
Large Language Model (LLM): The LLM (e.g., GPT-4o) reads the question and the provided context to generate a fact-based response.
Final Output: The user receives an AI-generated response that includes citations back to the source documents, ensuring accuracy and transparency.

Why this Architecture Works

By separating retrieval from generation, the model doesn't have to rely on its memory (which can be outdated). Instead, it "looks up" the answer in your specific data sources before speaking, effectively giving the AI an open-book exam.

From Text to Vectors:

To understand RAG, you have to understand a fundamental "translation" that happens behind the scenes. Think of saving a document into a vector database as translating a book into a "map of meanings" that a computer can navigate.

However, a map is only useful if it leads you to a real destination. In RAG, we use a Vector as the "GPS Coordinates" and the Chunk as the "Physical House" located at those coordinates.

Phase 1: Creating the Map (Data Ingestion)

Following the Data Ingestion & Embedding Pipeline, here is how a single sentence from an airline manual becomes a searchable data point:

The Raw Input (Parsing): We start with a plain sentence: "Passengers on international flights are allowed one free checked bag up to 23kg".
Chunking: The system breaks the document into smaller pieces.
- The Chunk: "international flights... one free checked bag... 23kg"
The Embedding Model (The Translator): This chunk is sent to an AI model (like text-embedding-ada-002) which converts the words into a Vector, a long list of numbers.
- Numerical Representation: [0.12, -0.54, 0.89, 0.21, ...]
- This vector represents the "location" of that sentence in a multi-dimensional space. Sentences about "suitcases" will be mathematically "close" to sentences about "luggage".
Storage (The Vector Database): Instead of filing it alphabetically, the database stores it by these coordinates.

Rikam Palkar AI For Dummies Part 9 - RAG Vector

Phase 2: The Two Faces of Data

When you save that information, the Vector Database actually stores two things together in a single entry:

The Vector (Mathematical): The list of numbers. The LLM cannot "read" this; the database only uses it to calculate similarity during a search.
The Chunk (Human-readable): The actual snippet of text (e.g., "Economy Saver allows 1 bag").

Phase 3: The "Switch" (How the Flow Works)

This is the most critical part of the architecture: the moment the system switches from math back to language.

Step A (The Search): You ask, "Can I bring a suitcase to London?" The system turns your question into a vector and asks the database: "Find me the 3 vectors mathematically closest to these coordinates".
Step B (The Retrieval): The database finds the closest vectors, but it doesn't give those numbers to the AI. Instead, it looks at the Metadata attached to those vectors and pulls out the original text chunks.
Step C (The Prompt): These text chunks are what get "stapled" to your question in the Augmented Prompt.

Why chunks, not vectors?

The Large Language Model (LLM) at the end of the chain is a text-processor, not a calculator. If you gave it a list of 1,000 numbers, it would be lost.

By sending the Chunk, you are effectively handing the AI the open textbook and saying: "I found this specific paragraph in our manual. Read it and use these exact facts to answer the user". This ensures the AI stays grounded in reality rather than guessing based on its coordinates.

Your query and document texts are converted into vectors so the system can compare them and find the most relevant matches.

Does RAG use your data for Training?

One of the biggest concerns for companies is data privacy. You might wonder: If I give the AI my private booking details or company manuals, is it "training" on them?

The short answer is no. In a RAG architecture, your documents are used as a reference, not as a teaching tool.

The "Open-Book" vs. "Memorization" Difference

To understand why your data stays private, think of the difference between how an AI learns and how it researches:

Standard Training: Imagine a brilliant grandpa who knows everything about history up until the day he retired. He’s a walking encyclopedia, but if you ask him about a TikTok trend, he’s going to make something up.
RAG: It’s that same grandpa, but you’ve handed him an iPad with 5G. Now, he uses his massive life experience to understand your question, but he "Googles" the current facts before he answers.

And we learned where Your Data Actually Lives

In the architecture diagram, you'll notice a clear separation between the Knowledge Base and the Large Language Model (LLM):

Your Secure Vault: Your documents are parsed, chunked, and stored in your own Vector Database. You own and control this.
The Temporary Handshake: The LLM only sees a specific Augmented Prompt containing a small snippet (chunk) of your data.
No Permanent Trace: Once the Final Answer is generated, the LLM does not store that snippet in its memory. It remains a "pre-trained" model that stays exactly as it was, without absorbing your private details into its core weights.

This Matters for Business

Because the system uses Hybrid Information Retrieval to pull specific booking details as chunks only when needed, your sensitive data never leaves your controlled environment to train a public model.

Why is RAG a Game Changer?

Why not just retrain the model every time new info comes out? Because retraining (or "Fine-Tuning") is expensive, slow, and technically difficult. and no company wants to handover its secret to AI, so RAG offers a better way:

Feature	Standard LLM	RAG-Enabled LLM
Up-to-Date Info	Limited by training cutoff.	Real-time (as fast as you update your files).
Accuracy	Prone to "hallucinations".	High; cites specific sources.
Privacy	Hard to keep private data out of training.	Keeps data in your secure database.
Cost	Millions to retrain.	Pennies to update the search index.

The cat is neither alive nor dead and honestly, that's the most exciting place to be. There are a lot more layers to uncover.

Previous: Part 8: Inside the Model Context Protocol

Next: Part 10: Fine Tuning