08Memory, Knowledge & RAG

Three ways to give AI your knowledge

Memory in AI systems

• Short-term memory = the context window: the current conversation and task.
• Long-term memory = anything saved outside the model and re-inserted when relevant: user preferences, notes, summaries of past sessions, documents in a knowledge base.
• Procedural memory = saved “how-to” instructions (skills) the agent can reload — e.g. Hermes Agent writes skill files from experience (Lesson 12).

Embeddings: meaning as numbers

An embedding model turns text into a list of numbers (a vector) that captures its meaning. Texts with similar meanings get similar vectors — even with no words in common.

Comparing vectors (usually with cosine similarity) lets you find the most relevant passages for any question. That's semantic search, and it's the “retrieval” in RAG. Vector databases (pgvector, Chroma, Qdrant, and others) store and search millions of these efficiently.

The RAG pipeline

Build it: a working mini RAG with Ollama

This script answers questions about a small business knowledge base using only local models. No cloud, no vector database — just the idea, clearly.

1. Pull an embedding model and a chat model

   ollama pull nomic-embed-text
   ollama pull qwen3:8b
   pip install ollama

2. Save the script

   import ollama, math
   
   # 1) Your knowledge base — in real life, load and chunk your files
   docs = [
       "Barefoot Café is open Monday to Saturday 7am-9pm and Sunday 8am-6pm.",
       "We deliver within 5 km of Iloilo Business Park. Delivery fee is 49 pesos.",
       "Refunds: if an order is wrong or unsatisfactory, we replace it or refund within 24 hours.",
       "Our co-working corner upstairs has fiber Wi-Fi and power outlets at every seat.",
       "Sea-Salt Cold Brew costs 160 pesos and uses sea salt from Guimaras.",
   ]
   
   EMBED = "nomic-embed-text"
   CHAT = "qwen3:8b"
   
   def embed(texts):
       return ollama.embed(model=EMBED, input=texts)["embeddings"]
   
   def cosine(a, b):
       dot = sum(x * y for x, y in zip(a, b))
       return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
   
   # 2) Index once
   doc_vectors = embed(docs)
   
   def ask(question, k=2):
       # 3) Retrieve the k most similar chunks
       q_vec = embed([question])[0]
       ranked = sorted(zip(docs, doc_vectors), key=lambda d: cosine(q_vec, d[1]), reverse=True)
       context = "\n".join(f"[{i+1}] {text}" for i, (text, _) in enumerate(ranked[:k]))
   
       # 4) Generate a grounded answer
       prompt = f"""Answer the customer using ONLY the sources below.
   Cite sources like [1]. If the answer isn't in the sources, say you don't know.
   
   <sources>
   {context}
   </sources>
   
   Question: {question}"""
       reply = ollama.chat(model=CHAT, messages=[{"role": "user", "content": prompt}])
       return reply["message"]["content"]
   
   while True:
       q = input("\nAsk the café (or 'quit'): ")
       if q.lower() == "quit":
           break
       print(ask(q))

3. Run it and test

   python mini_rag.py

   Try: “Are you open Sunday evening?”, “How much is delivery?”, and something not in the docs like “Do you sell cakes?” — a good RAG system should say it doesn't know.

Making RAG good in production