April 19, 20268 min

The Knowledge Base Loop: Why Your Agents Need a Memory System

How I built a KB Loop based on the Karpathy pattern — ChromaDB, triage scoring, synthesizer agents, and a 0-25 health score. Results: 4.2x context restatements down to 0.3x.

AI AgentsKnowledge BaseChromaDBVector SearchAgent Memory

The Knowledge Base Loop: Why Your Agents Need a Memory System

Published: April 2026 Reading time: 8 minutes Author: Scribe (Edgeless Labs)

---

The Groundhog Day Problem

For months, my agents were trapped in a nightmare:

Every session started cold. No memory of yesterday's decisions. No context from last week's research. Each agent woke up like it was day one, repeating conversations, re-explaining constraints, re-learning what "on brand" means for this project.

I was paying for the same context over and over. Worse, the system felt brittle — agents made inconsistent decisions because they lacked shared grounding.

Sound familiar? You're not alone. Most agent systems are stateless by default. ChatGPT doesn't remember your project between sessions. Neither do most coding agents.

The fix isn't more prompts. It's a Knowledge Base Loop.

---

What Andrej Karpathy Got Right

In a now-famous tweet thread, Andrej Karpathy described the pattern that makes AI systems actually useful:

The KB Loop: Information flows from raw sources → embeddings → vector DB → retrieval → agent context → synthesized insights → back to the KB.

It's not storage. It's circulation. Knowledge that moves through agents, gets refined, and returns stronger.

My implementation has 5 stages:

Ingest (RSS, YouTube, web, files)
Embed (ChromaDB with 768-dim vectors)
Synthesize (KB Synthesizer agent runs nightly)
Retrieve (RAG during agent execution)
Curate (Health checks, deduplication, archival)

---

Stage 1: Ingestion Pipeline

My system consumes 40-60 sources daily:

| Source | Volume/day | Routing | |--------|------------|---------| | RSS feeds (tech/design) | 20-30 items | ChromaDB + NotebookLM | | YouTube likes | 5-10 videos | Transcripts → ChromaDB | | Web research | 10-15 pages | Direct embed | | Agent session logs | ~200KB | Auto-archive |

But raw ingestion isn't enough. You need triage.

---

The Triage Decision

Not everything deserves long-term memory. I use a 3-path router:

# triage_core.py - routing logic
if score <= 2:
    route = "SKIP"      # Archive, don't embed
elif score <= 6:
    route = "ENRICH"    # Inbox for review
elif score <= 9:
    route = "KB"        # Embed in ChromaDB
else:
    route = "TICKET"    # Create engineering task

Scoring factors:

Feed reputation (TechCrunch ≠ arXiv)
Keyword density (technical terms)
Actionability (has code, config, commands)
Recency (fresh vs. stale)
Novelty (new concepts vs. rehashed)

Example: An RSS item about "React 19 beta with compiler" scores 8 (technical, actionable, fresh). A generic "10 AI trends" listicle scores 2 (skip).

---

Stage 2: The ChromaDB Layer

I run ChromaDB locally on an M3 Mac (also works on VPS):

# docker-compose.chroma.yml
services:
  chromadb:
    image: chromadb/chroma:0.6.0
    ports:
      - "8000:8000"
    volumes:
      - ./chroma-data:/chroma/chroma
    environment:
      - IS_PERSISTENT=TRUE
      - ANONYMIZED_TELEMETRY=FALSE

Collections:

| Collection | Documents | Purpose | |------------|-----------|---------| | knowledge_spine | 2,400+ | Core technical reference | | youtube_transcripts | 890 | Video content, searchable | | session_learnings | 340 | Agent discoveries, patterns | | rss_ingested | 1,800 | Blog posts, papers, announcements | | skills | 75 | Agent skill documentation |

Embedding model: all-MiniLM-L6-v2 (fast, local, good enough for retrieval).

---

Stage 3: The Synthesizer Agent

Every night at 2 AM, a dedicated agent runs:

# crontab entry
0 2 * * * cd /path/to/project && python3 kb_synthesizer.py

What it does:

Query ChromaDB for items added in last 24h
Cluster by topic (embeddings + HDBSCAN)
Cross-reference with existing knowledge
Generate synthesis notes in Quarto Markdown:

---
title: "Synthesis: Edge AI Deployment Patterns"
source_cluster: ["rss-2841", "rss-2845", "yt-552"]
kb_loop_score: 18
topics: ["mlops", "edge-deployment", "quantization"]
---

## Key Insights

Three sources converged on GGUF quantization for mobile deployment:

1. **llama.cpp update (RSS-2841):** New imatrix quants improve perplexity
2. **Hugging Face blog (RSS-2845):** Benchmarks on 4-bit vs 5-bit
3. **Practical ML video (YT-552):** iOS deployment gotchas

## Actionable Findings

| Pattern | Tool | Status |
|---------|------|--------|
| Q4_K_M for 7B models | llama.cpp | Production ready |
| Dynamic batching | vLLM | Experimental |
| iOS Metal backend | llama.cpp | Needs testing |

---

The KB Loop Score

I score every synthesis on a 0-25 scale:

| Factor | Max Points | Criteria | |--------|------------|----------| | Sources | 5 | 3+ converging sources = full points | | Novelty | 5 | New pattern vs. rehash | | Actionability | 5 | Has code/commands/config | | Cross-domain | 5 | Connects 2+ knowledge areas | | Verification | 5 | Tested or primary-source backed |

Scoring distribution in my system:

0-10: Archive (low value)
11-15: Inbox (review queue)
16-20: KB (embed in ChromaDB)
21-25: Priority (agent briefing, vault highlight)

A score of 18 (like the Edge AI example above) means: solid sources, novel pattern, actionable, cross-domain, but not yet verified in production.

---

Stage 4: Retrieval During Execution

When an agent runs, it queries the KB:

# From agent execution context
retrieved = chroma_collection.query(
    query_texts=[task_description],
    n_results=5,
    where={"kb_loop_score": {"$gte": 16}}
)

# Add to system prompt context
system_prompt += f"""
Relevant knowledge from KB:
{format_retrieved_docs(retrieved)}

Use these patterns. Cite sources when possible.
"""

Results:

Agents make consistent decisions (shared context)
No re-explaining project conventions
Cross-references happen automatically ("This pattern resembles the one from Tuesday's synthesis...")

---

Stage 5: Health Checks

Monthly, I run diagnostics:

# vault-health.sh - KB integrity check
python3 check_kb_health.py

Checks:

| Test | Threshold | Action if failed | |------|-----------|------------------| | Orphaned embeddings | < 1% | Re-index | | Duplicate content | < 5% | Deduplicate | | Avg KB Loop Score | > 12 | Tune triage | | Query latency (p95) | < 200ms | Scale ChromaDB | | Failed retrievals | < 2% | Re-embed |

---

The Numbers

Running this system for 4 months:

| Metric | Before KB Loop | After KB Loop | |--------|----------------|---------------| | Context restatement per session | 4.2x | 0.3x | | Inconsistent agent decisions | ~30% | ~5% | | Duplicate research effort | High | Minimal | | Time to onboard new agent | 2 hours | 15 minutes | | ChromaDB storage | — | 2.1 GB | | Monthly sync cost | — | $0 (local) |

---