RAG Pipeline¶

The RAG (Retrieval-Augmented Generation) pipeline is how content enters the knowledge base and how student questions get answered. It has two sides: ingestion (getting content in) and retrieval (answering questions).

Content Ingestion¶

All content ingestion is fully manual, run by an operator (currently Hedley) from a local machine. There are no scheduled jobs, no admin interface, and no automation. The configuration file sources.yaml in the repository serves as both the source list and the record of what has been ingested.

Operator bottleneck

Adding, updating, or removing content from the knowledge base requires running scripts from Hedley's local machine. No one else on the team can currently do this without his involvement.

Knowledge base workspaces¶

Content is stored across 5 separate workspaces, each served by its own Azure Container App with an isolated Qdrant vector database:

Workspace	Content scope	Container App
`education`	Programs, admissions, dual credit, universities	`rag-api-education`
`financial_aid`	FAFSA, WASFA, scholarships, grants	`rag-api-financial-aid`
`career_resources`	Career Bridge, O*NET, labor market data, job training	`rag-api-career-resources`
`supporting_services`	WA 2-1-1, housing, healthcare, basic needs	`rag-api-supporting-services`
`yakima`	All Yakima Valley-specific content (cross-category)	`rag-api-yakima`

The workspace is chosen by the operator when adding a source. It is a human decision, not automated classification. The AI relevance filter only decides keep-or-discard for quality; it does not route content between workspaces.

Two ingestion pipelines¶

There are two separate pipelines. Scraping and uploading are always separate manual steps.

	Pipeline A: Generic web scraper	Pipeline B: Yakima custom scrapers
Used for	Statewide content (FAFSA, Career Bridge, O*NET, WA labor data, Peninsula College, etc.)	Yakima Valley institutions (YVCC, Heritage, PerryTech, Rod's House, AJAC, etc.)
How scraping works	Firecrawl SaaS crawls websites and converts pages to markdown. A local vLLM relevance filter scores each page and discards low-quality content.	17 custom Python scripts, one per institution, each with site-specific HTML selectors.
Sources configured	19 sources across 4 categories in `sources.yaml`	~17 Yakima Valley institutions
How upload works	A separate upload script reads scraped files from disk and POSTs to the correct Container App.	A separate upload script sends all files to the Yakima Container App.
Scalable?	Yes -- adding a source means editing `sources.yaml` and running the scraper.	No -- each new institution requires a custom Python scraper.

Pipeline A: generic web scraper¶

Step 1: Scrape¶

The operator runs the scraper from their local machine. Firecrawl (a hosted SaaS at api.firecrawl.dev) crawls the target site. Each page is evaluated by a local vLLM instance for relevance. Pages scoring >= 0.5 are saved as markdown with YAML frontmatter.

sequenceDiagram
    participant Op as Operator
    participant Script as ingest_web.py
    participant YAML as sources.yaml
    participant FC as Firecrawl SaaS
    participant vLLM as Local vLLM
    participant Disk as Local filesystem

    Op->>Script: python -m scripts.ingest.web.ingest_web<br/>--source <slug>

    Script->>YAML: Load source config<br/>{slug, url, primary_category, local_path}

    loop Per page (max 30 pages, depth 3)
        Script->>FC: POST /v1/scrape<br/>{url, formats: [markdown, links]}
        FC-->>Script: {markdown, links[], metadata}

        Script->>vLLM: Evaluate relevance
        vLLM-->>Script: {relevance, clarity,<br/>completeness, currency}

        alt relevance >= 0.5
            Script->>Disk: Save markdown with<br/>YAML frontmatter
        end
    end

    Script->>YAML: Update ingestion metadata<br/>{last_scraped, pages_scraped, quality_summary}

The scraped markdown files look like this:

---
category: career_resources
institution: AJAC Training
scraped_at: '2026-02-06T16:38:01'
section: Apprenticeship Programs
source_url: https://www.ajactraining.org/programs/youth
summary: AJAC Training offers a Youth Apprenticeship...
---

# Youth Apprenticeship

**Organization:** AJAC Training
**Region:** Washington State (Kent, Seattle, Tacoma, Everett, Lacey, Yakima)

For High School Students...

Step 2: Upload¶

The operator runs a separate upload script. It walks the scraped markdown files, maps each to the correct Container App based on workspace, and POSTs via the upload API.

sequenceDiagram
    participant Op as Operator
    participant Upload as upload_directory.py
    participant Disk as Local filesystem
    participant CA as Container App<br/>(per workspace)
    participant LR as LightRAG
    participant Embed as OpenAI Embeddings<br/>(text-embedding-3-large)
    participant Qdrant as Qdrant (gRPC)
    participant Neo4j as Neo4j
    participant Redis as Redis

    Op->>Upload: python upload_directory.py<br/>--dir content/ --workspace education

    Upload->>Disk: Walk directory for .md files

    loop Per file
        Upload->>CA: POST /api/documents/upload<br/>multipart: file, source_url, scraped_at<br/>Header: X-API-Key

        CA->>LR: lightrag.ainsert(content)

        Note over LR: Chunk: 1200 tokens,<br/>100 token overlap

        LR->>LR: Extract entities &<br/>relationships from chunks

        par Embed all content
            LR->>Embed: Embed chunks, entities,<br/>relationships (max 16 concurrent)
            Embed-->>LR: Vectors (3072 dims)
        end

        par Store to 3 backends
            LR->>Qdrant: Upsert to chunks_vdb,<br/>entities_vdb, relationships_vdb
            LR->>Neo4j: Store entity-relation graph
            LR->>Redis: Cache docs, entities,<br/>relations, chunks
        end

        CA-->>Upload: {status: "success"}
    end

Pipeline B: Yakima custom scrapers¶

These are 17 site-specific Python scripts that scrape individual Yakima Valley institutions. Each uses custom HTML selectors and a two-phase approach (discovery, then scrape). Output is markdown with YAML frontmatter, uploaded via a separate ingest_to_rag.py script that POSTs to the Yakima Container App.

This pipeline is not scalable -- adding a new institution requires writing a new custom scraper.

Source configuration¶

All generic web sources are defined in sources.yaml (located at rag-agent-service/scripts/ingest/sources.yaml). Each source entry contains:

Field	Purpose
`name`, `slug`	Human label and CLI identifier
`url`	Web URL or local file path (PDFs, CSVs)
`primary_category`	Which workspace it goes to (education, financial_aid, etc.)
`secondary_categories`	Cross-category tags (currently unused by upload scripts)
`local_path`	Where scraped files land on disk
`confidence`	HIGH/MEDIUM quality signal
`notes`	Operator notes (bot protection issues, re-ingestion schedule)
`ingestion.last_scraped`	When the source was last scraped
`ingestion.pages_scraped`	How many pages were saved
`ingestion.files`	List of output markdown file paths
`ingestion.quality_summary`	Average relevance, clarity, completeness, currency scores

As of January 2026, there are 19 configured sources: 7 education, 9 career resources, 2 financial aid, 1 supporting services.

The file also contains an update_schedule section noting which sources should be refreshed quarterly, monthly, or annually -- but nothing enforces this schedule. It relies on the operator remembering to re-run the scripts.

Storage architecture¶

When a document is uploaded, LightRAG processes it through several stages:

Chunking: Content is split into 1,200-token chunks with 100-token overlap
Entity extraction: LLM extracts named entities and relationships from each chunk
Embedding: All chunks, entities, and relationships are embedded using OpenAI text-embedding-3-large (3,072 dimensions, max 16 concurrent requests)
Storage: Results are written to three backends in parallel:

Store	What it holds
Qdrant (`chunks_vdb`)	Text chunk embeddings + source_url, scraped_at metadata
Qdrant (`entities_vdb`)	Entity embeddings + entity_name, entity_type
Qdrant (`relationships_vdb`)	Relationship embeddings + source entity, target entity, relation
Neo4j	Entity-relationship knowledge graph for traversal
Redis	Cached full documents, entities, relations, text chunks

Custom metadata (source_url, scraped_at) is injected into Qdrant payloads via a patched upsert function using thread-safe context variables.

Known limitations¶

No content inventory: There is no dashboard showing what is currently indexed in each workspace. The only record is sources.yaml and what is in Qdrant.
No deduplication: Re-running upload inserts duplicate content. There is no "this URL is already indexed" check.
No scheduled re-ingestion: Update schedules in sources.yaml are aspirational notes, not enforced.
Secondary categories are unused: Some sources are tagged with multiple categories, but uploads only go to the primary category workspace.
Yakima scrapers don't scale: Adding a new institution requires writing custom code. The generic Firecrawl pipeline is the scalable path.

Query Retrieval¶

When a student asks a question in the app, the following sequence runs:

sequenceDiagram
    participant App as appapi (Express.js)
    participant API as RAG Agent API<br/>(FastAPI)
    participant Mod as Content Moderation
    participant Orch as Orchestrator
    participant Router as Gemini (Router)
    participant Agent as Domain Agent
    participant MCP as MCP Session
    participant RAG as RAG Service
    participant LR as LightRAG
    participant Qdrant as Qdrant
    participant Neo4j as Neo4j
    participant Redis as Redis
    participant Gen as Gemini (Generator)

    App->>API: POST /api/query<br/>{query, mode, conversation_history}

    par Parallel execution
        API->>Mod: check_content(query)
        Note over Mod: BLOCK/ESCALATE<br/>cancels search early

        API->>Orch: query(query, mode, ...)
    end

    Note over Orch: Step 1: Route to domain agent

    Orch->>Router: "Which agent handles this?"<br/>+ agent descriptions + query<br/>temperature=0, max_tokens=50
    Router-->>Orch: "education"

    Note over Orch: Step 2: Query selected agent

    Orch->>Agent: query(user_query, mode, history)

    Agent->>MCP: call_tool("rag_query",<br/>{query, mode: "hybrid",<br/>only_need_context: true})
    MCP->>RAG: rag_query tool call
    RAG->>LR: aquery(query, mode="hybrid")

    par Hybrid search
        LR->>Qdrant: Vector search: chunks, entities, relationships
        LR->>Neo4j: Graph traversal (entity neighbors)
        LR->>Redis: KV lookup (cached content)
    end

    LR-->>RAG: Combined context
    RAG-->>Agent: Context text

    Note over Agent: Post-process context

    Agent->>Agent: TOON compression
    Agent->>Agent: Strip dead URLs
    Agent->>Agent: Enforce 10,000 char budget

    Note over Agent: Generate response

    Agent->>Gen: generate_content(<br/>system_instruction,<br/>conversation_history,<br/>context + query + template)
    Gen-->>Agent: Response text

    Agent-->>API: Response + timing + moderation

    API-->>App: {response, timing,<br/>moderation, prompt_context}

Routing¶

An AI router (Gemini, temperature=0) reads the student's question and decides which domain agent should handle it. The router chooses from the 4 routable agents:

Agent	Routes when the question is about
`education`	Programs, admissions, academic standards, curricula
`financial_aid`	FAFSA, WASFA, scholarships, grants, tuition
`career_resources`	Jobs, careers, training programs, labor market
`supporting_services`	Housing, healthcare, food assistance, basic needs

Non-routable agents

The yakima and port_angeles agents are not included in automatic routing. They can only be queried via the additional_rags parameter, which the app does not currently use.

Retrieval¶

The selected agent searches its Qdrant workspace using LightRAG's hybrid mode, which combines:

Vector similarity search across chunks, entities, and relationships collections
Graph traversal through Neo4j for related entities and connections
KV cache lookup from Redis for previously cached content

Context processing¶

Before sending context to the LLM for answer generation, the agent applies several post-processing steps:

TOON compression (if enabled) -- reduces token count while preserving meaning
Dead URL stripping -- async validation removes broken links (2s timeout, max 10 concurrent)
Context budget -- truncates to 10,000 characters maximum
Flash-lite extraction (if enabled) -- uses a lightweight Gemini model to distill only the relevant portions

Response generation¶

The final answer is generated by Gemini using:

A system instruction from agents.yml specific to the domain agent
The student's conversation history for multi-turn context
A prompt template that includes the retrieved context, the question, and domain-specific focus directives

Parallel multi-agent queries¶

When parallel=true or additional_rags is specified, the orchestrator queries multiple agents simultaneously and synthesizes their responses into a single answer using Gemini.

Configuration reference¶

Setting	Value
Embedding model	OpenAI `text-embedding-3-large`
Embedding dimensions	3,072
Chunk size	1,200 tokens
Chunk overlap	100 tokens
Max concurrent embeddings	16
Max context characters	10,000
Default query mode	`hybrid` (local + global)
Generation LLM	Gemini 2.0 Flash (configurable)
Routing LLM	Gemini (temperature=0, max 50 tokens)
Qdrant transport	gRPC (60s timeout)
Scraping service	Firecrawl SaaS (api.firecrawl.dev)
Relevance filter	Local vLLM, threshold >= 0.5