# Examcooker Agentic RAG Architecture

> Blueprint v1.0 — a course-scoped, citation-first intelligence layer over university notes and past-year question papers.

## Executive summary

Examcooker stores university notes and past-year question papers in S3. Each document is associated with stable academic identifiers such as `university_id`, `course_id`, `professor_id`, document type, term and year. The proposed system turns that corpus into a conversational course workspace that can explain topics, analyze a selected paper, compare years and identify strongly represented question families.

The central design rule is simple: retrieval is a controlled evidence system, not a prompt with vector search attached. Scope is enforced before search, evidence is ranked before generation and every meaningful factual claim is traceable to a document, page and source version.

### Architecture principles

1. **Hard course boundary.** Every retrieval call carries `university_id`, `course_id`, the authenticated user's ACL and a published `corpus_version`.
2. **Evidence before generation.** Hybrid search, graph expansion, exam analytics and reranking produce a bounded evidence pack before the tutor model writes an answer.
3. **Traceability by design.** Citations identify the document, page, section or question and immutable source version.
4. **Deterministic control plane.** Scope, tool permissions, budgets, retries and abstention rules live in code rather than model prompts.
5. **Measured quality.** Retrieval, grounding, pedagogy, exam analytics, latency, cost and leakage are evaluated separately.

## 1. Reference system architecture

Use one orchestrator and a small set of specialist tools. The evidence pack is the contract between retrieval and generation; the tutor must not query stores directly.

```mermaid
graph TB
  Student["Student · Examcooker course room"] --> Edge["Web app + streaming API"]
  Edge --> Guard["Course boundary guard"]
  Guard --> Orchestrator["RAG orchestrator"]

  Orchestrator --> Router["Intent + task router"]
  Orchestrator --> Memory["Conversation state"]
  Router --> Planner["Evidence planner"]

  Planner --> Hybrid["Hybrid retriever"]
  Planner --> Graph["Graph traversal"]
  Planner --> Analytics["Exam analytics tool"]

  Hybrid --> Vector[("Vector index")]
  Hybrid --> Lexical[("BM25 index")]
  Graph --> KG[("Knowledge graph")]
  Analytics --> Meta[("Document metadata")]

  Vector --> Pack["Ranked evidence pack"]
  Lexical --> Pack
  KG --> Pack
  Meta --> Pack
  Pack --> Tutor["Tutor + synthesis agent"]
  Tutor --> Verifier["Citation + claim verifier"]
  Verifier --> Edge
```

The runtime should begin as one deployable service with a typed state object. Split components into independent services only when scale, isolation or team ownership requires it.

## 2. Agent topology

The following names describe logical responsibilities, not necessarily separate model calls.

| Role | Responsibility |
|---|---|
| Boundary guard | Bind the turn to university, course, role and corpus version before retrieval can run. |
| Intent router | Choose explain, summarize, paper analysis, important-question mining, comparison or study-plan workflows. |
| Evidence planner | Decompose the request, select tools and assign retrieval budgets without drafting an answer. |
| Exam analyst | Normalize question families and calculate recurrence, marks, recency and syllabus coverage. |
| Tutor synthesizer | Explain at the requested depth using only the ranked evidence pack. |
| Claim verifier | Check factual statements against cited spans, retry weak retrieval once or abstain. |

```mermaid
graph TB
  Q["User question"] --> A0["01 · Boundary guard"]
  A0 -->|"course_id + ACL valid"| A1["02 · Intent router"]
  A0 -->|"invalid or cross-course"| Stop["Reject / request scope"]
  A1 -->|"Explain topic"| R1["Notes retrieval"]
  A1 -->|"Analyze paper"| R2["Paper parser + question map"]
  A1 -->|"Important questions"| R3["Cross-year exam analytics"]
  A1 -->|"Compare / plan"| R4["Multi-step planner"]
  R1 --> E["Evidence pack"]
  R2 --> E
  R3 --> E
  R4 --> E
  E --> A2["03 · Tutor synthesizer"]
  A2 --> A3["04 · Claim verifier"]
  A3 -->|"grounded"| Answer["Answer + page citations + confidence"]
  A3 -->|"unsupported"| Retry["Retrieve again or abstain"]
  Retry --> E
```

Only the verifier may trigger a second retrieval pass. Maximum retries, model calls, tool calls, retrieved tokens and total latency must be configured in orchestration code.

## 3. Retrieval pipeline

Dense vector similarity is not sufficient for academic material because exact terminology, question numbering, marks, years and paper structure matter. Examcooker should use filtered hybrid retrieval.

1. Bind authenticated scope.
2. Classify the intent and rewrite the query for the selected workflow.
3. Search exact fields and full text with BM25.
4. Search dense embeddings with identical metadata filters.
5. Expand relevant topics and question families through one or two graph hops.
6. Fuse candidate lists with reciprocal rank fusion.
7. Rerank against the actual user request with a cross-encoder or ranking model.
8. Diversify by topic, source type and year.
9. Package citable spans with complete lineage.

```mermaid
sequenceDiagram
  autonumber
  actor S as Student
  participant O as Orchestrator
  participant P as Query planner
  participant H as Hybrid search
  participant G as Knowledge graph
  participant X as Exam analytics
  participant L as Tutor LLM
  participant V as Verifier

  S->>O: “Important Module 3 questions from past papers”
  O->>O: Bind university_id + course_id + ACL
  O->>P: Classify and decompose request
  par Evidence retrieval
    P->>H: BM25 + vector search with hard filters
    H-->>P: Notes and paper chunks
  and Topic expansion
    P->>G: Module 3 → topics → related questions
    G-->>P: 1–2 hop neighborhood
  and Frequency analysis
    P->>X: Group questions by topic, year, marks
    X-->>P: Recurrence and weighted importance
  end
  P->>L: Ranked evidence pack + answer contract
  L->>V: Draft with atomic citations
  V-->>O: Supported claims + calibrated confidence
  O-->>S: Answer, rationale, sources, uncertainty
```

### Important-question ranking

A launch hypothesis for a topic or normalized question-family score is:

```text
0.30 frequency
+ 0.20 recency
+ 0.20 marks weight
+ 0.15 syllabus alignment
+ 0.10 professor/course similarity
+ 0.05 extraction confidence
```

These weights must be calibrated against faculty-labelled historical sets. The assistant must say that a topic is strongly represented in available past papers, show the supporting years and marks and explicitly state that future exam content cannot be guaranteed.

## 4. Academic knowledge graph

The graph provides relationships that vectors do not represent reliably: module ownership, prerequisites, paper chronology, question variants, professor associations and marks.

```mermaid
graph TB
  U["University"] -->|"OFFERS"| C["Course"]
  C -->|"HAS_MODULE"| M["Module"]
  M -->|"COVERS"| T["Topic"]
  T -->|"PREREQUISITE_OF"| T2["Topic"]
  P["Professor"] -->|"TEACHES"| C
  D["Document"] -->|"BELONGS_TO"| C
  D -->|"AUTHORED_BY"| P
  D -->|"HAS_SECTION"| S["Section / chunk"]
  E["Exam paper"] -->|"CONTAINS"| Q["Question"]
  Q -->|"ASSESSES"| T
  Q -->|"WORTH_MARKS"| K["Marks"]
  Q -->|"VARIANT_OF"| Q2["Question family"]
  E -->|"FOR_COURSE"| C
  E -->|"SET_BY"| P
  E -->|"HELD_IN"| Y["Term + year"]
```

### Canonical identifiers

- `Course`: `course_id`, `university_id`, course code, title and `syllabus_version`.
- `Document`: `document_id`, `course_id`, `professor_id`, type, term, year and version.
- `Question`: `question_id`, `exam_id`, marks, page and `normalized_family_id`.
- Extracted relationships: confidence, `source_document_id`, extraction method and review status.

Use Amazon Neptune when an AWS-managed graph is preferred, or Neo4j when its tooling and query experience better fit the team. The graph is an augmentation layer; source text and authorization remain in the canonical stores.

## 5. Data plane

Use each store for one clear responsibility.

| Store | Role | Content |
|---|---|---|
| S3 | Canonical content | Versioned PDFs, slides, extracted JSON, page images and immutable lineage. |
| Aurora PostgreSQL | Control plane | IDs, permissions, document status, ingestion jobs, sessions and feedback. |
| OpenSearch | Hybrid retrieval | BM25 fields and dense vectors with hard metadata filtering. |
| Neptune or Neo4j | Knowledge graph | Course, module, topic, document, exam and question relationships. |
| Redis | Latency layer | Session state, retrieval cache, rate limits and short-lived evidence packs. |

S3 is the source of truth. Every derived representation must be reproducible from an immutable S3 object, content hash, parser version and ingestion version.

## 6. Versioned ingestion pipeline

```mermaid
graph TB
  S3["S3 · raw PDFs, slides, notes"] --> Event["Object-created event"]
  Event --> Workflow["Ingestion workflow"]
  Workflow --> Validate["Checksum, MIME, malware, ACL validation"]
  Validate --> Parse["OCR + layout-aware parsing"]
  Parse --> Classify["Document / year / professor classifier"]
  Classify --> Chunk["Question-aware + semantic chunking"]
  Chunk --> Enrich["Topics, modules, pages, marks, lineage"]
  Enrich --> Embed["Embedding model"]
  Enrich --> Extract["Entity + relationship extraction"]
  Embed --> Search[("Vector + BM25 indexes")]
  Extract --> Graph[("Knowledge graph")]
  Enrich --> Catalog[("Metadata catalog")]
  Search --> QA["Automated quality gates"]
  Graph --> QA
  Catalog --> QA
  QA -->|"pass"| Ready["Publish corpus_version"]
  QA -->|"review"| Queue["Human review queue"]
```

Publish derived indexes atomically by `corpus_version` so a live conversation never observes a half-ingested course.

### Chunking strategy

- **Notes:** split by heading and semantic boundary while preserving page, heading path, slide number and neighboring context.
- **Question papers:** keep one question or sub-question per unit with marks, instructions, diagrams and answer-option relationships intact.
- **Context expansion:** attach parent summaries and small neighbor windows at retrieval time instead of relying on arbitrary fixed overlap.
- **Embedding lineage:** record model, dimensions, normalization, chunker version and content hash on every vector.

## 7. API and evidence contracts

Scope must be impossible to omit from a retrieval request.

```json
{
  "university_id": "uni_01",
  "course_id": "cs_301",
  "professor_id": "prof_17",
  "corpus_version": "2026.07",
  "intent": "important_questions",
  "filters": { "years": [2022, 2023, 2024, 2025] },
  "budgets": { "top_k": 30, "rerank_k": 10 }
}
```

Every evidence item should include enough lineage to render a meaningful citation.

```json
{
  "document_id": "exam_2025_endsem",
  "source_version": "sha256:…",
  "page": 3,
  "question_id": "q_2025_07b",
  "topic_ids": ["topic_deadlocks"],
  "quote": "Explain the necessary conditions…",
  "scores": { "hybrid": 0.87, "reranker": 0.93 }
}
```

## 8. Trust and security architecture

Assume both user prompts and uploaded documents are untrusted.

1. **Authorization before retrieval:** derive allowed courses from the signed session; never trust scope supplied only by text or model output.
2. **Document prompt isolation:** label retrieved text as evidence, strip active content and ignore instructions found inside documents.
3. **Private source delivery:** use short-lived signed URLs for authorized source previews and never expose raw S3 keys publicly.
4. **Auditability:** log scope, plan, source IDs, model version, citations and policy outcomes without retaining unnecessary personal data.
5. **Abstention:** if evidence is weak, contradictory or outside the authorized scope, explain the limitation and offer the closest supported material.
6. **Abuse and cost control:** enforce per-user quotas, retrieval caps, model-tier routing, stream limits and anomaly detection.

## 9. Evaluation and observability

Build gold evaluation sets course by course with faculty and student reviewers.

| Layer | Launch metric | Test method |
|---|---|---|
| Retrieval | Recall@20, nDCG@10, topic coverage | Gold question-to-source sets for each pilot course. |
| Grounding | Citation precision at least 95% | Atomic claim-to-span verification. |
| Answer | Correctness, completeness and pedagogy | Faculty rubric and blinded student review. |
| Exam analytics | Question-family F1 and year/marks accuracy | Human-normalized past-paper benchmark. |
| Operations | P95 latency, cost per answer and cache hit rate | Traces segmented by workflow and course. |
| Safety | Zero cross-course leakage | Adversarial ACL and prompt-injection suite. |

Capture traces for the plan, every tool call, candidate counts, filters, ranks, model versions, token usage, citations, verification result and user feedback. Keep sensitive source text out of telemetry by default.

## 10. Recommended deployment

Keep ingestion and data close to the existing S3 corpus while placing models behind a provider-neutral gateway.

```mermaid
graph TB
  Web["Next.js / React web"] --> CDN["CDN + WAF"]
  CDN --> API["API Gateway / RAG API"]
  API --> Auth["University SSO / JWT"]
  API --> Runtime["Agent runtime · containers or serverless"]
  Runtime --> Models["Model gateway"]
  Runtime --> Cache[("Redis response + retrieval cache")]
  Runtime --> OpenSearch[("OpenSearch hybrid index")]
  Runtime --> Neptune[("Neptune / Neo4j graph")]
  Runtime --> Postgres[("Aurora PostgreSQL metadata")]
  Runtime --> S3[("S3 source of truth")]
  S3 --> Pipeline["Step Functions ingestion"]
  Pipeline --> OpenSearch
  Pipeline --> Neptune
  Pipeline --> Postgres
  Runtime --> Observe["Traces, evals, cost, audit logs"]
```

### Launch recommendation

- **Search:** OpenSearch for filtered BM25 and vector search.
- **Metadata:** Aurora PostgreSQL.
- **Graph:** Neptune or Neo4j, selected after a short query and operations spike.
- **Runtime:** ECS/Fargate for streaming and predictable dependencies; Lambda for bursty ingestion steps.
- **Workflow:** Step Functions, queues and dead-letter handling for durable ingestion.
- **Model gateway:** Bedrock or a provider-neutral proxy that supports task-based routing and model version tracking.

## 11. Delivery roadmap

### Phase 1 — Corpus foundation, weeks 1–3

Define canonical IDs, S3 conventions, parser benchmarks, metadata catalog and ACLs for three pilot courses. Exit when page and metadata accuracy reach at least 95%.

### Phase 2 — Citation-first RAG, weeks 4–6

Implement hybrid retrieval, reranking, evidence packs, tutor responses, feedback and offline evaluation. Exit when citation precision reaches at least 95% on the pilot benchmark.

### Phase 3 — Exam intelligence, weeks 7–10

Add question extraction, family clustering, the academic knowledge graph, recurrence scoring and explanation UI. Exit when faculty reviewers approve the analytics methodology and sampled outputs.

### Phase 4 — Production hardening, weeks 11–14

Complete red-team tests, observability, caching, cost routing, load testing and staged course rollout. Exit with SLOs, security review and an operational runbook.

## North-star answer contract

> Here are the topics most strongly represented in the last four papers, why they rank highly, and the exact pages and questions that support the recommendation.

Every production answer should be scoped, grounded, explainable and measurable.