Examcooker RAG Architecture

A practical design for answering course questions from notes and past papers, with links back to the original pages.

Executive summary

Examcooker already keeps course notes and past-year papers in S3. Each file has a course_id, professor_id, document type, term and year. This design adds chat inside a course. A student can ask for an explanation, inspect a particular paper, compare papers from different years or find question types that appear often.

Search is handled by application code, not left to the model. The server checks course access first, gathers and ranks useful passages, then gives the model a small evidence pack. Answers link back to the document, page and version that support them.

Rules for the first version

1. Never search outside the active course. Each retrieval call includes course_id, the signed-in user's access rules and a published corpus_version.
2. Search before writing. Keyword search, vector search, graph lookups and paper analysis all finish before the answer is drafted.
3. Keep citations useful. A citation points to a document, page, section or question, along with the exact source version.
4. Put limits in code. Tool access, retries, token budgets and refusal rules belong in the orchestrator, not in prompt wording.
5. Measure each part separately. A good answer can still hide weak retrieval, slow queries or bad paper analysis. Track those failures independently.

1. Reference system architecture

Start with one orchestrator and a few focused tools. Every tool returns results in the same evidence-pack format. The answer writer reads that pack and does not query the stores on its own.

graph TB
  Student["Student · Examcooker course room"] --> Edge["Web app + streaming API"]
  Edge --> Guard["Course boundary guard"]
  Guard --> Orchestrator["RAG orchestrator"]

  Orchestrator --> Router["Request router"]
  Orchestrator --> Memory["Conversation state"]
  Router --> Planner["Search planner"]

  Planner --> Hybrid["Hybrid retriever"]
  Planner --> Graph["Graph traversal"]
  Planner --> Analytics["Exam analytics tool"]

  Hybrid --> Vector[("Vector index")]
  Hybrid --> Lexical[("BM25 index")]
  Graph --> KG[("Knowledge graph")]
  Analytics --> Meta[("Document metadata")]

  Vector --> Pack["Ranked evidence pack"]
  Lexical --> Pack
  KG --> Pack
  Meta --> Pack
  Pack --> Tutor["Answer writer"]
  Tutor --> Verifier["Citation checker"]
  Verifier --> Edge

Keep this as one deployable service at first, with a typed state object passed through the workflow. Split out services later only if load, security boundaries or team ownership make it necessary.

2. Agent topology

These are jobs in the workflow. They do not each need a separate service or model call.

graph TB
  Q["User question"] --> A0["01 · Boundary guard"]
  A0 -->|"course_id + ACL valid"| A1["02 · Request router"]
  A0 -->|"invalid or cross-course"| Stop["Reject / request scope"]
  A1 -->|"Explain topic"| R1["Notes retrieval"]
  A1 -->|"Analyze paper"| R2["Paper parser + question map"]
  A1 -->|"Important questions"| R3["Cross-year exam analytics"]
  A1 -->|"Compare / plan"| R4["Multi-step planner"]
  R1 --> E["Evidence pack"]
  R2 --> E
  R3 --> E
  R4 --> E
  E --> A2["03 · Answer writer"]
  A2 --> A3["04 · Citation checker"]
  A3 -->|"grounded"| Answer["Answer + page citations + confidence"]
  A3 -->|"unsupported"| Retry["Search again or explain what is missing"]
  Retry --> E

The citation checker gets at most one extra search. The orchestrator sets hard limits for retries, model calls, tool calls, retrieved text and total request time.

3. Retrieval pipeline

Vector search alone misses too much in course material. Exact terms, question numbers, marks, years and paper structure all matter, so Examcooker should combine keyword and vector search under the same course filters.

1. Read the course and access rules from the signed session.
2. Work out what the student is asking and prepare queries for that task.
3. Run BM25 over exact fields and document text.
4. Run vector search with the same metadata filters.
5. Follow related topics and question families for one or two graph hops.
6. Merge the result lists with reciprocal rank fusion.
7. Rerank the merged list against the student's actual question.
8. Avoid returning five near-duplicate passages from the same year or document type.
9. Build the evidence pack with the text, page number and source lineage needed for citations.

sequenceDiagram
  autonumber
  actor S as Student
  participant O as Orchestrator
  participant P as Query planner
  participant H as Hybrid search
  participant G as Knowledge graph
  participant X as Exam analytics
  participant L as Tutor LLM
  participant V as Verifier

  S->>O: “Important Module 3 questions from past papers”
  O->>O: Bind course_id + ACL
  O->>P: Classify and decompose request
  par Evidence retrieval
    P->>H: BM25 + vector search with hard filters
    H-->>P: Notes and paper chunks
  and Topic expansion
    P->>G: Module 3 → topics → related questions
    G-->>P: 1–2 hop neighborhood
  and Frequency analysis
    P->>X: Group questions by topic, year, marks
    X-->>P: Recurrence and weighted importance
  end
  P->>L: Ranked evidence pack + response format
  L->>V: Draft with citations
  V-->>O: Checked claims + support notes
  O-->>S: Answer, reasoning, sources, limits

How important questions are ranked

For the first version, score a topic or question family with these weights:

0.30 frequency
+ 0.20 recency
+ 0.20 marks weight
+ 0.15 syllabus alignment
+ 0.10 professor/course similarity
+ 0.05 extraction confidence

Treat these weights as a starting point. Test them against question sets reviewed by faculty, then adjust them. When the assistant calls something important, it should show the years and marks behind that judgment and say plainly that past papers do not guarantee what will appear next.

4. Academic knowledge graph

Vector search can find similar passages, but it is poor at following explicit relationships. The graph keeps track of modules, prerequisites, paper dates, question variants, professors and marks.

graph TB
  U["University"] -->|"OFFERS"| C["Course"]
  C -->|"HAS_MODULE"| M["Module"]
  M -->|"COVERS"| T["Topic"]
  T -->|"PREREQUISITE_OF"| T2["Topic"]
  P["Professor"] -->|"TEACHES"| C
  D["Document"] -->|"BELONGS_TO"| C
  D -->|"AUTHORED_BY"| P
  D -->|"HAS_SECTION"| S["Section / chunk"]
  E["Exam paper"] -->|"CONTAINS"| Q["Question"]
  Q -->|"ASSESSES"| T
  Q -->|"WORTH_MARKS"| K["Marks"]
  Q -->|"VARIANT_OF"| Q2["Question family"]
  E -->|"FOR_COURSE"| C
  E -->|"SET_BY"| P
  E -->|"HELD_IN"| Y["Term + year"]

IDs used across the system

• Course: course_id, course code, title and syllabus_version.
• Document: document_id, course_id, professor_id, type, term, year and version.
• Question: question_id, exam_id, marks, page and normalized_family_id.
• Extracted relationships: confidence, source_document_id, extraction method and review status.

Use Amazon Neptune if the team wants an AWS-managed graph. Choose Neo4j if its query tools are a better fit. In either case, the graph is only an index over relationships. Source text and access rules stay in the main stores.

5. Data plane

Give each store one job.

S3 holds the original files. The team should be able to rebuild every index from an immutable S3 object, its content hash, the parser version and the ingestion version.

6. Versioned ingestion pipeline

graph TB
  S3["S3 · raw PDFs, slides, notes"] --> Event["Object-created event"]
  Event --> Workflow["Ingestion workflow"]
  Workflow --> Validate["Checksum, MIME, malware, ACL validation"]
  Validate --> Parse["OCR + layout-aware parsing"]
  Parse --> Classify["Document / year / professor classifier"]
  Classify --> Chunk["Question-aware + semantic chunking"]
  Chunk --> Enrich["Topics, modules, pages, marks, lineage"]
  Enrich --> Embed["Embedding model"]
  Enrich --> Extract["Entity + relationship extraction"]
  Embed --> Search[("Vector + BM25 indexes")]
  Extract --> Graph[("Knowledge graph")]
  Enrich --> Catalog[("Metadata catalog")]
  Search --> QA["Automated quality gates"]
  Graph --> QA
  Catalog --> QA
  QA -->|"pass"| Ready["Publish corpus_version"]
  QA -->|"review"| Queue["Human review queue"]

Publish all derived indexes together under one corpus_version. A live chat should never see half of a course update.

Chunking strategy

• Notes: split at headings and natural topic boundaries. Keep the page, heading path, slide number and nearby context.
• Question papers: keep each question or sub-question together with its marks, instructions, diagrams and answer choices.
• Extra context: add the parent summary and a small neighboring window when retrieving. Do not depend on a fixed overlap for every document.
• Embedding record: store the model, dimensions, normalization, chunker version and content hash with every vector.

7. API and evidence contracts

The server fills in the required course fields. The model cannot add, remove or override them.

{
  "course_id": "cs_301",
  "professor_id": "prof_17",
  "corpus_version": "2026.07",
  "intent": "important_questions",
  "filters": { "years": [2022, 2023, 2024, 2025] },
  "budgets": { "top_k": 30, "rerank_k": 10 }
}

Each evidence item carries the fields needed to show a useful citation.

{
  "document_id": "exam_2025_endsem",
  "source_version": "sha256:…",
  "page": 3,
  "question_id": "q_2025_07b",
  "topic_ids": ["topic_deadlocks"],
  "quote": "Explain the necessary conditions…",
  "scores": { "hybrid": 0.87, "reranker": 0.93 }
}

8. Trust and security architecture

Treat prompts and document contents as untrusted input.

1. Check access first: read allowed courses from the signed session. Do not accept a course scope that appears only in user text or model output.
2. Keep document text in its lane: mark retrieved passages as evidence, strip active content and ignore instructions embedded in a PDF or note.
3. Protect source files: use short-lived signed URLs for previews. Never send raw S3 keys to the browser.
4. Leave an audit trail: record the course scope, search plan, source IDs, model version, citations and policy result. Avoid storing personal data that is not needed.
5. Say when the evidence is not enough: explain missing or conflicting sources instead of filling the gap with a guess.
6. Cap cost and abuse: apply user quotas, retrieval limits, model routing, stream limits and anomaly detection in code.

9. Evaluation and observability

Build a small reviewed test set for each course. Include questions from faculty as well as the kinds of questions students actually ask.

Trace the search plan, tool calls, candidate counts, filters, ranks, model versions, token use, citations, checker result and user feedback. Do not put private source text into telemetry unless there is a clear operational need.

10. Recommended deployment

Keep ingestion and search close to the existing S3 corpus. Put model calls behind a gateway so changing providers does not require rewriting the workflow.

graph TB
  Web["Next.js / React web"] --> CDN["CDN + WAF"]
  CDN --> API["API Gateway / course chat API"]
  API --> Auth["University SSO / JWT"]
  API --> Runtime["RAG service · containers or serverless"]
  Runtime --> Models["Model gateway"]
  Runtime --> Cache[("Redis response + retrieval cache")]
  Runtime --> OpenSearch[("OpenSearch hybrid index")]
  Runtime --> Neptune[("Neptune / Neo4j graph")]
  Runtime --> Postgres[("Aurora PostgreSQL metadata")]
  Runtime --> S3[("S3 original files")]
  S3 --> Pipeline["Step Functions ingestion"]
  Pipeline --> OpenSearch
  Pipeline --> Neptune
  Pipeline --> Postgres
  Runtime --> Observe["Traces, evals, cost, audit logs"]

Stack for the first release

• Search: OpenSearch for filtered BM25 and vector search.
• Metadata: Aurora PostgreSQL.
• Graph: Neptune or Neo4j. Pick one after testing the queries the product actually needs and reviewing the operational tradeoffs.
• Runtime: ECS/Fargate for streaming and predictable dependencies; Lambda for bursty ingestion steps.
• Workflow: Step Functions, queues and dead-letter handling for durable ingestion.
• Model gateway: Bedrock or a provider-neutral proxy with task routing and model version tracking.

11. Delivery roadmap

Phase 1: Corpus foundation

Set the IDs, S3 layout, parser benchmark, metadata catalog and access rules for three pilot courses. This phase is done when page extraction and metadata accuracy both reach 95%.

Phase 2: Search and cited answers

Build hybrid search, reranking, evidence packs, cited answers, feedback and offline tests. This phase is done when citation precision reaches 95% on the pilot benchmark.

Phase 3: Exam intelligence

Add question extraction, question-family grouping, the knowledge graph, recurrence scoring and the explanation UI. This phase is done when faculty reviewers approve the method and a sample of its results.

Phase 4: Production hardening

Finish adversarial testing, monitoring, caching, cost routing and load tests, then roll out courses in small batches. Before launch, document the SLOs, complete the security review and write the operations runbook.

What a good answer looks like

These topics appeared most often in the last four papers. Here are the years, marks and exact questions behind that result.

A useful answer is specific about its sources and honest about what those sources cannot prove.