ECExamcookerArchitecture · 01Recommended stack
Agentic RAG blueprint · v1.0

The intelligence layer for Examcooker.

A course-scoped, citation-first architecture that turns years of university notes and past papers into a tutor that can explain, compare and reason over exam evidence.

Explore the systemRead the rollout planOpen Markdown spec
01Hard course boundary

Every retrieval call carries university_id, course_id, ACL and corpus_version.

02Evidence, then answer

Hybrid search, graph expansion and reranking form a bounded evidence pack.

03Traceable by design

Answers cite document, page, section and source version—not opaque chunk IDs.

01 / System architecture
The whole system

One orchestrator.
Specialist tools. Hard boundaries.

Keep the runtime deterministic where correctness matters. A small state machine owns scope, tool permissions and retry budgets; model-driven agents operate only inside those rails.

Mermaid · graph TB

Reference runtime topology

The evidence pack is the contract between retrieval and generation. The tutor never queries stores directly.

View Mermaid source
graph TB
  Student["Student · Examcooker course room"] --> Edge["Web app + streaming API"]
  Edge --> Guard["Course boundary guard"]
  Guard --> Orchestrator["RAG orchestrator"]

  Orchestrator --> Router["Intent + task router"]
  Orchestrator --> Memory["Conversation state"]
  Router --> Planner["Evidence planner"]

  Planner --> Hybrid["Hybrid retriever"]
  Planner --> Graph["Graph traversal"]
  Planner --> Analytics["Exam analytics tool"]

  Hybrid --> Vector[("Vector index")]
  Hybrid --> Lexical[("BM25 index")]
  Graph --> KG[("Knowledge graph")]
  Analytics --> Meta[("Document metadata")]

  Vector --> Pack["Ranked evidence pack"]
  Lexical --> Pack
  KG --> Pack
  Meta --> Pack
  Pack --> Tutor["Tutor + synthesis agent"]
  Tutor --> Verifier["Citation + claim verifier"]
  Verifier --> Edge

  classDef focus fill:#e2e8f9,stroke:#1747d1,stroke-width:2px,color:#1c211f;
  classDef store fill:#faf8f1,stroke:#6f746f,color:#1c211f;
  class Orchestrator,Pack,Verifier focus;
  class Vector,Lexical,KG,Meta store;
02 / Agent topology
A bounded agent system

Six roles, one shared state.

These are logical responsibilities, not necessarily six separate model calls. Start with one orchestrated runtime and split services only when evaluation proves the need.

01

Boundary guard

Binds every turn to university, course, role and corpus version before retrieval can run.

02

Intent router

Chooses explain, summarize, paper analysis, important-question mining or study-plan workflows.

03

Evidence planner

Decomposes the task, selects tools and sets retrieval budgets—without writing the answer.

04

Exam analyst

Normalizes question families and calculates recurrence, marks, recency and syllabus coverage.

05

Tutor synthesizer

Explains at the requested depth using only the assembled, ranked evidence pack.

06

Claim verifier

Checks each factual sentence against cited spans; retries weak retrieval or abstains.

Mermaid · controlled workflow

Routing and retry policy

Only the verifier may trigger a second retrieval pass. The maximum retry count and token budget live in orchestration code.

View Mermaid source
graph TB
  Q["User question"] --> A0["01 · Boundary guard"]
  A0 -->|"course_id + ACL valid"| A1["02 · Intent router"]
  A0 -->|"invalid or cross-course"| Stop["Reject / request scope"]
  A1 -->|"Explain topic"| R1["Notes retrieval"]
  A1 -->|"Analyze paper"| R2["Paper parser + question map"]
  A1 -->|"Important questions"| R3["Cross-year exam analytics"]
  A1 -->|"Compare / plan"| R4["Multi-step planner"]
  R1 --> E["Evidence pack"]
  R2 --> E
  R3 --> E
  R4 --> E
  E --> A2["03 · Tutor synthesizer"]
  A2 --> A3["04 · Claim verifier"]
  A3 -->|"grounded"| Answer["Answer + page citations + confidence"]
  A3 -->|"unsupported"| Retry["Retrieve again or abstain"]
  Retry --> E

  classDef focus fill:#e2e8f9,stroke:#1747d1,stroke-width:2px,color:#1c211f;
  class A0,A1,E,A3 focus;
03 / Retrieval pipeline
Retrieval that understands exams

Filter. Search. Expand. Rerank.

Dense similarity alone misses exact terminology, marks and paper structure. Examcooker should combine lexical, semantic and graph signals, then rerank against the actual task.

01Bind scopecourse + ACL
02Rewriteintent-aware
03Hybrid searchBM25 + vector
04Graph expand1–2 hops
05FuseRRF
06Rerankcross-encoder
07Diversifytopic + source
08Packagecitable spans
Mermaid · sequence

“Important questions” request

Parallel retrieval reduces latency while preserving an explicit, inspectable chain from question to evidence.

View Mermaid source
sequenceDiagram
  autonumber
  actor S as Student
  participant O as Orchestrator
  participant P as Query planner
  participant H as Hybrid search
  participant G as Knowledge graph
  participant X as Exam analytics
  participant L as Tutor LLM
  participant V as Verifier

  S->>O: “Important Module 3 questions from past papers”
  O->>O: Bind university_id + course_id + ACL
  O->>P: Classify and decompose request
  par Evidence retrieval
    P->>H: BM25 + vector search with hard filters
    H-->>P: Notes and paper chunks
  and Topic expansion
    P->>G: Module 3 → topics → related questions
    G-->>P: 1–2 hop neighborhood
  and Frequency analysis
    P->>X: Group questions by topic, year, marks
    X-->>P: Recurrence and weighted importance
  end
  P->>L: Ranked evidence pack + answer contract
  L->>V: Draft with atomic citations
  V-->>O: Supported claims + calibrated confidence
  O-->>S: Answer, rationale, sources, uncertainty
Evidence-based ranking

Topic importance score

0.30 frequency + 0.20 recency + 0.20 marks weight
+ 0.15 syllabus alignment + 0.10 professor similarity
+ 0.05 extraction confidence

Weights are a launch hypothesis. Calibrate them against faculty-labelled historical sets rather than presenting them as truth.

Language contract

High evidence is not a prediction.

The assistant should say “strongly represented in past papers,” show the years and marks, and explicitly note that future exam content cannot be guaranteed.

04 / Knowledge graph
Graph DB layer

The connective tissue vectors cannot model.

The graph makes questions explainable: which topic they assess, which module owns it, who taught it, when it appeared and which other questions are semantic variants.

Mermaid · graph TB

Canonical academic ontology

Use stable IDs from the metadata catalog. Extracted edges carry confidence, source_document_id and review status.

View Mermaid source
graph TB
  U["University"] -->|"OFFERS"| C["Course"]
  C -->|"HAS_MODULE"| M["Module"]
  M -->|"COVERS"| T["Topic"]
  T -->|"PREREQUISITE_OF"| T2["Topic"]
  P["Professor"] -->|"TEACHES"| C
  D["Document"] -->|"BELONGS_TO"| C
  D -->|"AUTHORED_BY"| P
  D -->|"HAS_SECTION"| S["Section / chunk"]
  E["Exam paper"] -->|"CONTAINS"| Q["Question"]
  Q -->|"ASSESSES"| T
  Q -->|"WORTH_MARKS"| K["Marks"]
  Q -->|"VARIANT_OF"| Q2["Question family"]
  E -->|"FOR_COURSE"| C
  E -->|"SET_BY"| P
  E -->|"HELD_IN"| Y["Term + year"]

  classDef focus fill:#e2e8f9,stroke:#1747d1,stroke-width:2px,color:#1c211f;
  class C,M,T,Q focus;
Node

Course

course_id · university_id · code · title · syllabus_version
Node

Document

document_id · course_id · professor_id · type · year · version
Node

Question

question_id · exam_id · marks · page · normalized_family_id
Edge metadata

ASSESSES

topic_id · confidence · extraction_method · verified_at
05 / Data plane
Right store, right job

Five storage roles—no overloaded database.

S3 remains the source of truth. Every derived representation is reproducible from an immutable source object and ingestion version.

01

S3

Canonical content

Versioned PDFs, slides, extracted JSON, page images and immutable lineage.

02

Aurora PostgreSQL

Control plane

IDs, permissions, document status, ingestion jobs, sessions and feedback.

03

OpenSearch

Hybrid retrieval

BM25 fields and dense vectors in one filtered, scalable retrieval surface.

04

Neptune / Neo4j

Knowledge graph

Course → module → topic → question relationships and GraphRAG expansion.

05

Redis

Latency layer

Session state, semantic cache, rate limits and short-lived evidence packs.

06 / Corpus engineering
Ingestion is a product

From uploaded PDF to trusted evidence.

Treat parsing, metadata and quality as first-class. Most RAG failures blamed on the model are broken page structure, missing scope or weak chunk lineage.

Mermaid · graph TB

Versioned ingestion pipeline

Publish indexes atomically by corpus_version so live conversations never see a half-ingested course.

View Mermaid source
graph TB
  S3["S3 · raw PDFs, slides, notes"] --> Event["Object-created event"]
  Event --> Workflow["Ingestion workflow"]
  Workflow --> Validate["Checksum, MIME, malware, ACL validation"]
  Validate --> Parse["OCR + layout-aware parsing"]
  Parse --> Classify["Document / year / professor classifier"]
  Classify --> Chunk["Question-aware + semantic chunking"]
  Chunk --> Enrich["Topics, modules, pages, marks, lineage"]
  Enrich --> Embed["Embedding model"]
  Enrich --> Extract["Entity + relationship extraction"]
  Embed --> Search[("Vector + BM25 indexes")]
  Extract --> Graph[("Knowledge graph")]
  Enrich --> Catalog[("Metadata catalog")]
  Search --> QA["Automated quality gates"]
  Graph --> QA
  Catalog --> QA
  QA -->|"pass"| Ready["Publish corpus_version"]
  QA -->|"review"| Queue["Human review queue"]

  classDef focus fill:#e2e8f9,stroke:#1747d1,stroke-width:2px,color:#1c211f;
  class Workflow,Chunk,QA,Ready focus;
Notes

Section-aware

Chunk by heading and semantic boundary; preserve page, heading path, slide number and neighboring context.

Question papers

Question-aware

One question or sub-question per unit with marks, instructions, diagrams and answer-option relationships intact.

Overlap

Purposeful, not fixed

Add parent summaries and small neighbor windows at retrieval time instead of duplicating arbitrary token overlaps.

Embeddings

Version everything

Record model, dimensions, normalization, chunker version and content hash on every vector.

07 / Contracts
Stable interfaces

Make scope and citations impossible to omit.

Typed contracts keep model output downstream of policy. The API—not the prompt—enforces identifiers, budgets and visible provenance.

Retrieval requestapplication/json
{
  "university_id": "uni_01",
  "course_id": "cs_301",
  "professor_id": "prof_17",
  "corpus_version": "2026.07",
  "intent": "important_questions",
  "filters": { "years": [2022, 2023, 2024, 2025] },
  "budgets": { "top_k": 30, "rerank_k": 10 }
}
Evidence itemapplication/json
{
  "document_id": "exam_2025_endsem",
  "source_version": "sha256:…",
  "page": 3,
  "question_id": "q_2025_07b",
  "topic_ids": ["topic_deadlocks"],
  "quote": "Explain the necessary conditions…",
  "scores": { "hybrid": 0.87, "reranker": 0.93 }
}
08 / Trust architecture
Secure by construction

Assume documents and prompts are untrusted.

University content has access boundaries and uploaded documents can contain adversarial instructions. Both require enforcement outside the model.

01

Authorization before retrieval

Derive allowed course_ids from the signed session. Never accept scope only from user text or model output.

02

Document prompt isolation

Mark retrieved text as evidence, strip active content and instruct tools to ignore commands found inside documents.

03

Private source delivery

Use short-lived signed URLs only for authorized source previews; never expose raw S3 keys as public links.

04

Auditability

Log user scope, plan, source IDs, model version, citations and policy outcomes—without storing unnecessary personal data.

05

Abstention policy

If evidence is weak, contradictory or out of scope, say so and offer the closest supported material.

06

Cost and abuse controls

Per-user quotas, retrieval caps, model-tier routing, streaming limits and anomaly detection protect the service.

09 / Evaluation
Ship with a test set

Measure the chain, not just the prose.

Build evaluation sets course-by-course with faculty and student reviewers. A good-looking answer can still be grounded in the wrong paper.

LayerLaunch metricTest method
RetrievalRecall@20, nDCG@10, topic coverageGold question → source sets per course
GroundingCitation precision ≥ 95%Atomic claim-to-span verification
AnswerCorrectness, completeness, pedagogyFaculty rubric + blinded student review
Exam analyticsQuestion-family F1, year/marks accuracyHuman-normalized past-paper benchmark
OperationsP95 latency, cost/answer, cache hitTraces split by workflow and course
SafetyCross-course leakage = 0Adversarial ACL and prompt-injection suite
10 / Deployment
Recommended reference stack

AWS-native data plane, model-neutral runtime.

Because the corpus already lives in S3, keep ingestion and data close to it. Put every model behind a gateway so quality and cost can evolve independently.

Mermaid · graph TB

Production deployment topology

Start with managed services and one agent runtime. Split components only when scale, ownership or isolation requires it.

View Mermaid source
graph TB
  Web["Next.js / React web"] --> CDN["CDN + WAF"]
  CDN --> API["API Gateway / RAG API"]
  API --> Auth["University SSO / JWT"]
  API --> Runtime["Agent runtime · containers or serverless"]
  Runtime --> Models["Model gateway"]
  Runtime --> Cache[("Redis response + retrieval cache")]
  Runtime --> OpenSearch[("OpenSearch hybrid index")]
  Runtime --> Neptune[("Neptune / Neo4j graph")]
  Runtime --> Postgres[("Aurora PostgreSQL metadata")]
  Runtime --> S3[("S3 source of truth")]
  S3 --> Pipeline["Step Functions ingestion"]
  Pipeline --> OpenSearch
  Pipeline --> Neptune
  Pipeline --> Postgres
  Runtime --> Observe["Traces, evals, cost, audit logs"]

  classDef focus fill:#e2e8f9,stroke:#1747d1,stroke-width:2px,color:#1c211f;
  class Runtime,OpenSearch,Neptune,Observe focus;
Recommended launch

OpenSearch + Aurora + Neptune

Strong fit for S3-based ingestion, filtered hybrid search, graph traversal and clear operational ownership.

Model gateway

Bedrock or provider-neutral proxy

Route small models for classification, larger models for synthesis and a dedicated reranker for evidence quality.

Runtime

ECS/Fargate first

Streaming, predictable dependencies and easier long-running workflows; move bursty ingestion steps to Lambda.

Workflow

Step Functions + queues

Durable ingestion, idempotent retries, review queues and dead-letter handling for malformed documents.

11 / Delivery roadmap
Build evidence, then sophistication

A four-phase path to production.

Avoid building the full graph and multi-agent workflow before proving clean ingestion, retrieval quality and citation accuracy on representative courses.

Phase 01Weeks 1–3

Corpus foundation

Canonical IDs, S3 conventions, parser benchmark, metadata catalog, ACLs and 3 pilot courses.

Exit: ≥ 95% page and metadata accuracy
Phase 02Weeks 4–6

Citation-first RAG

Hybrid retrieval, reranking, evidence packs, tutor responses, feedback and offline evaluation.

Exit: ≥ 95% citation precision
Phase 03Weeks 7–10

Exam intelligence

Question extraction, family clustering, knowledge graph, recurrence scoring and explanation UI.

Exit: faculty-approved analytics
Phase 04Weeks 11–14

Production hardening

Red-team tests, observability, caching, cost routing, load tests and staged course rollout.

Exit: SLOs and security sign-off
North-star answer contract
“Here are the topics most strongly represented in the last four papers, why they rank highly, and the exact pages and questions that support the recommendation.”
ScopedGroundedExplainableMeasurable