Mini Search Engine with RAG Extension
Build a classical information retrieval system first, then extend it with grounded answer generation.
Local project explainer
Ask about this system
A local, predefined assistant for the documented details in Mini Search Engine with RAG Extension.
Problem
What this project had to solve
RAG systems are easier to market than to evaluate. This project starts with a classical IR system on Cranfield, then adds a generation layer so retrieval quality stays measurable instead of disappearing behind an LLM response.
Role / contribution
What I owned in the system
Project author responsible for dataset parsing, indexing, retrieval, evaluation, and the RAG extension described in the repository.
Technical implementation
How the system is built
The core engine parses Cranfield, preprocesses text, builds an inverted index, and constructs a sparse TF-IDF retrieval pipeline ranked with cosine similarity.
Evaluation is explicit through MAP and NDCG, which keeps retrieval behavior inspectable before generation is layered on top.
The RAG extension passes top retrieved documents into Gemini so answers stay connected to the retrieval output rather than free-form generation.
Architecture flow
System flow from input to output
Dataset parsing
What happens
Cranfield documents, queries, and relevance labels are extracted into a structured IR pipeline.
Why it matters
A structured benchmark pipeline is what makes retrieval evaluation possible from the start.
Related tools
Engineering decisions
Tradeoffs that shaped the build
Keep classical IR visible
The project does not hide retrieval behind the generation layer; evaluation remains a first-class output.
Separate retrieval and generation concerns
The RAG extension is layered on top of a measurable baseline rather than mixed into the core retrieval logic.
Use benchmark-driven reasoning
Cranfield gives the project a concrete evaluation frame instead of anecdotal search results.
Stack
Core tools and system layers
What I would improve next
Next technical upgrades
- Add a cleaner API boundary so retrieval and RAG can be consumed as services, not just scripts.
- Test hybrid retrieval strategies against the current classical baseline.
- Add response-side faithfulness checks to complement retrieval metrics.