CIS 4394 Agentic AI · Fall 2026 · Dr. Xinyu Fu
CIS 4394 Agentic AI · Fall 2026 · Dr. Xinyu Fu
The model you installed in Week 4 has never seen this course. Today you embed the course website on your own laptop, retrieve the right passage for a question, and watch the same model go from a confident wrong answer to a cited right one. No API key, no vector database to install, nothing leaves your machine.
7_rag_local.py), the corpus builder, and corpus/: the text of this course website, 38 pages. Unzip it into the same week5_ollama_starter folder you used two weeks ago, so the files sit next to 1_agent_raw.py. Nothing new to pip install.Your chat model turns text into more text. Retrieval needs a model that turns text into numbers: a vector whose direction encodes meaning (page 02). That is a different kind of model, and Ollama will refuse to use a chat model for it. Pull one (274 MB; it takes about a minute on campus Wi-Fi):
Slow connection? ollama pull all-minilm is 46 MB and works for this lab (retrieval is a little worse). If you use it, put EMBED_MODEL=all-minilm in front of every command below, the same way MODEL= works in the Week 5 README.
Open a terminal and move into your week5_ollama_starter folder, the one that contains 1_agent_raw.py. If it is on your Desktop:
Forgot how to find a folder? Week 5, page 03, Step 1 walks through it. The prompt shows the folder you are standing in, if it does not say week5_ollama_starter, you are not there yet.
Before you give it anything, ask your model a question about this course. It has never seen the course. Watch what it does with that.
What qwen2.5:0.5b said on the instructor's laptop:
A date, a weekday, four lecture numbers; none of it exists. It did not say "I don't know." Write down what your model said; you will need it for the group task.
First look at the cutting without embedding anything. This is the Lakeside exercise from page 04, done by code:
Each chunk keeps its metadata, which page it came from and where on the page. That label is what lets the answer carry a citation later. Now embed for real (about 20 seconds; a progress counter runs):
Most RAG failures are retrieval failures, and you can only see them if you look at this step on its own. search embeds your question, finds the three closest chunks, and stops: no model, no answer.
Read the top hit. It is about Quiz 1. The right chunk is second, by 0.014. An embedding encodes meaning, and "quiz next week, closed-book, five multiple-choice" means almost the same thing whether the number is 1 or 3. Numbers are weak signals in this geometry. Keep that in mind for Step 6.
ask runs the retrieval you just saw, pastes the three chunks into the prompt as numbered passages, and tells the model to answer only from them and cite the number.
With a 3B reader (llama3.2, 2 GB), the instructor's laptop gets this every time:
Nothing about the model changed between Step 2 and now. The desk changed. And it cited [2], the chunk that was ranked second, because the model, not the ranking, decided which passage answered the question.
qwen2.5:0.5b, expect a rougher ride. Run it five times and count:
ollama pull llama3.2 (2 GB) and put MODEL=llama3.2 in front of the command (Windows PowerShell: $env:MODEL="llama3.2"; python 7_rag_local.py ask …).Look at the 0.5B answers. Rand-Hendriksen is a real name. It is the author of next week's LinkedIn Learning course, sitting in one of the retrieved passages. The model took a real name from the desk and stapled it to your question. RAG does not end hallucination; it changes its raw material. The only reason the bigger model said "not provided" is that the prompt told it that answer was allowed: the same abstention lesson as the cannot tell option in Week 5. If you do not write the exit into the instructions, the model has no exit.
Retrieval has settings, and the settings are yours, not the model's. Change one at a time. (Windows PowerShell: set the variable first, e.g. $env:TOP_K="1", then run the command without the prefix.)
The right chunk was ranked second, so with one slot it never reached the desk. The model did the honest thing and still failed you. Retrieval quality is destiny: the generator cannot recover a passage it was never shown.
Chunks four times bigger carry four times more unrelated text, so every vector points in a vaguer direction and the Quiz 3 chunk falls out of the top three. The model then answered from the wrong page: wrong, confident, and with a citation. A citation proves the model read the passage. It does not prove the passage was the right one.
1 · Between Step 2 and Step 5, what changed: list everything that did and everything that did not.
2 · In Step 7, Knob 1 produced an honest "no mention of Quiz 3." Where is the bug: in the model, the retriever, or the prompt?
TOP_K=1 only the wrong one was shown. The model behaved correctly on what it was given and the prompt did its job. Fixes live at the retrieval layer: a larger k, chunk metadata the query can filter on ("week 7"), a keyword match combined with the vector match (hybrid search), or a re-ranking step. Blaming the model here would send you to a bigger model that would fail the same way.3 · Knob 2 produced a wrong answer with a citation. In an HR or legal deployment, why is that outcome worse than Step 2's hallucination?
Pick three questions about this course whose answers you can verify on the website: one easy (the answer sits in one place), one hard (the answer is spread over two pages or two weeks), and one trap (sounds like a course question but is not answered anywhere). For each, run ask --no-rag, search, and ask, and fill one row:
| question | closed book said | right chunk in top 3? | open book said | verdict: right · wrong · abstained |
|---|---|---|---|---|
| easy | ||||
| hard | ||||
| trap |
Produce: the table, plus one sentence each on (1) which failure you saw (retrieval, reading, or abstention) and how you know, and (2) the single setting change you would try first for the hard question. Bring your trap question; we will run the best ones on the projector.
search output as evidence, not the final answer: "the right chunk was ranked 4th, so this is a retrieval failure; raising k to 5 is the first thing to try" or "the right chunk was ranked 1st and the 0.5B model still answered 'Week 3', so this is a reading failure: a bigger reader, not a bigger k." For the trap question, a strong pair checks whether the model abstained or stitched (a real name from the passages attached to a false claim) and notes that only the abstention instruction in the prompt made the honest answer possible. Weak pairs report only what the final answer said.Replace corpus/ with the text your agent needs (a product catalog, a policy manual, past tickets) and 7_rag_local.py is a working retrieval tool for your capstone with no API bill. Then make retrieval a tool the agent calls when it decides it needs to (the agentic RAG pattern on page 02): the retrieve() function in this script, registered the way calculator was in Week 5's 1_agent_raw.py. Three decisions are yours and will be graded: how you chunk, how many passages you pass, and what the model is told to do when the passages do not contain the answer.
| You see | What it means | Fix |
|---|---|---|
This server does not support embeddings | You are embedding with a chat model. Ollama will not do that. | ollama pull nomic-embed-text, and do not set EMBED_MODEL to your chat model. |
index was built with X, but EMBED_MODEL=Y | You changed embedding models after building the index. Vectors from two models are not comparable. | Run index again. |
No such file or directory: corpus | You are not in the week5_ollama_starter folder, or the add-on was unzipped somewhere else. | Read the prompt; cd to the folder that contains both 1_agent_raw.py and corpus/. |
No module named ollama | Python cannot see the package. | pip install -r requirements.txt (Week 5, Step 1). |
The answer is just [2] or one word | A very small model latched onto the format instruction. | Run with --runs=5 and count; or use MODEL=llama3.2. |
| Embedding is slow | Older laptop, or another model is loaded in memory. | Use all-minilm; close other Ollama sessions. |