AI Vendor Risk Assessor

0Start here

You have finished three courses. Each one built a serious component in isolation: a production RAG pipeline over company documents, an agentic RAG crew that routes and grades, and a deep search agent that researches the open web and refuses to be fooled. This capstone puts the three together into one product that a company would use, and adds the glue that none of the courses taught: a deterministic requirement matcher, a questionnaire drafter, a human approval gate and a UI.

Prerequisites

• Completed Production RAG from Scratch (you have a working rag_core/ folder).
• Completed Agentic RAG with DeepSeek + CrewAI (you understand Router, Retriever and the two graders).
• Completed Deep Search Agents (you have run run_agent.py offline and live, and Lab 12 in LangGraph).
• Python 3.10+, a free LLM key (Gemini or Groq) or Ollama, and a free web-search option (ddgs, no key).

Day one: run the starter kit

cd vendor-risk-assessor
python -m venv .venv && source .venv/bin/activate      # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txt
cp .env.example .env
python scripts/check_setup.py
python run_assessor.py                                   # LedgerLens AI, customer data, 35 users
python -m pytest -q                                      # 70 tests
python eval/run_eval.py                                  # the scorecard

The default run finishes in under a second with no key and no internet, and prints a full review file ending in BLOCKED. Read runs/<timestamp>-assess/report.md and then trace.jsonl. That report is the target; the rest of this page is how you make the system that produces it real.

1The problem

Every company now has employees asking “can I use this AI tool?” several times a week. Each request should trigger a vendor security review: does the vendor hold a current SOC 2 Type II or ISO 27001? Where is our data stored? Do they train on it? Who are their sub-processors? Have they had an incident? Is a DPA available? Today a security engineer spends two to six hours per vendor reading trust centres, privacy policies and news, and most of that time is finding and dating facts, not judging them. Teams either wait weeks for a review or skip it.

The assessor does the finding, dating and checking. It leaves the judging to a person, with every fact already attached to its source.

The running example: Nimbu Mart

You know Nimbu Mart from the Production RAG course. Its Information Security team has eight internal documents — two editions of the security policy, a data-protection policy written for India’s DPDP Act, an AI acceptable-use policy, a procurement checklist, the restricted Approved Vendor List, a questionnaire template and an untrusted chat export. Three fictional AI vendors want in:

What “good” looks like

The review file the starter kit produces for LedgerLens at the customer-data tier looks like this (abridged):

Then: the facts with confidence and conflicts, a security section (“1 page quarantined for prompt injection; 1 rejected as a different company; 1 rumour dropped”), six ready-to-send questions for the vendor, and an approval block with the two roles who must sign (Head of Information Security and DPO). The outcome line says BLOCKED for this data tier — never “rejected”, and never “approved”.

2Architecture

The components

3Course map: which part needs which course section

Use this table as your index. When you open a LEARNER TODO in the code, this is where to look first. “Reuse” names the files from the course code packs you can copy or import.

4Milestones

Do them in order. Each one ends with a check you can run. Tick the box when the check passes; the page remembers it in your browser.

5The data pack

6Rules of the project

1. The model never decides a verdict. It reads pages, drafts text and classifies routes. matcher.py decides, from rules you can unit-test. Two runs on the same evidence give the same verdict.
2. UNKNOWN beats a guess. A PASS or FAIL without an evidence URL is a bug; the output guardrail downgrades it. The evaluation’s guess_rate must stay at zero.
3. A claim is not evidence. “We are SOC 2 compliant” is claimed. A report, a certificate number, a registry entry or a legal page is documented. Only documented evidence passes a high-stakes requirement.
4. Identity is a gate, not a weight. A page about a same-name company is rejected before anything is read from it. Down-weighting it would still let its facts leak in.
5. Untrusted content never reaches the verdict. Pages that address the AI are quarantined whole. Forum posts and rumours are dropped and logged, not scored low.
6. Repetition is not corroboration. A syndicated copy of the vendor’s page counts once.
7. Access control lives in retrieval. The Approved Vendor List is invisible to people outside Security and Procurement, and the assistant says so instead of leaking it. Groups come from identity on the server, never from the request body.
8. Every fact has a date. Freshness decays per field. A 2024 security page loses to a 2026 trust centre, and the conflict is recorded, not hidden.
9. Humans approve. The report ends in an approval block that only a named person can fill. The word “approved” never appears in anything the system writes about a vendor.
10. Research companies, not people. No names of vendor employees in the schema, the memory or the report.

7Evaluation & rubric

The scorecard your evaluation must report

Rubric (100 points)

8Demo checklist (12 minutes)

1. The question (1 min): a support lead asks whether the team can use LedgerLens for tickets. Show the review file. Point at the BLOCKED line and at DP-3.1 with its source and date.
2. The trace (2 min): open trace.jsonl. Show the rejected UK firm, the quarantined blog page, the dropped rumour, the 2024 page losing to the 2026 trust centre.
3. Access control (1 min): ask the same question as an employee outside Security. PROC-1.1 turns UNKNOWN; the Approved Vendor List never appears. Ask “which vendors are approved?” and get the process, not the list.
4. The good vendor (2 min): Parakeet at T3 — every requirement passes on documented evidence, yet the outcome is CONDITIONAL because approval is for T2 only. Switch to T2 and show READY FOR HUMAN APPROVAL. Try to approve the BLOCKED LedgerLens report: refused.
5. Live (3 min): a real vendor from your gold set, from the UI, with progress streaming. Compare two fields against what you found by hand.
6. Attack (1 min): add an injection line to a cached page and re-run. Quarantined; verdicts unchanged.
7. Numbers (2 min): the scorecard, the ablation table, cost and latency, and the two things that still go wrong.

9Stretch goals

• Continuous monitoring. A weekly scheduled re-review of every approved vendor that only reports changes (new sub-processor, new incident, certificate expiring in 60 days). Deep Search Module 7 memory + Module 12 “regulatory circular monitor” idea.
• MCP server. Expose assess_vendor and ask_policy as MCP tools so Claude Desktop or your IDE can call them (Deep Search Lab 5).
• Debate layer. Before the human gate, a “vendor advocate” and a “risk officer” agent each write one paragraph from the same evidence; the report shows both. Useful when evidence conflicts.
• Document intake. Let the vendor upload its SOC 2 report PDF; extract the audit period and the auditor’s opinion with the Production RAG OCR/table pipeline (Lab 02) and upgrade claimed to documented automatically.
• Multi-tenant. A second company with different policies (tenant_id on every chunk and rule), as in the Production RAG capstone exercise.

10FAQ

Why not let the LLM read the policy and the evidence and decide?

Because the verdict must be the same on Tuesday as it was on Monday, and a reviewer must be able to point at the line of policy and the line of evidence behind it. Models are excellent at reading and extracting; they are not consistent judges. Keeping the judgement in code is what makes the system auditable and the evaluation meaningful.

Isn’t a rule engine “not AI”?

The intelligence is in reading a thousand messy pages and turning them into typed, dated, sourced facts — and in knowing which pages to distrust. That is where every hour of the project goes. The rule engine is 300 lines and stays that way.

What if a vendor’s pages block my fetcher?

Cache what you can, record UNKNOWN for what you cannot, and let the questionnaire ask the vendor. An honest UNKNOWN with a question attached is a correct output. Playwright for JavaScript-rendered trust centres is a reasonable stretch.

Can I do this solo?

Yes: do M0–M4 fully, M5 as the CLI + API only, and three real vendors instead of ten. That is a complete, demonstrable project in about three weeks.

Which LLM should I use?

Whatever is free and supports JSON output: Gemini’s free tier or Groq for development, Ollama for offline. Use the small model for extraction and grading and a stronger one only for planning and synthesis — then show the cost difference in your report.

AGlossary