FailSafeRAG
Retrieval checks and targeted recovery for damaged documents.
A document-answering system can retrieve plausible text even when optical character recognition has damaged the page. A fluent answer can hide that failure.
I implemented FailSafeRAG to test whether a system can detect weak retrieval evidence and recover before relying on it.
Check the evidence before accepting an answer
The pipeline chunks OCR text and retains document metadata. It combines BM25 keyword retrieval with dense embeddings through reciprocal rank fusion.
A quality check evaluates OCR word noise, layout damage, weak keyword evidence, and weak embedding evidence separately. Failed examples are routed to a recovery step.
| Failure | Recovery |
|---|---|
| Word-level damage | Correct the OCR text |
| Structural damage | Inspect selected page images |
| Weak semantic retrieval | Retry retrieval |
Only the visual recovery path uses image input. Text correction and retrieval retries do not require a vision model.
Compare recovered answers with direct answers
Each answer includes answer_mode, support_score, and its source chunk. The experiment runner compares the recovered answer with the direct answer for the same example.
The runner records ndcg@5 and recall@5 for retrieval, plus exact match and F1 for answers. Those comparisons are more informative than whether recovery completed.
The default visual path uses a mock model
The vision backend can use a mock or a real provider. The default demo uses the mock, which lets the recovery path run without external API calls.
A successful mock run verifies the execution path. It does not show that a vision model made a useful judgment or improved the answer.
Detection limits recovery
Thresholds were tuned to the benchmark corpus. A different document collection needs its own evaluation.
The system can recover only from failures that its checks detect. Incorrect evidence can still pass when the text looks clean.
This is a research implementation. The documented workflow does not establish a general improvement across document types or models.