Cutting RAG Inference Costs 6x by Filtering Data Before LLM
VentureBeat's analysis proposes an architectural shift for RAG systems, significantly reducing LLM calls. This approach can cut costs and improve auditability by prioritizing deterministic and retrieval-based resolutions.
VentureBeat outlines a new approach to building Retrieval Augmented Generation (RAG) systems, particularly for regulated industries, that dramatically cuts costs and improves auditability. The conventional method of routing all ambiguous cases directly to a Large Language Model (LLM) is expensive and difficult to scrutinize after the fact. The proposed "cascade architecture" aims to resolve most cases without LLM intervention, reducing inference costs by up to six times.
The analysis highlights that a common RAG architecture, where every ambiguous case is sent to an LLM, works for demos but fails in high-stakes environments requiring long-term accountability. Key issues include auditability, as "the model decided based on retrieved context" is insufficient explanation. Costs also escalate rapidly with volume when every case requires an LLM call. Furthermore, LLMs can be inconsistent on straightforward cases that should have deterministic outcomes.
The cascade architecture addresses this by using the LLM as an escalation path. Stage one involves deterministic methods: resolving clear rules and exact matches without any model calls. This stage is expected to handle the majority of cases, often more than half. Stage two employs retrieval: for cases not resolved deterministically, it fetches relevant evidence. Only the remaining cases, the genuine residue that stages one and two could not resolve, are sent to the LLM in stage three. This final stage processes a small fraction of cases (around 10-15%), significantly reducing costs and improving consistency.
The article also emphasizes designing prompts for "asymmetric risk," instructing the LLM to treat uncertainty as a reason for escalation and explicitly considering the consequences of different error types. This contrasts with neutral prompts that treat all errors equally. Evaluation metrics also need adaptation, with separate measurements for retrieval quality and final classification, and evaluation sets oversampling difficult cases that reach the final LLM stage.