AI Engineering · 2-Week Proof of Concept
2 weeks PoC · 8× faster appeal drafting · 90%+ first-pass accuracy
A 2-week PoC sprint that built a RAG pipeline against payer policy documents — automatically generating grounded, first-pass insurance appeal letters from denied claims and patient records. Greenlit for production on day 14.
Challenge
Every denied claim required a billing specialist to manually investigate the denial reason, locate the relevant payer policy, and draft an appeal letter from scratch — before any actual appeal work had begun. The process took hours per claim and was the primary throughput constraint on the revenue cycle team.
Approach
A 14-day PoC sprint structured in three phases: discovery and data access (D1–3), rapid prototyping of the RAG pipeline (D4–10), and validation and benchmarking against real historical appeals (D11–12), culminating in a results presentation and production roadmap on D13–14.
Outcome
A working pipeline that parses denial notices, retrieves relevant payer policy sections, and generates a complete, grounded appeal letter ready for human review. 90%+ first-pass accuracy on the validation set. Greenlit for full production development on the day of the results presentation.
The background
The hospital's billing team processed hundreds of claim denials monthly. Each denial arrived with a denial code and a brief reason — but turning that into an effective appeal required a billing specialist to understand the denial type, locate the relevant section of the payer's coverage policy, match it against the patient's clinical documentation, and write a letter that cited specific policy language in the context of the specific patient case. This was not a templating problem. The payer policies were long, frequently updated, and different for each insurer. A billing specialist doing this work by hand was spending two to four hours per appeal before writing a single word of the letter itself.
The hospital had looked at commercial appeal automation tools but found them too rigid — they generated letters from fixed templates that didn't cite the specific policy clauses relevant to each denial. The appeals team needed something that could reason about payer policy in the context of a specific patient record, not fill in a form. The question was whether RAG could do that — and whether it could be demonstrated quickly enough to justify the investment in finding out.
The challenge
The first three days were spent mapping the actual data landscape: what denial types were most common, which payers generated the most denials, where the payer policy documents were stored and in what format, and what clinical documentation was available to support appeals. This produced the scope for the prototype — two payer types, three denial categories — narrow enough to validate in ten days, representative enough to be meaningful.
A RAG pipeline built on LangGraph with Qdrant as the vector store. Payer policy documents were chunked, embedded, and indexed. The pipeline parsed the denial notice to extract the denial code, denial reason, and relevant patient identifiers; retrieved the top-k policy sections from Qdrant; and fed the retrieved policy text along with the patient record summary into GPT-4o with a structured prompt that required the output to cite specific policy language. The appeal letter was generated as a structured document ready for human review.
The prototype was run against 40 historical denials where the outcome was known — successful appeals that the billing team had won manually. Generated letters were evaluated against the winning human-written letters on three dimensions: policy citation accuracy (did it cite the right clause?), clinical grounding (did it correctly use the patient record?), and completeness (did it address the specific denial reason?). 90%+ of the generated letters were assessed as acceptable first drafts by the billing team lead.
The final two days were a structured presentation of results — benchmark numbers, example outputs, failure analysis, and a proposed production roadmap. The roadmap covered the additional payer types to be indexed, the human review and approval workflow, PIPEDA compliance requirements for handling patient data in the pipeline, and a phased rollout plan. The hospital approved the production investment on day 14.
Architecture
Results