FirmGround Agentic RAG

This page follows one real question from start to finish. It comes from a library of audit standards. Section 4 then measures the same system on financial filings.

Midyear, the team concludes that controls over revenue are ineffective. The manager says, “We’re testing 100% of revenue transactions over $50K, so we can lower our assessed risk of material misstatement for revenue.” Evaluate this statement.

This question is harder than it looks, for two reasons:

• The answer is spread across several standards. How audit risk is defined (AS 1101), how auditors respond to assessed risk (AS 2301) and how they evaluate audit results (AS 2810) are separate documents. A system that finds one relevant passage and stops will have only part of the answer.
• The manager’s claim sounds reasonable. More testing feels like less risk. A useful answer has to separate what testing changes from what it doesn’t, and point to the paragraph that says so.

FirmGround does not answer in a single step. It runs as an agent: a language model that decides what to do next, uses tools such as search and a code runner, reads the results, and keeps going until it has enough to answer.

1. It searches each standard separately

   The run took 18 steps. Instead of one broad search, the agent searched the library several times, one topic at a time. Two of those searches are below, on AS 2810 and AS 2301. Other searches in the same run found AS 1101.

   

2. It gives a verdict, with reasons

   The manager has it backwards. Weak controls push the company’s side of the equation up, and testing only pulls the auditor’s side down. So the assessed risk should go up, and the extra testing is the right response to that, not a reason to lower it. Every claim carries a numbered citation.

   

3. Every citation opens its source

   Citation 28 opens AS 1101 at page 15, with paragraph .04 outlined. It is the sentence the answer rests on: audit risk is a function of the risk of material misstatement and detection risk.

   

Each step on its own is simple. What matters is the loop: the agent checks whether it has enough evidence before it answers, and keeps searching if it does not.

The question path

The compute step only runs when a question needs arithmetic, such as FinanceBench’s calculation questions in Section 4. The agent writes a short calculation and runs it in a sandbox, an isolated computer that runs its code without touching anything else, instead of doing the arithmetic in its head. The audit question above needed none.

At ingestion, every document is split into small passages for matching. Each passage keeps a link to the larger section it belongs to, so the agent reads results with their surrounding context and section headings. Search then runs in three stages:

• Two kinds of search run side by side. Keyword search (BM25, the ranking method behind most search engines) is good at exact terms such as a standard number or a line-item name. Semantic search compares meaning, so it finds relevant passages even when the wording differs from the question.
• The two result lists are merged by rank. The two methods score passages on unrelated scales, so their scores can’t be added. Reciprocal rank fusion (RRF) merges the lists by each passage’s position in each.
• A reranker makes the final cut. A Qwen3 reranker reads the question with each candidate passage, scores how well it answers, and the best few go back to the agent.

Search only covers documents the user is allowed to see, and every result carries a link back to its place in the original.

Ingestion and streaming

Uploading a 150-page PDF shouldn’t block the page. The upload returns at once. A background worker parses the PDF into text and tables, splits it into passages, computes their numeric representations and adds them to the search index.

FastAPI

The web API. Everything runs asynchronously, so one process can wait on many slow operations at once.

Taskiq

Background jobs, built for asynchronous Python. Runs both ingestion and the agent itself.

PostgreSQL

The system of record for documents, versions, permissions and conversation history.

Redis Streams

The job queue, and the live feed of each answer while the agent works.

• No lost uploads. Every upload is recorded in the database before it is queued. If a job is lost, a scheduler finds it and sends it again.
• Answers survive a refresh. Answers stream to the browser as the agent works. If the connection drops, the page reconnects and picks up from the last event it received, and the agent keeps working in the meantime.

Accuracy on FinanceBench

FinanceBench is a public benchmark for financial question answering, published by Patronus AI. Its open sample has 150 questions over 84 real filings: 10-Ks (annual reports), 10-Qs (quarterly reports) and earnings releases (the short results announcements companies publish each quarter) from companies such as 3M, Amazon, Boeing and JPMorgan. The questions come in three groups of 50: questions an analyst would ask, figures that must be calculated from line items, and open-ended questions written for the dataset.

I loaded all 84 filings into one shared library, so every question searches the whole corpus instead of being handed the right document. While checking the reference answers, I found four that contradicted their source filings. I corrected those four from the documents and labelled them as corrected.

I compared three systems. Naive RAG runs one search, then answers from whatever came back, using the same language model and embedding model as FirmGround, so the gap measures the agent loop, not a better model. A closed-book model answers from memory with no documents, as a floor. A refused question counts as a failure.

Two things stand out. First, naive RAG scores below the closed-book model. When it did answer, it was right about 70% of the time. Its problem is that it refused 104 of 150 questions, because a single search usually didn’t bring back enough evidence.

Second, the gap is widest on calculation questions, which need several line items and arithmetic. On those, FirmGround scored 88% and naive RAG 2%. FirmGround’s weakest group was the open-ended questions, at 74%.

Ingestion speed and memory

I uploaded four Chinese history textbooks at once, 585 pages and 133 MiB of PDF, and timed each until it was searchable, on a single background worker running on a laptop.

Throughput

372 pages/min

All four books searchable after 94 s. Median book ready in 58 s.

Peak memory

~650 MiB

Whole worker container (644 MiB measured). The worker process peaked at 449 MiB.

Most of each book’s time, 79–90%, went to parsing the PDF on an external document-parsing service. Splitting, embedding and indexing took about ten seconds per book. The largest file (74 MiB) is parsed in three parts, one after another. I chose that to keep memory bounded. Parsing the parts in parallel would be faster, at the cost of higher peak memory.

This was a single run on one machine. Treat it as a measured data point, not a capacity guarantee.

Nothing in the system is specific to one field. The same pipeline has been used on very different material.

Audit. Besides the standards library above, an IPO audit scenario keeps two libraries apart: PCAOB and SEC standards in one, a fictional client’s filing materials in the other. The agent cites both, so a reviewer can check the rule and the evidence side by side.

Education, in use. FirmGround is piloting at a public secondary school in Tianjin, where teachers use it as a knowledge base for history.

For an analyst desk, the next step is a tool that compares a company’s new filing with the previous one, flags changes in risk-factor language and guidance, and cites every flag in both documents. Most of the pieces already exist:

• Sections line up. Ingestion already keeps each passage’s section heading, so this quarter’s “Risk Factors” can be matched against last quarter’s.
• Real changes, not rewording. A section-by-section comparison finds passages that were added, removed or reworded. The agent sorts them into real changes, such as a new risk, a softened forecast or a changed number, and routine rewording.
• Two citations per flag. The old passage and the new one, so the analyst can confirm or dismiss it in one view.
• Measured before anyone relies on it. Build a set of filing pairs with known changes, then measure how many the tool catches and how many of its flags are noise.

The output is a review list for an analyst to confirm, not an automatic alert. The desk keeps the judgement, and the tool does the reading.