Financial filings punish plausible language
A model can write an excellent explanation of a company and still be wrong in the only way that matters: the claim is not supported by the filing. Financial documents contain repeated terms, shifting definitions, adjusted metrics, footnotes, and tables that require context. A fluent answer can quietly merge facts from different periods or turn management guidance into an observed result.
That is why my preferred contract for a filing assistant is strict: quote the source or say that the evidence was not found. The citation is not decoration added after generation. It is part of the answer itself. The system should retrieve narrow passages, keep page and section metadata, and ask the model to make claims only from that packet.
Local models change the engineering trade-offs
Running a model locally can improve privacy, cost control, and iteration speed, but it also removes the comfort of assuming a larger hosted model will rescue weak retrieval. Chunking, table extraction, context selection, and prompt structure become visible engineering decisions. Smaller models often perform well when the task is bounded and the evidence is clean; they fail loudly when context is noisy or the question requires joining distant parts of a filing.
A useful pipeline separates ingestion, retrieval, answer generation, and verification. Each stage should emit artifacts that can be inspected. When an answer is wrong, I want to know whether the passage was never retrieved, whether the model ignored it, or whether the question itself was ambiguous. Without that separation, debugging becomes prompt superstition.
Evaluation is the product discipline
The evaluation set should look like the questions an investor actually asks: revenue concentration, changes in margins, debt covenants, segment performance, dilution, customer risk, and management claims. Each question needs an expected answer, accepted supporting quotes, and rules for abstention. Then the system can be scored on retrieval, factual support, citation quality, and whether it correctly refuses unsupported questions.
The surprising benefit is product clarity. Once failures are categorized, the next feature becomes obvious. Weak retrieval suggests better indexing. Correct facts with poor citations suggest better source alignment. Confident unsupported claims require a stronger abstention policy. An evaluation harness turns reliability from a feeling into a backlog. That is how a filing demo becomes a research tool someone can trust.