A 32-page case study went through my document pipeline. It filed a clean summary and skipped all 17 exhibits, the pages with the financial tables. What I changed, and what I learned testing vision models on numbers.

I fed a 32-page case study into my document pipeline. It filed a clean summary and moved on.

In reality -> It had skipped all 17 exhibits, ALL the pages with the FINANCIAL TABLES !

Nothing flagged the gap. The summary read well, and a summary that reads well hides what is missing.

The cause was ordinary. The scheduled job could not find the PDF tools it needed, and the prompt never required it to read every page.

I made two changes.

(1) The pipeline now pulls the text layer and renders every page as an image.

(2) Every filed page must also report how many pages it read out of how many it received.

The same case now comes back 32 of 32, and the exhibits match a version I checked by hand.

Then I asked the harder question - >
When there is no text layer, as with a scan/screenshot, can a model read numbers exactly?

I tested LOCAL VISION models on real pages.

On clean statements they read at or near 100 percent. On a 16-column income statement with 141 numbers, every model I tested put 0 to 2 wrong digits into each read, with no warning. One model misread the same digit the same way on every run. Higher resolution did not fix it.

Four checks did work:

(1) Read the text layer first. Many PDFs have one, and it is exact.

(2) Run two different (LLM) models on the same item. They missed different numbers, so a disagreement flagged the problem.

(3) Check the arithmetic in code. A running-balance check caught all 3 errors one model made on scanned statements.

(4) Send wide tables to a person.

My rule for diligence work is now simple. I NEVER accept an AI summary without a COUNT of what it read and a CHECK on the NUMBERS it pulled.

If you use AI on data rooms or financial statements, what CHECKS do you run?

This perspective was originally published on LinkedIn. Join the discussion, add your thoughts, and follow for regular updates.

View on LinkedIn