Close Menu
Digital Connect Mag
    Facebook X (Twitter) Instagram
    • About
    • Meet Our Team
    • Write for Us
    • Advertise
    • Contact Us
    Digital Connect Mag
    • Websites
      • Free Movie Streaming Sites
      • Best Anime Sites
      • Best Manga Sites
      • Free Sports Streaming Sites
      • Torrents & Proxies
    • News
    • Blog
      • Fintech
    • IP Address
    • How To
      • Activation
    • Social Media
    • Gaming
      • Classroom Games
    • Software
      • Apps
    • Business
      • Crypto
      • Finance
    • AI
    Digital Connect Mag
    Business

    Notes From Integrating a Data Extraction SDK Into Our Invoice Flow

    Anthony BergsBy Anthony BergsNovember 17, 20268 Mins Read

    One week of API, two weeks of documents, and the threshold design that made the rest work.

    There was a spreadsheet. Four hundred invoices a month, keyed in by hand by two people who deserved better, and an error rate that climbed from Monday to Friday the way boredom-driven error rates do.

    We wired a data extraction SDK to our upload path in an afternoon. These notes are about the three weeks that followed.

    “Integrating a data extraction SDK into our invoice flow took three weeks: one for the happy path, two for reality (skewed scans, multi-page invoices, and deciding what to do with 80 percent confidence fields). The system now processes 2,000 invoices a month with 6 percent routed to human review.”

    Extraction integration is one week of API and two weeks of documents behaving like documents. Plan for the ratio, not the demo.

    Key Takeaways

    • Budget one week for the happy path and two for real documents; that ratio held for us and has held since.
    • Skewed scans were the largest single accuracy problem until preprocessing joined the pipeline.
    • Confidence thresholds belong per field, set by the cost of a wrong value rather than by one global number.
    • Fields below their threshold route to a human review queue, which for us means 6 percent of invoices.
    • Upload and extraction in one pipeline removed a handoff between two services we had planned to build.

    The Retyping Problem

    The manual baseline was about twenty-seven hours a month, and the hours were the least of it. The errors had shape.

    Transposed digits in totals. Dates read as US format when they were written in European format, or the reverse.

    Line items silently dropped when an invoice ran past one page. Supplier names spelled three different ways, which quietly created three suppliers in our ledger.

    What justified the project was not the hours or even the errors. It was that both people doing the work had started asking, reasonably, whether this was a job for a person in this decade.

    Week 1, The Happy Path

    We wired the data extraction SDK to our upload path in an afternoon, and the real work came later. A clean single-page PDF went in, structured JSON came out, and the fields were right.

    By the end of that week we had a working demo, a proud Slack message, and a completely false sense of how far along we were.

    This is the part that looks like the answer when you go looking for the best way to add OCR and text extraction to uploaded PDFs.

    The mechanism is genuinely easy now, which is exactly why it is the wrong thing to plan around. The demo works on the first day because the demo uses the documents you chose.

    One thing from week one did turn out to matter later. Because extraction happened in the same pipeline as the upload rather than in a separate service downstream, there was no handoff to build: no queue between our uploader and our extractor, no retry semantics between them, no second place for a document to get lost. We had budgeted a fortnight for exactly that plumbing. It never needed writing.

    Week 2, Reality Arrives

    Then we pointed it at a month of real invoices, and reality arrived in three waves.

    Skew was the worst of them. Suppliers photograph invoices on phones, at an angle, on a desk, under an office light, and a page tilted by four degrees is measurably harder to read than the same page straight.

    This is not a surprising finding — OCR accuracy has always depended heavily on image quality, and de-skewing is a standard preprocessing step for exactly this reason. It was a surprise to us only because every document in our test set had been a clean export.

    Second wave: multi-page. A twelve-page invoice is normal in our world, and page one is frequently a cover note, a delivery docket, or a compliments slip from a supplier who likes compliments slips.

    Extracting from page one gave us confident, well-structured nonsense. We needed a step that finds the invoice inside the document before anything tries to read it.

    Third wave: the format zoo. PDF, JPEG, TIFF, HEIC from newer iPhones, and one supplier who consistently sends a Word document containing a photograph of an invoice.

    Asking what can convert and optimize documents after upload stopped being an idle question and became the second stage of our pipeline: normalize everything to one working format before any decision depends on it.

    Pipeline notes from integrating a data extraction SDK into an invoice flow.
    Three stages were added in week two, all of them before extraction ever runs.

    Week 3, The Confidence Question

    Week three was one question. What do you do with a field the extractor returns at 80 percent confidence?

    Our first answer was a single global threshold, wrong in both directions at once. Set it high and half the invoices went to review, which is no automation. Set it low and a wrong total reached the ledger, which is worse than typing it by hand.

    The design that worked treats confidence as a routing decision rather than an error, and sets the bar per field according to what a wrong value actually costs.

    This is the precision and recall trade in its most concrete form: every threshold you move buys accuracy on one side and work on the other, and the correct place to stand is different for a total than for a supplier name.

    Per-field confidence thresholds and the share of invoices auto-accepted
    One global number was the first thing we tried and the first thing we discarded.

    The total carries the highest bar at 0.98, because a wrong total is the expensive mistake and we would rather look at it.

    The supplier name carries the lowest at 0.85, because a fuzzy match against our existing supplier list rescues almost every weak read.

    Tax has arithmetic as a second opinion. An invoice routes to review if any single field falls under its own bar, and that rule is what produces the 6 percent.

    Steady State and the Pipeline Ambition

    The system now handles about 2,000 invoices a month, which is five times the volume the two people were keying, because automation let us onboard suppliers we had previously turned away as too much administrative work.

    Six percent land in the review queue. A person clears that queue in under an hour a day, and the job is checking rather than typing.

    The queue is the part I would defend hardest. Automation that silently guesses is not trustworthy, and nobody in finance would have signed off on it. Automation that says plainly which fields it was unsure about, and shows them to a human, is something a finance team will actually adopt, because it fails visibly.

    What is next is less an extraction question than a pipeline one. Once documents flow through a chain of swappable stages, the interesting problem becomes composing complex file-processing pipelines rather than tuning any single step.

    We have four stages that did not exist eight months ago, and adding a fifth is configuration rather than a project.

    Notes for the Next Team

    • Do not evaluate on clean exports. Test on the ugliest fifty documents you can find on day one, because those are the ones that set the schedule.
    • Preprocessing is part of extraction. Deskew, de-noise and normalize the format before anything tries to read the page.
    • Page one is not the invoice. Find the document inside the document first, or you will extract confident nonsense.
    • Thresholds go per field, set by the cost of being wrong. We wasted three days on a single global number that could not work.
    • Build the review queue before you need it. It is what makes the automation adoptable, and it is not a fallback.

    One Week of API, Two Weeks of Documents

    Three weeks, in a ratio worth planning around: one for the API and two for the documents. The API part is genuinely easy now, and that ease is a trap, because it produces a demo on day two and an estimate based on the demo.

    The threshold design is the part I would hand to another team unchanged. Confidence per field, bars set by the cost of a wrong value, anything below its bar routed to a person, and the whole thing measured against invoices you have already keyed by hand so you know what each bar would have caught.

    The review queue is not the compromise in that design. It is the feature that made the rest of it trustworthy enough to turn on.

    FAQs

    How long does extraction integration take?

    For us, three weeks: one for the happy path on clean PDFs, and two for real documents, which meant skew, multi-page files, mixed formats and threshold design.

    What do you do with low-confidence fields?

    Route them to a human review queue rather than treating them as errors. Ours reviews about 6 percent of invoices, and an invoice goes to review if any single field falls below its own threshold.

    What hurt accuracy most?

    Skewed scans, by a wide margin, until deskewing and cleanup joined the pipeline ahead of extraction. Phone photographs of paper invoices are the normal case, not the edge case.

    Should confidence thresholds be one number or several?

    Several. A wrong total costs more than a wrong supplier name, so the total earns a higher bar and the supplier name a lower one, especially where a fuzzy match against known suppliers can rescue a weak read.

    Anthony Bergs

    Anthony Bergs is the CMO at a writing services company, Writers Per Hour. A certified inbound marketer with a strong background in implementation of complex marketing strategies.

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    Address: 330, Soi Rama 16, Bangklo, Bangkholaem,
    Bangkok 10120, Thailand

    • Home
    • About
    • Contact Us
    • Write For Us
    • Sitemap

    Type above and press Enter to search. Press Esc to cancel.