Shōhyō (証憑) — verified ground truth for Japanese invoice, receipt & payment notice extraction

Shōhyō (証憑 — the accounting term for source documents such as invoices and receipts) is a labeled evaluation dataset for measuring how well a document-to-JSON extractor performs on Japanese documents. The Japanese-specific logic — era dates, reduced (8%) versus standard (10%) tax, withholding, qualified-invoice (インボイス) rules, revenue stamps — is arithmetically verified on the answer side.

Building an extraction pipeline always hits the same wall: how do you measure accuracy? Real invoices can’t be used for evaluation because of PII, and hand-authoring gold JSON one record at a time doesn’t scale. Shōhyō stands in for that ground truth.

What it measures / doesn’t

It scores extraction correctness on clean text input: date normalization (era to Gregorian, relative dates), the reduced-vs-standard tax split, withholding amounts, revenue stamp, and full-width / symbol normalization — the spots where Japanese documents tend to trip extractors up.

It does not score OCR or layout analysis (image to text). The input is a document_text field, so accuracy at reading paper or scans is out of scope.

Contents

• 2,000 invoices / 2,000 receipts / 2,000 payment notices in the full set (6,000 records) (the free sample is 45 records). All synthetic — no real companies or personal data.
• Each record pairs document_text (a realistic body) with expected_output (gold JSON) plus difficulty and edge-case tags.
• A JSON Schema, a validator, and a scorer are included. The scorer breaks your extractor’s accuracy down by edge case rather than handing you one flat number.

Payment notices invert the structure (added 2026-08)

A payment notice (支払通知書 / 仕入明細書) is issued by the buyer, so the roles swap relative to an invoice. issuer is the payer (buyer), payee is the recipient of the payment (seller), and the registration number belongs to the seller — it is required when the notice is retained as a purchase statement. An extractor that confuses the two fails here.

Edge cases specific to this document type:

• Aggregating several invoices — one line item corresponds to one invoice and carries its invoice_number
• Offsets, discounts and transfer fees (deductions) — the payable is tax-inclusive total − withholding − total deductions
• Closing date vs scheduled payment date — two dates, easy to pick the wrong one

How it’s built (the answer is fixed first, then the document is rendered)

Synthetic data is usually built in one order: write the document body, then read the answer back out of it. That order lets reading errors slip in. Shōhyō is built the other way around.

1. Fix the answer — amounts, tax-rate buckets, tax per bucket (floor / round / ceil), withholding, and revenue stamp duty are all determined first.
2. Render the document — document_text is drawn from those fixed values. Notation variance (Japanese era years, full-width digits, ￥ vs 円, relative due dates) is introduced deliberately at this step.

Because the answer exists first, the body and the answer cannot disagree by construction. On top of that, every record goes through a machine check — schema conformance plus full arithmetic recomputation (line item = quantity × unit price, per-bucket sums, subtotal and tax-inclusive totals, withholding, revenue stamp duty). All 6,000 records in the current body pass.

Generation is deterministic with a fixed seed, so any given release reproduces byte-for-byte.

When to use it

• Regression testing an in-house extractor — when you swap a model or a prompt, watch which edge-case score moved. “Only the reduced-tax split got worse” points you straight at the cause.
• A benchmark for accounting / finance SaaS — measure your extraction accuracy against a shared yardstick that doesn’t depend on your own data, where in-house evaluation tends to grade itself too kindly.
• CI for an LLM app or agent — drop the scorer into your pipeline and gate releases on an extraction-accuracy threshold.

Compared to rolling your own

Authoring ground truth yourself runs into the PII wall on real invoices, and even synthetic data leaves the question no one checked: is the answer actually correct? Withholding and the reduced-tax split in particular are easy to get wrong by hand.

What Shōhyō adds is an answer side that is fixed before the document is written, then recomputed arithmetically for every record. Because it stays on clean text input, your score doesn’t wobble with OCR quality — you see the accuracy of the extraction logic itself.

Get it

Start with the free sample to see what the data is actually like, then pick the license that covers your use.

Free sample (Hugging Face)

20 invoices / 10 receipts / 15 payment notices, frozen. Built by the same generator and against the same schema as the full set, so you can check what edge cases are included and how granular the answers are before paying. CC BY-NC 4.0 (non-commercial, evaluation use). → dataset page

Paid (6,000 records across 3 document types)

All plans include validate.py (full arithmetic recomputation) and score.py (per-edge-case scoring).

The annual license promises exactly three things. It does not promise a fixed monthly or quarterly cadence. Tax rules don’t change every month, and adding a new edge case takes design work — committing to a cadence we can’t keep would be the dishonest option.

• Every document type added during your term, at no extra cost (payment notices were added in 2026-08; delivery notes and contracts are next, timing not guaranteed)
• Schema and data follow any tax or format amendment within 90 days
• Existing document types grow in volume and edge-case coverage (at least once a year)

Build notes will follow on the Build section.

For evaluating extraction accuracy only. Not tax or accounting advice. Tax-rate buckets and the like are examples under specific assumptions; confirm real-world judgments with a professional.