The result: 31.3% geometric mean throughput improvement across all API workloads (n=6, p=0.002 for every number-dense comparison), with zero conformance regressions. The performance gain concentrates where the algorithm change predicts it should (number-dense payloads), and disappears where it should (string-dominant payloads).

What Burger-Dybvig Actually Costs

The Burger-Dybvig implementation from Part 1 does four things per number. It initializes four big.Int values (R, S, M+, M-). It scales them by a power of 10 from a 700-entry precomputed cache. It extracts digits one at a time via big.Int division. It terminates when boundary conditions indicate the shortest representation. Every operation in that pipeline touches the heap. big.Int arithmetic allocates on multiply, on division, on comparison when the operand exceeds one machine word.

The CPU profile for the Burger-Dybvig path shows where math/big time goes:

2.70s  1.32%  math/big.mulAddVWW
2.33s  1.14%  math/big.nat.mulAddWW
2.28s  1.12%  math/big.(*Int).mul
1.80s  0.88%  math/big.nat.norm
1.71s  0.84%  math/big.nat.mul

That is 10.82s of CPU time in math/big during a benchmark run covering 19 workload categories (total sample time 204.18s). The memory profile adds detail. jcsfloat.FormatDouble accounts for 2.91% of cumulative heap allocation. That breaks down through the call chain: generateDigits (2.01% cumulative), which calls scaleByPower10 → pow10Big (1.29% flat). The pow10Big function returns a defensive new(big.Int).Set(cached) copy on every call to prevent callers from corrupting the cache. That defensive copy is correct. It also means every FormatDouble invocation allocates at least one big.Int just to look up a power of 10, before any digit extraction begins.

The per-call allocation profile from the standalone jcsfloat benchmarks in Part 1:

Four to six allocations per number. For a JSON document with 2,048 numbers in an array, that is 8,000 to 12,000 allocations spent solely on converting numbers to strings.

Schubfach: The Same Problem, Different Arithmetic

Schubfach solves the same problem as Burger-Dybvig: given an IEEE 754 binary64 value, find the shortest decimal string that round-trips. The difference is how it computes the answer.

Where Burger-Dybvig represents the value and its boundaries as ratios of arbitrary-precision integers and iterates to extract digits, Schubfach works entirely in fixed-width integer arithmetic. The core operation is a 64x64-to-128-bit multiplication of the float’s significand against a precomputed power-of-10 constant, followed by a shift and comparison against interval bounds that are themselves fixed-width. No heap allocation in the digit generation path. No math/big. No defensive copies of cached values.

Go does not have a native uint128 type. The 128-bit multiplication uses math/bits.Mul64, which returns a (hi, lo) pair:

hi, lo := bits.Mul64(significand, pow10TableHi[index])

The upper 64 bits contain the information needed to determine the digit string. The lower 64 bits determine rounding. Carry propagation through the 128-bit product must be exact. A single bit error in the table or the multiplication produces wrong output for specific IEEE 754 values that may not surface during random testing. The oracle vectors catch these errors; random fuzzing alone does not reliably reach the affected bit patterns.

The precomputed table replaces Burger-Dybvig’s 700-entry *big.Int cache with a fixed-size array of [2]uint64 pairs (696 entries, covering exponents -348 to 347). Where pow10Big returns a heap-allocated defensive copy on every call, the Schubfach table is read-only after initialization and accessed by index with no allocation.

The ECMA-262 formatting stage (formatECMA and its four branch helpers from Part 1) is shared between implementations. That code takes (negative bool, digits string, n int) and produces the final string. It is algorithm-independent and unchanged.

The Tie-Breaking Problem

Part 1 described ECMA-262 even-digit tie-breaking as “the most subtle part of the algorithm.” That description applies to any shortest-round-trip algorithm, not just Burger-Dybvig.

In Burger-Dybvig, tie-breaking is explicit. The midpointDigit function compares 2R against S using exact big.Int arithmetic. When 2R == S, the digit d is checked: if even, keep it; if odd, round up. The invariants are visible in the code because the arithmetic is exact and the comparison is direct.

// Burger-Dybvig: exact midpoint detection
func midpointDigit(d int, state *digitState) byte {
    state.scratch3.Lsh(&state.r, 1)
    cmp := state.scratch3.Cmp(&state.s)
    if cmp < 0 {
        return byte('0' + d)
    }
    if cmp > 0 {
        return byte('0' + d + 1)
    }
    if d%2 == 0 {
        return byte('0' + d)
    }
    return byte('0' + d + 1)
}

In Schubfach, the midpoint condition is encoded in the interval bounds. The algorithm computes whether the value falls strictly inside, at the boundary of, or outside the interval of valid shortest representations. Modifying the boundary comparison to prefer even digits requires understanding where in the interval arithmetic the rounding decision is made, and adjusting the comparison predicates accordingly.

The adjustment is small in code. A few comparison operators change from strict to inclusive (or vice versa) depending on the parity of the significand’s least significant bit. But the reasoning behind which operators to change requires following the proof in Giulietti’s paper to understand which invariant each comparison maintains.

The even-digit preference also influences the boundary comparisons themselves. This is the same pattern described in Part 1 for Burger-Dybvig’s cmpRoundDown and cmpHigh functions, where isEven switches between inclusive and strict comparisons:

// Burger-Dybvig boundary comparison (from Part 1)
func cmpRoundDown(lhs, rhs *big.Int, isEven bool) bool {
    if isEven {
        return lhs.Cmp(rhs) <= 0   // Inclusive at boundary
    }
    return lhs.Cmp(rhs) < 0        // Strict
}

Schubfach has an analogous structure, but the comparison operates on 64-bit values derived from the 128-bit product rather than on big.Int ratios. The oracle vectors serve as the correctness proof. Any tie-breaking error produces a different digit string for at least one value in the 286,362-vector dataset. The oracle test fails with the exact bit pattern that triggered the divergence.

Benchmark Methodology

Comparing two implementations of the same specification requires more than go test -bench. Both binaries must produce byte-identical output for every input. The performance comparison must separate the number formatting cost from everything else in the pipeline: parsing, string escaping, UTF-16 key sorting, memory allocation for the value tree.

The benchmark lab runs both implementations against identical workloads at three levels:

1. API benchmarks via Go’s testing.B with -count=6, measuring Canonicalize() and Verify() calls directly. benchstat requires a minimum of 6 samples per comparison to compute confidence intervals at the 0.95 level.
2. CLI end-to-end timings with process startup, file I/O, and argument parsing included (9 runs per workload per implementation, 2 warmup runs discarded)
3. Conformance verification with SHA-256 digest comparison on every output, differential fuzzing (1,000 random cases), and determinism checks (identical output across repeated runs)

The 19 workload categories span the space intentionally. Some are dominated by number formatting (number-heavy, numeric-boundary). Some have zero numbers (long-string, control-escapes, surrogate-pair). Some mix both (small, medium, mixed-prod, nested-mixed). The zero-number workloads serve as a control group: any performance change on those workloads would indicate the algorithm swap affected code paths it should not have touched.

The environment: Linux 6.8, Go 1.25.6, 12th Gen Intel i7-12700K (20 logical cores), CPU governor performance. Both implementations run under the same conditions on the same machine in the same benchmark invocation.

Conformance Evidence

Before any performance discussion, the gates:

Additional evidence:

• SHA-256 digests of canonical output are byte-identical between implementations for every valid workload. Not equivalent. Identical.
• 1,000 differential fuzz cases with randomized seeds. Zero divergences.
• Zero oracle mismatches. Zero determinism failures. Zero invalid-input parity issues (both implementations reject the same inputs with the same failure classes).

The Schubfach implementation produces the same bytes as Burger-Dybvig for every input tested. The remainder of this article is about cost, not correctness.

API Benchmarks: Number-Dense Workloads

These are workloads where FormatDouble is invoked frequently enough to dominate the profile. All results are means of 6 samples:

Every comparison reaches statistical significance at p=0.002. The speedups range from 1.31x (mixed-prod) to 1.71x (medium).

The allocation column is the structural signature. number-heavy drops from 80 to 60 allocations (25% reduction). numeric-boundary drops from 52 to 38 (27% reduction). array-2048 drops from 57,341 to 49,135 (14% reduction). Each eliminated allocation corresponds to a math/big operation that Schubfach replaces with fixed-width arithmetic.

API Benchmarks: String-Dominant Workloads (Control Group)

These workloads contain few or no numbers. The number formatter is not the bottleneck:

Allocation counts are identical across all seven workloads. The number formatter was not invoked (or invoked for a trivial count of values). Burger-Dybvig wins on control-escapes (6%), long-string (2%), rfc-key-sorting (2%), and surrogate-pair (1%). These are small effects. deep, deep-64, and unicode show no statistically significant difference.

This is the control group. Identical allocation counts confirm that the Schubfach change is isolated to the number formatting path. The small Burger-Dybvig advantages on string workloads are consistent with binary layout, code alignment, or instruction cache effects from the different compilation units.

API Benchmarks: Large Payloads

The large workload (approximately 460KB of JSON) shows the largest absolute time savings:

Schubfach is 72-74% faster across all three modes. The allocation reduction (204,776 vs 237,568, a 14% drop) understates the effect: the eliminated allocations are math/big operations that carry CPU cost beyond their memory footprint. At this payload size the math/big overhead accumulates to approximately 8.6ms of additional wall-clock time per canonicalization.

Memory Profile Comparison

The memory profile is where the implementation difference is most directly visible.

Burger-Dybvig, jcsfloat allocation footprint (cumulative, showing call chain):

jcsfloat.FormatDouble             2.91% cum
  └─ jcsfloat.generateDigits     2.01% cum
       └─ jcsfloat.pow10Big      1.29% flat

The cumulative values overlap: FormatDouble’s 2.91% includes generateDigits’s 2.01%, which includes pow10Big’s 1.29%. The flat allocation in pow10Big represents the defensive new(big.Int).Set() copies.

Schubfach, jcsfloat allocation footprint:

jcsfloat.FormatDouble             1.16% cum
  └─ jcsfloat.appendIntegerFixed  0.57% flat
  └─ jcsfloat.formatECMA         0.20% flat

The FormatDouble cumulative drops from 2.91% (Burger-Dybvig) to 1.16% (Schubfach). The remaining allocations are in appendIntegerFixed (byte slice growth for trailing zeros) and formatECMA (string building). Both are shared formatting concerns, not digit generation.

The math/big entries are gone entirely. No pow10Big. No nat.make. No nat.set. No generateDigits (which in Burger-Dybvig performs big.Int division in a loop). The 700-entry *big.Int power-of-10 cache is eliminated. The digitState pool with its nine big.Int fields (r, s, mPlus, mMinus, scratch1, scratch2, scratch3, quot, rem) is eliminated.

The CPU profile shows the same pattern from the time axis. The math/big functions that appear in the Burger-Dybvig profile do not appear anywhere in the Schubfach profile:

They are replaced by math/bits.Mul64, which the Go compiler inlines. It does not appear as a named function in the profile because it executes as a single MULQ instruction on x86-64.

What the Profiles Reveal About the Pipeline

The top application-level functions in both CPU profiles are structurally identical:

Schubfach:
  jcs.serializeString             7.60s   3.73%
  jcstoken.(*parser).parseString  6.30s   3.09%
  jcs.validateValueTree           2.73s   1.34%
  jcs.serializeObject             2.60s   1.28%

Burger-Dybvig:
  jcs.serializeString             6.97s   3.41%
  jcstoken.(*parser).parseString  5.25s   2.57%
  jcs.validateValueTree           2.64s   1.29%
  jcs.serializeObject             2.48s   1.21%

(Both profiles are dominated by runtime functions above these: strconv.leftShift, runtime.nextFreeFast, runtime.memclrNoHeapPointers, and aeshashbody occupy the top positions in both builds. The application-level functions listed here appear below them.)

String serialization, string parsing, value tree validation, and object serialization (which includes UTF-16 key sorting via sort.Slice) dominate the application-level profile in both builds. These are the same code in both builds. Replacing the number formatter moved it off the critical path for number-dense payloads, but the pipeline bottleneck for general workloads was never number formatting. It was always parsing and serialization.

The Schubfach profile shows higher absolute times for these functions (7.60s vs 6.97s for serializeString) because the benchmark ran more iterations in the same wall-clock budget. Schubfach is faster per-call, so Go’s benchmark framework runs more iterations, and the cumulative CPU time in the shared code paths increases accordingly.

The next meaningful performance improvement for json-canon is not in number formatting. It is in the parser (which builds a full value tree with heap-allocated strings for every key and value) and the serializer (which sorts keys via sort.Slice with a closure). Those are architectural decisions that would require API changes to address, not algorithm swaps within an existing interface.

Geometric Means

benchstat reports the following geometric means across all 51 API benchmark comparisons (n=6):

The B/op reduction (3.81%) is smaller than the allocs/op reduction (10.54%) because the eliminated allocations are small (big.Int internal slices, typically 8-64 bytes each). The throughput improvement exceeds the allocation reduction because the saved allocations are not just memory operations. They are also CPU operations: big.Int multiply involves digit-by-digit multiplication with carry propagation, normalization, and bounds checking. Eliminating the allocations eliminates the arithmetic they supported.

Statistical Evidence

The API benchmarks were run with -count=6. benchstat computes valid confidence intervals for all 51 comparisons.

Of the 51 comparisons, 40 reach statistical significance at p < 0.05. The 11 that do not are workloads where the number formatter is irrelevant: deep (p=0.784), deep-64 (p=0.589), unicode (p=0.108 to 0.781 across modes), surrogate-pair/noncanonical (p=0.058), long-string/canonical (p=0.818), and rfc-key-sorting/canonical (p=0.240). These are the exact workloads where no performance difference is expected.

Every number-dense workload reaches p=0.002 (the minimum achievable p-value with n=6 using a permutation test). The Burger-Dybvig wins on string-dominant workloads also reach significance (control-escapes at p=0.002, 6% slower for Schubfach), confirming these are real effects from binary layout or instruction cache differences, not measurement noise.

The CLI end-to-end results (9 runs per comparison, permutation test with 3,000 resamples) provide independent corroboration. Five of 28 CLI comparisons reach significance at p < 0.05: canonicalize/array-2048 at p=0.0007 (1.32x, d=-3.14), canonicalize/nested-mixed at p=0.0017 (1.36x, d=-2.01), canonicalize/numeric-boundary at p=0.036 (1.22x, d=-1.08), verify/array-2048 at p=0.0007 (1.30x, d=-3.97), and verify/array-256 at p=0.038 (1.22x, d=-1.09). All five favor Schubfach. The remaining 23 comparisons do not reach significance because process startup noise dominates at the 1-7ms timescales of CLI execution.

The Decision

The conformance gate is clean. The performance improvement is statistically significant across every number-dense workload. The implementation is a drop-in replacement: same FormatDouble signature, same error types, same output bytes for every tested input.

Burger-Dybvig was the right choice for the initial implementation. It was straightforward to verify against the original paper (Burger and Dybvig, PLDI 1996). It was straightforward to debug when the oracle vectors caught errors during initial development. It was straightforward to reason about when modifying the tie-breaking policy for ECMA-262 compliance. It served its purpose: it proved the approach was viable and produced correct output while the rest of the pipeline was built around it.

Schubfach replaces Burger-Dybvig in json-canon. The change ships as a minor version. The API does not change. The output does not change. The canonical bytes for every input remain identical. What changes is the cost of producing them: fewer allocations, less memory, higher throughput on number-dense workloads, and the elimination of math/big from the formatting path’s runtime dependency graph.

Revision History

The implementation lives in the jcsfloat package of json-canon, an RFC 8785 JSON canonicalization library written in Go.

return lBest*seqIOCostFactor + selfJoinCard*(randIOCostFactor+seqIOCostFactor), nil

On ARM, the Go compiler emitted a Fused-Multiply-Add instruction for this expression. On x86, it did not. The FMA rounded once instead of twice, producing a result that differed by one unit in the last place. That was enough to flip a less-than comparison between two nearly identical plan costs and select a different join order.

The values: 3.09928472e+06 on one platform, 3.0992847200000007e+06 on the other. Both IEEE 754 compliant. Both correct. Different bits.

The DoltHub team documented the discovery and the fix on their blog. The fix is interesting because it reveals a deliberate design choice in the Go compiler, and that design choice has implications for any Go program whose correctness depends on identical floating-point output across platforms.

FMA at the Hardware Level

A Fused-Multiply-Add computes a * b + c as a single operation with a single rounding step. Without FMA, the CPU computes a * b, rounds the result to the destination precision, adds c, and rounds again. Two operations, two roundings. FMA eliminates the intermediate rounding. The result is generally more accurate, closer to the true mathematical value, because only one rounding error is introduced instead of two. But when the intermediate rounding would have rounded in a different direction than the final rounding, the FMA and non-FMA paths produce results that differ by one ULP.

IEEE 754 Section 5.4.1 defines fusedMultiplyAdd as a sanctioned operation. As Doug Priest noted in his appendix to Goldberg’s “What Every Computer Scientist Should Know About Floating-Point Arithmetic,” the standard requires correct rounding to the destination precision but does not require that intermediate precision be determined by the programmer’s source code.

The hardware landscape: ARMv8 chips universally support FMA (FMADD/FMSUB). On x86, FMA requires the FMA3 extension, available since Haswell (Intel, 2013) and Piledriver (AMD, 2012), but absent on older and some low-power chips. A compiler targeting ARM can emit FMA for any multiply-add expression. The same compiler targeting older x86 cannot. The same source code, the same compiler version, the same optimization level. Different instructions, different rounding, different bits.

Go’s Compiler Policy

Go addressed FMA in issue #17895, accepted for Go 1.9. The consensus:

A float64 conversion should be an explicit signal that a rounded float64 should be materialized.

The Go compiler is free to fuse a*b + c into a single FMA instruction whenever the target hardware supports it. The opt-out mechanism is an explicit float64() cast, which forces the compiler to materialize a rounded intermediate result:

// FMA allowed -- compiler may fuse on ARM
result := a*b + c

// FMA prevented -- explicit rounding after multiply
result := float64(a*b) + c

The DoltHub fix applied this directly:

return float64(lBest*seqIOCostFactor) +
    float64(selfJoinCard*(randIOCostFactor+seqIOCostFactor)), nil

This is a clean solution for expressions where you can identify and annotate the vulnerable multiply-add patterns. For a query planner, that may be a handful of cost-calculation expressions. For a digit-generation algorithm that involves extended sequences of floating-point arithmetic, the question becomes harder: how do you ensure that every expression in the pipeline is either not fusible or explicitly guarded?

Standard Library Formatters as Implementation Details

In January 2019, Anders Rundgren, the author of RFC 8785, filed Go issue #29491 against strconv.FormatFloat. His reference JCS implementation was producing incorrect rounding on Windows/amd64 for specific values:

The root cause was a bug in strconv’s roundShortest function. The fix shipped in Go 1.13. The RFC author’s own reference implementation, broken by the standard library it delegated to.

This is not an indictment of strconv.FormatFloat. It is a high-quality implementation. But it is a general-purpose formatting function whose underlying algorithm has changed over Go’s history: Grisu3 with exact-arithmetic fallback through Go 1.16, Ryu from Go 1.17, and Dragonbox from Go 1.26. Each transition preserved round-trip correctness. None contractually guaranteed digit-sequence stability. The strconv documentation guarantees a shortest round-trip representation. It does not guarantee which valid shortest representation it will choose when two are equally correct, and it does not guarantee that the choice will remain stable across releases.

For a general-purpose formatting function, this is fine. Any shortest round-trip string is equally useful. For a canonicalization scheme, “valid but different” is a conformance failure, because the specification requires one specific digit sequence for each IEEE 754 bit pattern.

A Common Pattern in JCS Implementations

gowebpki/jcs, the maintained Go fork of Anders Rundgren’s RFC 8785 reference implementation, takes the obvious approach to number formatting: delegate to the standard library.

func NumberToJSON(ieeeF64 float64) (res string, err error) {
    ieeeU64 := math.Float64bits(ieeeF64)

    if (ieeeU64 & invalidPattern) == invalidPattern {
        return "null", errors.New("Invalid JSON number: " +
            strconv.FormatUint(ieeeU64, 16))
    }
    if ieeeF64 == 0 {
        return "0", nil
    }

    var sign string = ""
    if ieeeF64 < 0 {
        ieeeF64 = -ieeeF64
        sign = "-"
    }

    var format byte = 'e'
    if ieeeF64 < 1e+21 && ieeeF64 >= 1e-6 {
        format = 'f'
    }

    // The following should (in "theory") do the trick:
    es6Formatted := strconv.FormatFloat(ieeeF64, format, -1, 64)

    exponent := strings.IndexByte(es6Formatted, 'e')
    if exponent > 0 {
        if es6Formatted[exponent+2] == '0' {
            es6Formatted = es6Formatted[:exponent+2] +
                es6Formatted[exponent+3:]
        }
    }
    return sign + es6Formatted, nil
}

This is a reasonable design. The function handles the special cases (NaN, infinity, negative zero, sign extraction, ECMA-262 format selection, exponent normalization) and delegates the hard part, digit generation, to strconv.FormatFloat. This is the pattern most JCS implementations follow, in Go and in other languages, because digit generation is genuinely difficult and the standard library already does it.

The coupling this creates is straightforward: if strconv.FormatFloat changes its output for a given input, the canonical output changes. Whether that matters depends on requirements. For applications where JCS output is compared within a single Go version on a single architecture, it may not matter at all. For applications where canonical output must be identical across Go versions, across platforms, or across language implementations, the coupling is the mechanism by which platform differences propagate into the canonical output.

Approaches to Platform-Independent Digit Generation

Two approaches eliminate platform dependence in the digit-generation pipeline. They make different tradeoffs.

Arbitrary-Precision Integer Arithmetic

The Burger-Dybvig algorithm represents the float value and its rounding boundaries as ratios of arbitrary-precision integers (math/big.Int in Go). After the initial math.Float64bits bit-cast extracts the raw IEEE 754 pattern, every subsequent operation is exact integer arithmetic:

// Digit extraction: multiply R by 10, divide by S
state.r.Mul(state.r, bigTen)
d := new(big.Int)
d.DivMod(state.r, state.s, state.r)

big.Int.Mul, big.Int.DivMod, big.Int.Cmp: these are integer operations executed on the CPU’s integer ALU. The Go compiler cannot emit FMA instructions for them because FMA is a floating-point instruction. The guarantee holds not because the code is carefully written to avoid fusible expressions, but because the types involved make fusion inapplicable.

The single floating-point operation in the pipeline is a log10 estimate used for initial decimal scaling. This estimate is allowed to be wrong. Two integer fixup passes correct it using exact big.Int.Cmp comparisons. An FMA-affected log10 estimate that is off by one is corrected the same way as any other off-by-one estimate.

The tradeoff is performance. math/big operations allocate heap memory and are substantially slower than fixed-width integer arithmetic. For a canonicalization library where the output contract is more important than throughput, this is an acceptable cost. For a high-throughput formatter, it may not be.

Shortest Round-Trip: Implementing IEEE 754 to Decimal Conversion in Go covers the full Burger-Dybvig implementation. The determinism claim is backed by an offline replay harness that runs 60 independent executions across 12 Linux environments on both x86_64 and arm64, comparing SHA-256 digests of canonical output.

Fixed-Width Integer Algorithms

Ryu and Schubfach (the algorithm underlying Dragonbox) avoid arbitrary-precision arithmetic entirely. They use fixed-width 64-bit and 128-bit integer operations with precomputed lookup tables. Their core paths are substantially faster than Burger-Dybvig.

These core paths are mostly integer math. A Ryu implementation’s critical loop looks roughly like this:

// Ryu core: fixed-width 128-bit multiply against precomputed table
vr := mulShift64(m2, table[q], j)  // uint64 multiply + shift
vp := mulShift64(m2+1, table[q], j)
vm := mulShift64(m2-1, table[q], j)
// ... digit extraction from vr, vp, vm using integer division

These are integer multiplications and shifts, not floating-point multiply-adds. A careful Go implementation of Ryu or Schubfach would likely be immune to FMA for the same structural reason as Burger-Dybvig: the critical operations use integer types that the compiler cannot fuse.

The risk is at the edges. A Ryu port might use a floating-point log10 estimator, or a helper function that computes an exponent approximation through float64 arithmetic. These expressions could be fusible. Where Burger-Dybvig with math/big makes FMA inapplicable by type across the entire pipeline, a fixed-width implementation needs verification that:

• No floating-point helper functions (log10 estimators, exponent approximations) use expressions the compiler could fuse.
• No intermediate value is stored in a float64 where FMA could change the rounding.
• The lookup table generation does not depend on platform-specific floating-point behavior.
• These properties hold across Go compiler versions, as the compiler’s fusion heuristics evolve.

This is not a fundamental obstacle. It is ongoing maintenance work, analogous to the float64() cast discipline that the DoltHub team applied to their cost model. For a project where the performance difference between fixed-width and arbitrary-precision arithmetic matters, the audit cost is well worth paying.

The Broader Observation

IEEE 754 compliance guarantees that each result is correctly rounded to the destination precision. It does not guarantee that two IEEE 754 compliant implementations produce identical results for the same source expression, because the standard permits operations like FMA that change how many roundings occur.

For most software, this does not matter. A query planner that picks a different join order on ARM versus x86 is a correctness problem only because it affects deterministic testing. The alternative plan may be equally efficient. A numerical simulation that differs by one ULP across platforms is within tolerance for almost all applications. FMA is, in most contexts, a net benefit: faster and more accurate.

For software whose correctness property is byte-identical output across platforms (canonicalization schemes, content-addressed storage, reproducible builds), the platform independence of the digit-generation pipeline is an architectural requirement, not a detail. Delegating to strconv.FormatFloat binds the output to the standard library’s current algorithm, its FMA exposure on the current platform, and its stability guarantees, which are round-trip correctness for a shortest representation, not digit-sequence stability. The approaches to owning the pipeline differ in how they achieve independence and what they trade for it, but the requirement itself is a consequence of IEEE 754’s design: compliance governs accuracy, not identity.

The Burger-Dybvig approach discussed above is the one used in json-canon, an RFC 8785 JSON canonicalization library written in Go.

Revision History

0.1 + 0.2 = 0.30000000000000004

The joke is that floating-point arithmetic is broken. It isn’t. IEEE 754 is doing exactly what it specifies. The problem surfaces when you need to serialize these values to text, and when two different systems need to produce exactly the same text for the same value.

This is what RFC 8785 (JSON Canonicalization Scheme) requires: byte-deterministic JSON output. And the hardest part of that requirement is number formatting. You need the shortest decimal string that, when parsed back, recovers the original IEEE 754 bits. You need to agree on tie-breaking when two representations are equally short. And you need to match the exact output format specified by ECMA-262 Number serialization, because that’s what RFC 8785 mandates.

Go’s strconv.FormatFloat is a high-quality shortest-round-trip formatter, but it is not an ECMA-262 conformance contract. So I implemented the Burger-Dybvig algorithm from scratch in Go, validated against pinned oracle test vectors. This article walks through the entire implementation.

IEEE 754 Anatomy: 64 Bits of Structure

Before generating digits, you need to understand the raw material. An IEEE 754 double-precision float is 64 bits:

[1 bit: sign] [11 bits: biased exponent] [52 bits: mantissa]

The value it represents (for normal numbers) is:

(-1)^sign × 1.mantissa × 2^(exponent - 1023)

The 1.mantissa is key: normal numbers have an implicit leading 1 bit, giving you 53 bits of precision total. Subnormal numbers (biased exponent = 0) lose the implicit bit, giving 0.mantissa × 2^(-1022).

Here’s how the implementation extracts these parts:

func decodeFloatParts(f float64) floatParts {
    bits := math.Float64bits(f)
    mantissa := bits & ((uint64(1) << 52) - 1)
    expBits := exponentBits(bits)
    biasedExp := int(expBits)

    fMant := mantissa
    fExp := 1 - 1023 - 52
    if biasedExp != 0 {
        fMant = (uint64(1) << 52) | mantissa
        fExp = biasedExp - 1023 - 52
    }

    lowerBoundary := biasedExp > 1 && mantissa == 0

    return floatParts{
        mantissa:      mantissa,
        biasedExp:     biasedExp,
        fMant:         fMant,
        fExp:          fExp,
        lowerBoundary: lowerBoundary,
        isEven:        fMant%2 == 0,
    }
}

Two things to note here. First, the subnormal branch: when biasedExp == 0, there’s no implicit leading 1, and the effective exponent is fixed at 2^(-1074) (the smallest representable power). Second, the lowerBoundary flag: when the mantissa is zero and the exponent is above the minimum normal range, the float sits at a power-of-two boundary where the gap to the lower adjacent representable is half the gap to the upper adjacent. This asymmetry matters for the algorithm.

The exponent extraction itself works on the raw bit pattern:

func exponentBits(bits uint64) uint16 {
    hi := byte((bits >> 56) & 0xFF)
    lo := byte((bits >> 48) & 0xFF)
    return (uint16(hi&0x7F) << 4) | uint16(lo>>4)
}

This reconstructs the 11-bit biased exponent by extracting the relevant bytes and masking away the sign bit.

The Core Insight: Why You Need Multiprecision Arithmetic

The Burger-Dybvig algorithm answers this question: what is the shortest decimal string d such that parse(d) == f for our original float f?

To answer it, you need to compute exact boundaries. Every float f has two adjacent representable values. The “shortest round-trip” string must map back to f and not to either neighbor. This means computing the midpoints between f and its neighbors with exact arithmetic, not floating-point arithmetic, which would introduce the very imprecision you’re trying to eliminate.

The algorithm represents the value and its boundaries as ratios of big integers. For a float with fractional mantissa fMant and exponent fExp, the value is fMant × 2^fExp. The boundaries M- and M+ define the interval within which any decimal representation will round back to f.

State Initialization: R, S, M+, M-

The algorithm works with four multiprecision integers:

• R (remainder): represents the current value, scaled into the digit-extraction space
• S (scale): the denominator; digits are extracted by dividing R by S
• M+ (upper margin): how far R can grow before rounding to the next float
• M- (lower margin): how far R can shrink before rounding to the previous float

Initialization depends on the exponent sign:

func initScaledPositiveExp(state *digitState, parts floatParts) {
    if !parts.lowerBoundary {
        state.r.SetUint64(parts.fMant)
        lshByInt(state.r, parts.fExp+1)  // r = fMant × 2^(fExp+1)
        state.s.SetInt64(2)               // s = 2
        state.mPlus.SetInt64(1)
        lshByInt(state.mPlus, parts.fExp) // m+ = 2^fExp
        state.mMinus.Set(state.mPlus)     // m- = m+
        return
    }

    // Lower boundary: asymmetric margins
    state.r.SetUint64(parts.fMant)
    lshByInt(state.r, parts.fExp+2)       // r = fMant × 2^(fExp+2)
    state.s.SetInt64(4)                    // s = 4
    state.mPlus.SetInt64(1)
    lshByInt(state.mPlus, parts.fExp+1)   // m+ = 2^(fExp+1)
    state.mMinus.SetInt64(1)
    lshByInt(state.mMinus, parts.fExp)    // m- = 2^fExp (half of m+)
}

The lowerBoundary case (mantissa is zero, exponent above minimum) requires special handling. At a power-of-two boundary, the float sits at the top of one binade and the bottom of the next. The gap to the lower neighbor is half the gap to the upper neighbor. The algorithm accounts for this by doubling all values (multiplying R and S by 2) and setting m+ = 2 × m-.

For negative exponents, the roles are similar but the scaling direction reverses. Instead of left-shifting R, we left-shift S:

func initScaledNegativeExp(state *digitState, parts floatParts) {
    if !parts.lowerBoundary {
        state.r.SetUint64(parts.fMant)
        lshByInt(state.r, 1)               // r = fMant × 2
        state.s.SetInt64(1)
        lshByInt(state.s, -parts.fExp+1)   // s = 2^(-fExp+1)
        state.mPlus.SetInt64(1)            // m+ = 1
        state.mMinus.SetInt64(1)           // m- = 1
        return
    }

    state.r.SetUint64(parts.fMant)
    lshByInt(state.r, 2)                   // r = fMant × 4
    state.s.SetInt64(1)
    lshByInt(state.s, -parts.fExp+2)       // s = 2^(-fExp+2)
    state.mPlus.SetInt64(2)                // m+ = 2
    state.mMinus.SetInt64(1)               // m- = 1
}

Power-of-10 Scaling

After initialization, R, S, M+, and M- are all in base-2 space. To extract decimal digits, we need to scale them so that the first digit comes from floor(R/S). This requires estimating k ≈ ceil(log10(f)) and scaling accordingly:

func scaleByPower10(state *digitState, k int) {
    switch {
    case k > 0:
        p := pow10Big(k)
        state.s.Mul(state.s, p)           // Scale denominator up
    case k < 0:
        p := pow10Big(-k)
        state.r.Mul(state.r, p)           // Scale numerator up
        state.mPlus.Mul(state.mPlus, p)   // Scale margins too
        state.mMinus.Mul(state.mMinus, p)
    }
}

The k estimate uses floating-point logarithms and is allowed to be off by one. Two fixup passes correct any estimation error:

func applyHighFixup(state *digitState, isEven bool, n int) int {
    high := new(big.Int).Add(state.r, state.mPlus)
    if cmpHigh(high, state.s, isEven) {
        state.s.Mul(state.s, bigTen)
        return n + 1
    }
    return n
}

If R + M+ already exceeds S after initial scaling, we need one more decimal position; multiply S by 10 and increment the exponent count. The low fixup works in the opposite direction, looping while 10R and 10(R + M+) are both less than S.

Computing pow10Big efficiently matters because these are big integer multiplications. The implementation pre-computes 700 powers of 10 at init time:

var pow10Cache [700]*big.Int

func init() {
    pow10Cache[0] = big.NewInt(1)
    for i := 1; i < len(pow10Cache); i++ {
        pow10Cache[i] = new(big.Int).Mul(pow10Cache[i-1], bigTen)
    }
}

func pow10Big(n int) *big.Int {
    if n >= 0 && n < len(pow10Cache) {
        return new(big.Int).Set(pow10Cache[n])  // Defensive copy
    }
    return new(big.Int).Exp(bigTen, big.NewInt(int64(n)), nil)
}

The defensive copy is critical. Without it, callers that mutate the returned *big.Int would corrupt the cache. IEEE 754 binary64 ranges from approximately 10^-324 to 10^308, so the 700-entry cache covers all practical k values.

Digit Generation: The Main Loop

With scaling complete, digit extraction is a simple loop. Multiply R, M+, and M- by 10, then divide R by S to get one decimal digit. Repeat until termination:

func extractDigits(state *digitState, isEven bool, n int) (string, int) {
    var digitBuf [30]byte
    dIdx := 0
    quot := new(big.Int)
    rem := new(big.Int)

    for {
        scaleDigitState(state)                              // R, M+, M- *= 10
        d := divideAndRemainder(state, quot, rem)           // d = R / S; R = R mod S

        tc1, tc2 := terminationConditions(state, isEven)
        if !tc1 && !tc2 {
            digitBuf[dIdx] = byte('0' + d)
            dIdx++
            continue
        }

        digitBuf[dIdx] = finalDigit(d, tc1, tc2, state.r, state.s)
        dIdx++
        break
    }

    n = normalizeDigitBuffer(digitBuf[:], dIdx, &dIdx, n)
    return string(digitBuf[:dIdx]), n
}

The 30-byte buffer is generous. IEEE 754 binary64 produces at most 17 significant digits in shortest form, with carry propagation adding at most one more.

The two termination conditions test whether we’ve reached a boundary:

func terminationConditions(state *digitState, isEven bool) (bool, bool) {
    tc1 := cmpRoundDown(state.r, state.mMinus, isEven)
    high := new(big.Int).Add(state.r, state.mPlus)
    tc2 := cmpHigh(high, state.s, isEven)
    return tc1, tc2
}

• tc1 (round down): the remainder R is small enough that rounding down to digit d would still land in the correct interval
• tc2 (round up): the remainder R plus the upper margin M+ meets or exceeds S, meaning rounding up to d+1 is within range

When neither condition fires, we aren’t yet at the shortest representation; emit the digit and continue. When at least one fires, this is the last digit.

Even-Digit Tie-Breaking

The most subtle part of the algorithm is what happens when both termination conditions fire simultaneously: the value sits at the exact midpoint between two representations. ECMA-262 Note 2 mandates even-digit tie-breaking (banker’s rounding):

func finalDigit(d int, tc1, tc2 bool, r, s *big.Int) byte {
    switch {
    case tc1 && !tc2:
        return byte('0' + d)      // Only round-down applies: use d
    case !tc1 && tc2:
        return byte('0' + d + 1)  // Only round-up applies: use d+1
    default:
        return midpointDigit(d, r, s)  // Both apply: tie-break
    }
}

func midpointDigit(d int, r, s *big.Int) byte {
    twoR := new(big.Int).Lsh(r, 1)
    cmp := twoR.Cmp(s)
    if cmp < 0 {
        return byte('0' + d)      // Closer to lower: round down
    }
    if cmp > 0 {
        return byte('0' + d + 1)  // Closer to upper: round up
    }
    // Exactly at midpoint: even-digit tie-breaking
    if d%2 == 0 {
        return byte('0' + d)      // d is even: keep it
    }
    return byte('0' + d + 1)      // d is odd: round up to even
}

The midpoint test compares 2R against S. If 2R < S, the remainder is less than half the scale, so round down. If 2R > S, round up. When 2R == S exactly, the algorithm breaks the tie by choosing the even digit.

This even-digit preference also influences the boundary comparisons themselves. The cmpRoundDown, cmpHigh, and cmpLow functions use different comparison operators depending on whether the mantissa is even or odd:

func cmpRoundDown(lhs, rhs *big.Int, isEven bool) bool {
    if isEven {
        return lhs.Cmp(rhs) <= 0   // Inclusive: allow rounding to even at boundary
    }
    return lhs.Cmp(rhs) < 0        // Strict: odd mantissa
}

func cmpHigh(lhs, rhs *big.Int, isEven bool) bool {
    if isEven {
        return lhs.Cmp(rhs) >= 0   // Inclusive
    }
    return lhs.Cmp(rhs) > 0        // Strict
}

This asymmetry is the difference between ECMA-262 compliance and “close enough.” When the mantissa is even, the boundary comparisons are inclusive, allowing termination at exact boundary values and creating the conditions for even-digit tie-breaking. When odd, strict comparisons ensure the algorithm doesn’t terminate prematurely.

Carry Propagation

When finalDigit returns d+1, it may produce a 10 (the character after ‘9’). This overflow must propagate:

func normalizeDigitBuffer(digitBuf []byte, dIdx int, dIdxPtr *int, n int) int {
    // Propagate carries
    for i := dIdx - 1; i > 0; i-- {
        if digitBuf[i] > '9' {
            digitBuf[i] = '0'
            digitBuf[i-1]++
        }
    }

    // Handle carry out of the first digit
    if dIdx > 0 && digitBuf[0] > '9' {
        copy(digitBuf[1:dIdx+1], digitBuf[0:dIdx])
        digitBuf[0] = '1'
        digitBuf[1] = '0'
        dIdx++
        n++    // One more integer digit
    }

    // Strip trailing zeros
    for dIdx > 1 && digitBuf[dIdx-1] == '0' {
        dIdx--
    }
    *dIdxPtr = dIdx
    return n
}

Consider a carry cascade: digits [9, 9, 10] become [9, 10, 0] then [10, 0, 0], and finally the carry-out case shifts everything right to produce [1, 0, 0, 0] with an incremented exponent. Trailing zeros are then stripped since the algorithm produces shortest representations.

ECMA-262 Output Formatting

The Burger-Dybvig algorithm produces a digit string and an exponent n. The ECMA-262 specification (§6.1.6.1.20) defines four formatting branches based on n:

func formatECMA(negative bool, digits string, n int) string {
    k := len(digits)

    var buf []byte
    if negative {
        buf = append(buf, '-')
    }

    switch {
    case k <= n && n <= 21:
        // Integer fixed: "12300" (digits + trailing zeros)
        buf = appendIntegerFixed(buf, digits, k, n)
    case 0 < n && n <= 21:
        // Fraction fixed: "12.345" (decimal point within digits)
        buf = appendFractionFixed(buf, digits, n)
    case -6 < n && n <= 0:
        // Small fraction: "0.00123" (leading zeros after decimal point)
        buf = appendSmallFraction(buf, digits, n)
    default:
        // Exponential: "1.23e+20" or "1e-7"
        buf = appendExponential(buf, digits, k, n)
    }

    return string(buf)
}

The boundary constants come directly from the ECMA-262 specification:

The boundaries are exact. A value of exactly 10^21 uses exponential notation (1e+21), while 999999999999999900000 (just below 10^21) uses integer fixed notation. A value of exactly 10^-6 uses small fraction notation (0.000001), while anything below that switches to exponential.

These boundary behaviors can be verified against specific bit patterns:

0x444b1ae4d6e2ef50 → "1e+21"                        (exponential)
0x444b1ae4d6e2ef4f → "999999999999999900000"         (integer fixed)
0x3eb0c6f7a0b5ed8d → "0.000001"                      (small fraction)
0x3eb0c6f7a0b5ed8c → "9.999999999999997e-7"          (exponential)

Why strconv.FormatFloat Is Insufficient

Go’s strconv.FormatFloat is the standard library’s number-to-string conversion. With format 'e', 'f', or 'g' and precision -1, it produces shortest-round-trip representations. The issue for JCS is not quality; the issue is contract mismatch. RFC 8785 requires ECMAScript-compatible rendering rules, and FormatFloat does not expose that contract.

The gaps are concrete.

Format-policy divergence. ECMA-262 and FormatFloat make different notation choices at key boundaries. Examples:

These are all valid shortest representations, but only one side matches the JCS contract (see RFC 8785’s ECMAScript-compatible serialization examples in Appendix B).

These three examples are sufficient to establish the core point: ECMA-262 rendering switches between multiple notation branches (fixed integer, fixed fraction, small fraction, exponential) at exact exponent thresholds. No single FormatFloat mode reproduces all of them. The 'e' and 'f' modes each use one fixed notation. The 'g' mode switches between 'e' and 'f' based on exponent magnitude, but its switching thresholds and formatting details do not match ECMA-262’s. Once boundary cases disagree, conformance requires a dedicated ECMA-262 formatting layer.

Upstream algorithm drift risk. Go’s shortest-mode internals are not static over time. Through Go 1.16, strconv used Grisu3 with an exact-arithmetic fallback. Go 1.17 replaced this with Ryu. Go 1.26 replaced Ryu with Dragonbox for shortest-mode formatting. Each transition preserved round-trip correctness and improved performance, but the strconv documentation does not guarantee that digit sequences will remain stable across releases. Three algorithm replacements in ten releases is normal for a standard library, but it is exactly why canonicalization code should not outsource its normative behavior to implementation details of an external runtime.

Building from scratch eliminates these issues. The Burger-Dybvig path always uses exact multiprecision arithmetic, applies explicit midpoint handling, and formats according to ECMA-262 branch rules. The cost is performance, but for canonicalization, correctness is the constraint.

Special Values

Three special cases are handled before the algorithm runs:

func FormatDouble(f float64) (string, *jcserr.Error) {
    if math.IsNaN(f) {
        return "", jcserr.New(jcserr.InvalidGrammar, -1,
            "NaN is not representable in JSON")
    }
    if f == 0 {
        return "0", nil    // Both +0 and -0 produce "0"
    }
    if math.IsInf(f, 0) {
        return "", jcserr.New(jcserr.InvalidGrammar, -1,
            "Infinity is not representable in JSON")
    }
    // ... proceed with Burger-Dybvig
}

NaN and Infinity are errors; JSON has no representation for them. Negative zero is normalized to "0" because IEEE 754’s -0 and +0 are mathematically equal, and canonical JSON should not distinguish them. (Note: lexical -0 in JSON input is a separate concern, rejected at parse time as a policy violation; the two requirements are independent.)

Validation: Pinned Oracle Vectors

The implementation is validated against two pinned oracle datasets:

• golden_vectors.csv: boundary values, powers of 10, subnormals, and edge cases
• golden_stress_vectors.csv: tie-breaking and carry propagation scenarios

Each CSV row contains a 16-character hex encoding of the IEEE 754 bits and the expected output string:

0000000000000000,0
0000000000000001,5e-324
3ff0000000000000,1
3eb0c6f7a0b5ed8d,0.000001

The test function validates three properties per dataset:

1. Semantic correctness: FormatDouble(bits) == expected for every row
2. Cardinality: The exact row count matches expectations (catches truncation)
3. Integrity: SHA-256 of the entire file matches a pinned hash (catches corruption)

func TestGoldenOracle(t *testing.T) {
    verifyOracle(t, "testdata/golden_vectors.csv", 54445,
        "593bdecbe0dccbc182bc3baf570b716887db25739fc61b7808764ecb966d5636")
}

func TestStressOracle(t *testing.T) {
    verifyOracle(t, "testdata/golden_stress_vectors.csv", 231917,
        "287d21ac87e5665550f1baf86038302a0afc67a74a020dffb872f1a93b26d410")
}

The SHA-256 checksums pin the test data against silent modification. If someone edits a single byte in either file, the test fails.

Round-Trip and Fuzz Testing

Beyond oracle vectors, the implementation uses two additional validation strategies.

Round-trip testing verifies the shortest representation property directly: format a value, parse it back with strconv.ParseFloat, and confirm the original bits are recovered:

cases := []float64{5e-324, 1e-7, 1e-6, 0.1, 0.2, 1.1, 1, 2, 1e20, 1e21, math.MaxFloat64}
for _, c := range cases {
    formatted, _ := jcsfloat.FormatDouble(c)
    parsed, _ := strconv.ParseFloat(formatted, 64)
    if parsed != c {
        t.Fatalf("round-trip failed for %.17g: formatted %q, parsed back as %.17g",
            c, formatted, parsed)
    }
}

Fuzz testing generates random 64-bit patterns, interprets them as IEEE 754 doubles, and verifies round-trip stability:

func FuzzFormatDoubleRoundTrip(f *testing.F) {
    f.Fuzz(func(t *testing.T, data []byte) {
        if len(data) < 8 { return }
        bits := binary.BigEndian.Uint64(data[:8])
        fval := math.Float64frombits(bits)

        if math.IsNaN(fval) || math.IsInf(fval, 0) {
            _, err := jcsfloat.FormatDouble(fval)
            if err == nil { t.Fatal("expected error for non-finite") }
            return
        }

        s, err := jcsfloat.FormatDouble(fval)
        if err != nil { t.Fatalf("unexpected error: %v", err) }

        parsed, _ := strconv.ParseFloat(s, 64)
        if fval == 0 {
            if parsed != 0 { t.Fatal("zero round-trip failed") }
            return
        }
        if math.Float64bits(parsed) != math.Float64bits(fval) {
            t.Fatalf("round-trip failed: bits=%016x → %q → bits=%016x",
                bits, s, math.Float64bits(parsed))
        }
    })
}

Note the special case for zero: IEEE 754 -0 formats as "0", which parses back as +0. The bit patterns differ (0x8000000000000000 vs 0x0000000000000000), but mathematically the values are equal, so the round-trip comparison uses == 0 instead of bit equality.

Idempotency testing verifies that format → parse → format produces the same string, which is a consequence of correct tie-breaking:

for i := uint64(1); i < 5000; i += 97 {
    v := math.Float64frombits(i * 0x9e3779b97f4a7c15)
    if math.IsNaN(v) || math.IsInf(v, 0) { continue }

    f1, _ := jcsfloat.FormatDouble(v)
    parsed, _ := strconv.ParseFloat(f1, 64)
    f2, _ := jcsfloat.FormatDouble(parsed)

    if f1 != f2 {
        t.Fatalf("idempotency failed for bits=%016x: %s != %s",
            math.Float64bits(v), f1, f2)
    }
}

If tie-breaking were inconsistent, formatting a value at the exact midpoint could oscillate between two representations. Even-digit tie-breaking prevents this by always choosing the same direction at midpoints.

The Complete Pipeline

Putting it all together, the conversion from IEEE 754 bits to canonical decimal string follows this path:

1. Decode the 64-bit pattern into mantissa, exponent, and boundary flags
2. Initialize R, S, M+, M- as big integers with appropriate scaling
3. Estimate the decimal exponent k using floating-point logarithms
4. Scale by 10^k using pre-computed powers from the 700-entry cache
5. Fix up the scaling if the estimate was off by one (high or low)
6. Extract digits one at a time by multiplying by 10 and dividing
7. Terminate when boundary conditions indicate the shortest representation
8. Tie-break using even-digit preference when at the exact midpoint
9. Propagate carries if rounding up causes overflow
10. Format using the appropriate ECMA-262 branch based on the exponent

The entire implementation has zero external dependencies (only math, math/big, and the project’s own error package). It is deterministic, locale-independent, and produces identical output regardless of platform or Go runtime version.

The implementation lives in the jcsfloat package of json-canon, an RFC 8785 JSON canonicalization library.

Revision History

I built a strict JSON parser in Go. It enforces every constraint in RFC 8259 and adds the RFC 7493 (I-JSON) restrictions that RFC 8785 requires. This article walks through the design and implementation, focusing on the parts where strictness requires real work: surrogate pairs, noncharacter detection, duplicate keys after escape decoding, and resource bounds.

The Gap: What Canonicalization Cannot Accept

Go’s encoding/json is a solid general-purpose parser. For canonicalization, some of its documented behaviors are intentionally too permissive. On Go 1.25.6, representative examples look like this:

Rows labeled RFC 7493 are normative I-JSON requirements. Rows labeled project policy are stricter acceptance rules chosen to keep canonicalization fail-closed and deterministic.

These are not bugs in encoding/json; they are compatibility choices. The Go package documentation for Unmarshal explicitly states that invalid UTF-8 / invalid UTF-16 surrogates are replaced by U+FFFD, and duplicate object keys are processed in order with later values replacing earlier ones.

Minimal reproduction (Go 1.25.6), with explicit error checks:

var v any
err := json.Unmarshal([]byte(`"\uD800\u0041"`), &v)
fmt.Printf("case surrogate err=%v v=%#v\n", err, v) // err=<nil>, v="�A"

err = json.Unmarshal([]byte(`{"a":1,"a":2}`), &v)
fmt.Printf("case dup-key err=%v v=%#v\n", err, v) // err=<nil>, later duplicate replaces earlier value

err = json.Unmarshal([]byte("1e-400"), &v)
fmt.Printf("case underflow err=%v v=%#v\n", err, v) // err=<nil>, v=0

For application code, replacement and merge behavior is pragmatic. A canonicalization engine has a different contract: reject ambiguous or lossy inputs so every accepted value has a single deterministic canonical form. If two parsers disagree on interpretation, canonical output is undefined. Rejection is the safe outcome.

Parser Architecture

The parser uses single-pass recursive descent with byte-level dispatch. The core structure is a parser struct that holds the input bytes, a cursor position, depth tracking, and configurable bounds:

type parser struct {
    data             []byte
    pos              int
    depth            int
    valueCount       int
    maxDepth         int
    maxValues        int
    maxObjectMembers int
    maxArrayElements int
    maxStringBytes   int
    maxNumberChars   int
}

Value dispatch is a single switch on the first byte:

func (p *parser) parseValue() (*Value, error) {
    p.valueCount++
    if p.valueCount > p.maxValues {
        return nil, p.newErrorf(jcserr.BoundExceeded,
            "value count %d exceeds maximum %d", p.valueCount, p.maxValues)
    }

    c, ok := p.peek()
    if !ok {
        return nil, p.newError("unexpected end of input")
    }

    switch c {
    case '{':  return p.parseObject()
    case '[':  return p.parseArray()
    case '"':  return p.parseString()
    case 't', 'f':  return p.parseBool()
    case 'n':  return p.parseNull()
    default:   return p.parseNumber()
    }
}

No lookahead beyond the first byte is needed for type discrimination. Every call to parseValue checks the bound counter first.

UTF-8 Validation: Upfront and Incremental

The parser validates UTF-8 twice, for different reasons.

Upfront bulk validation runs before any parsing begins:

if !utf8.Valid(data) {
    return nil, jcserr.New(jcserr.InvalidUTF8, firstInvalidUTF8Offset(data),
        "input is not valid UTF-8")
}

This catches structural UTF-8 violations: continuation bytes without start bytes, truncated multibyte sequences, overlong encodings, and raw UTF-8 encodings of surrogate code points. The firstInvalidUTF8Offset function locates the exact byte offset of the first violation by scanning rune-by-rune until utf8.DecodeRune returns a replacement character:

func firstInvalidUTF8Offset(data []byte) int {
    for i := 0; i < len(data); {
        _, size := utf8.DecodeRune(data[i:])
        if size == 1 && data[i] >= 0x80 {
            return i
        }
        i += size
    }
    return 0
}

Incremental validation happens during string parsing. After verifying the input is well-formed UTF-8, the string parser still decodes each rune individually to validate semantic constraints: noncharacter rejection and surrogate detection that apply to decoded values, not raw bytes.

Why both? The upfront check is fast (Go’s utf8.Valid is optimized) and gives clear error reporting for malformed input before the parser’s position tracking complicates offset calculation. The incremental check applies rules that operate on decoded code points, not byte patterns.

Number Grammar: Four Layers of Enforcement

RFC 8259 §6 defines a specific number grammar. The parser enforces it in four phases.

Leading Zero Rejection

func (p *parser) scanZeroIntegerPart() *jcserr.Error {
    p.pos++
    if p.pos < len(p.data) && isDigit(p.data[p.pos]) {
        return p.newError("leading zero in number")
    }
    return nil
}

01, 007, 00.5: all rejected. A zero integer part must be exactly 0, followed by a non-digit (or end of number). RFC 8259 §6 specifies: “Leading zeros are not allowed.”

Missing Fraction Digits

func (p *parser) scanFractionPart(numStart int) *jcserr.Error {
    if p.pos >= len(p.data) || p.data[p.pos] != '.' {
        return nil
    }
    p.pos++

    if p.pos >= len(p.data) || !isDigit(p.data[p.pos]) {
        return p.newError("expected digit after decimal point")
    }
    // ... scan remaining digits
}

1. and 1.e5 are rejected; the grammar requires at least one digit after the decimal point.

Lexical Negative Zero

This is a project policy enforcement, not a grammar rule. The value -0 is mathematically zero, but the lexical token -0 is ambiguous:

// PROF-NEGZ-001: lexical negative zero
if strings.HasPrefix(raw, "-") && tokenRepresentsZero(raw) {
    return nil, jcserr.New(jcserr.NumberNegZero, start,
        "negative zero token is not allowed")
}

This rejects -0, -0.0, -0e0, -0.0e+0, and every other way to spell negative zero in JSON number syntax. The tokenRepresentsZero function checks whether the mantissa portion of the number token contains any non-zero digit, ignoring the exponent entirely.

Note: this is a different requirement from serialization. At serialization time, the IEEE 754 bit pattern for -0 outputs as "0". At parse time, the lexical token -0 is rejected. The two requirements are independent: one governs input, the other governs output.

Overflow and Underflow

After grammar validation, the number string is parsed with strconv.ParseFloat:

f, err := strconv.ParseFloat(raw, 64)

// Overflow: value exceeds IEEE 754 range
if math.IsNaN(f) || math.IsInf(f, 0) {
    return nil, jcserr.New(jcserr.NumberOverflow, start,
        "number overflows IEEE 754 double")
}

// Underflow: non-zero token rounds to zero
if f == 0 && !tokenRepresentsZero(raw) {
    return nil, jcserr.New(jcserr.NumberUnderflow, start,
        "non-zero number underflows to IEEE 754 zero")
}

1e999999 overflows to infinity; rejected. 1e-400 is a non-zero number that underflows to zero in IEEE 754; also rejected. The underflow check uses the same tokenRepresentsZero function: if the token has non-zero digits but ParseFloat returns zero, the precision was lost.

String Parsing: Two Paths

The string parser is a linear scan with two code paths: escape sequences and raw UTF-8 bytes.

func (p *parser) parseString() (*Value, error) {
    if err := p.expect('"'); err != nil {
        return nil, err
    }

    var buf []byte
    for {
        if p.pos >= len(p.data) {
            return nil, p.newError("unterminated string")
        }
        b := p.data[p.pos]
        if b == '"' {
            p.pos++
            return &Value{Kind: KindString, Str: string(buf)}, nil
        }
        if b == '\\' {
            // Escape sequence path
            escapeStart := p.pos
            p.pos++
            r, err := p.parseEscape(escapeStart)
            if err != nil { return nil, err }
            if err := validateStringRune(r, escapeStart); err != nil { return nil, err }
            var tmp [4]byte
            n := utf8.EncodeRune(tmp[:], r)
            buf = append(buf, tmp[:n]...)
            continue
        }
        if b < 0x20 {
            // PARSE-GRAM-004: reject unescaped control characters
            return nil, p.newErrorf(jcserr.InvalidGrammar,
                "unescaped control character 0x%02X in string", b)
        }
        // Raw UTF-8 path
        sourceOffset := p.pos
        r, size := utf8.DecodeRune(p.data[p.pos:])
        if err := validateStringRune(r, sourceOffset); err != nil { return nil, err }
        buf = append(buf, p.data[p.pos:p.pos+size]...)
        p.pos += size
    }
}

Both paths call validateStringRune to check for noncharacters and surrogates. The escape path first decodes the escape to a rune, then validates the decoded value. The raw path decodes the rune from UTF-8 bytes, then validates. The decoded results are identical: \u0041 and A both produce rune 0x41. This is critical for duplicate key detection later.

The escape dispatch is a straightforward table:

func escapedRune(b byte) (rune, bool) {
    switch b {
    case '"':  return '"', true
    case '\\': return '\\', true
    case '/':  return '/', true
    case 'b':  return '\b', true
    case 'f':  return '\f', true
    case 'n':  return '\n', true
    case 'r':  return '\r', true
    case 't':  return '\t', true
    default:   return 0, false
    }
}

RFC 8259 §7 defines exactly these eight simple escapes plus \uXXXX. Anything else (\x41, \U0041, \a) is rejected with InvalidGrammar.

Control characters below U+0020 that appear unescaped are also rejected per §7: “All Unicode characters may be placed within the quotation marks, except for the characters that MUST be escaped: quotation mark, reverse solidus, and the control characters (U+0000 through U+001F).”

Surrogate Pairs: 2-Character Lookahead

UTF-16 surrogate pairs in JSON appear as \uD800\uDC00: two consecutive \uXXXX escapes where the first is a high surrogate (U+D800-U+DBFF) and the second is a low surrogate (U+DC00-U+DFFF). Together they encode a supplementary-plane character (U+10000 and above).

The parser must handle five cases:

1. Not a surrogate: Return the rune as-is
2. Lone low surrogate (U+DC00-U+DFFF appearing first): Error
3. High surrogate with no following \u: Error
4. High surrogate followed by non-low-surrogate: Error
5. Valid pair: Decode to supplementary-plane scalar

func (p *parser) parseUnicodeEscape(sourceOffset int) (rune, *jcserr.Error) {
    r1, err := p.readHex4(sourceOffset)
    if err != nil {
        return 0, err
    }

    if !utf16.IsSurrogate(r1) {
        return r1, nil                              // Case 1: not a surrogate
    }

    if r1 >= 0xDC00 {
        return 0, jcserr.New(jcserr.LoneSurrogate,  // Case 2: lone low surrogate
            sourceOffset, fmt.Sprintf("lone low surrogate U+%04X", r1))
    }

    // Case 3: high surrogate must be followed by \uXXXX
    if p.pos+1 >= len(p.data) || p.data[p.pos] != '\\' || p.data[p.pos+1] != 'u' {
        return 0, jcserr.New(jcserr.LoneSurrogate, sourceOffset,
            fmt.Sprintf("lone high surrogate U+%04X (no following \\u)", r1))
    }
    secondEscapeOffset := p.pos
    p.pos += 2                                      // Skip past '\u'

    r2, err := p.readHex4(secondEscapeOffset)
    if err != nil {
        return 0, err
    }
    if r2 < 0xDC00 || r2 > 0xDFFF {
        return 0, jcserr.New(jcserr.LoneSurrogate,  // Case 4: not a low surrogate
            secondEscapeOffset,
            fmt.Sprintf("high surrogate U+%04X followed by non-low-surrogate U+%04X", r1, r2))
    }

    // Case 5: valid pair
    decoded := utf16.DecodeRune(r1, r2)
    if decoded == unicode.ReplacementChar {
        return 0, jcserr.New(jcserr.LoneSurrogate, sourceOffset,
            fmt.Sprintf("invalid surrogate pair U+%04X U+%04X", r1, r2))
    }
    return decoded, nil
}

The 2-character lookahead checks for \u without consuming input. This is the only lookahead in the parser beyond single-byte dispatch. If the two bytes aren’t \u, the high surrogate is lone and the error points to the first escape’s offset. If the second escape exists but decodes to a non-low surrogate (like \uD800\u0041), the error points to the second escape’s offset, because that’s where the violation occurs.

Compare with encoding/json: it replaces lone surrogates with U+FFFD (documented behavior in the Unmarshal docs). This is convenient for display pipelines, but it silently changes semantic content. A canonicalization engine cannot do this. Changing input bytes means the canonical output no longer represents the original data.

Noncharacter Rejection: 66 Forbidden Code Points

RFC 7493 §2.1 forbids Unicode noncharacters in I-JSON text. There are exactly 66:

• 32 in the range U+FDD0 to U+FDEF
• 34 at U+xFFFE and U+xFFFF for each of the 17 Unicode planes (0-16)

The implementation uses a compact two-branch check:

func IsNoncharacter(r rune) bool {
    if r >= 0xFDD0 && r <= 0xFDEF {
        return true
    }
    if r&0xFFFE == 0xFFFE && r <= 0x10FFFF {
        return true
    }
    return false
}

The bitwise trick on the second branch is worth explaining. The mask r & 0xFFFE zeros the lowest bit, so both U+xFFFE and U+xFFFF match the pattern 0xFFFE. The bound r <= 0x10FFFF limits this to planes 0 through 16. This catches U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, all the way up to U+10FFFE and U+10FFFF, for 2 per plane times 17 planes = 34 code points.

The validation runs on decoded runes, not raw bytes. This matters because supplementary-plane noncharacters like U+1FFFE can appear as either raw UTF-8 bytes or as surrogate pair escapes (\uD83F\uDFFE). Both paths decode to the same rune and hit the same check.

Duplicate Key Detection: After Escape Decoding

RFC 7493 §2.3 requires that JSON objects not contain duplicate member names. The subtle requirement is that "\u0061" and "a" are the same key; both decode to the string “a”.

The parser handles this by comparing decoded strings, not raw lexemes:

func (p *parser) parseObject() (*Value, error) {
    // ...
    v := &Value{Kind: KindObject}
    seen := make(map[string]int)

    for {
        keyStart := p.pos

        keyVal, err := p.parseString()   // Decodes all escapes
        if err != nil { return nil, err }
        key := keyVal.Str                // Decoded Unicode string

        if firstOff, exists := seen[key]; exists {
            return nil, jcserr.New(jcserr.DuplicateKey, keyStart,
                fmt.Sprintf("duplicate object key %q (first at byte %d)", key, firstOff))
        }
        seen[key] = keyStart

        // ... parse colon and value
    }
}

The seen map uses the decoded string as the key. When the parser encounters "\u0061", it decodes the escape to produce “a”, and the map lookup finds a match with a previously seen raw "a". The error message includes the byte offset of both occurrences, enabling precise diagnostics.

This is a design choice that some parsers skip entirely (encoding/json processes duplicate keys in input order, with later values replacing or merging earlier ones) or implement on raw bytes (which misses escape-decoded equivalence). For canonicalization, decoded comparison is the only correct approach, because canonical output normalizes escape sequences.

Resource Bounds: Seven Independent Limits

The parser enforces seven configurable bounds, each checked at a different point in the parse:

All bounds are fail-fast: the parser stops at the first violation rather than continuing to consume input. The number character bound is checked during digit scanning, not just after:

func (p *parser) scanNonZeroIntegerDigits(numStart int) *jcserr.Error {
    for p.pos < len(p.data) && isDigit(p.data[p.pos]) {
        p.pos++
        if p.pos-numStart > p.maxNumberChars {
            return jcserr.New(jcserr.BoundExceeded, numStart,
                fmt.Sprintf("number token length %d exceeds maximum %d",
                    p.pos-numStart, p.maxNumberChars))
        }
    }
    return nil
}

This prevents a 100 MB number token from being fully scanned before the bound check fires.

Every bound violation returns a BoundExceeded failure class, regardless of which bound was hit. This is a deliberate classification choice: the cause is “resource policy violation,” not “invalid grammar” or “I/O error.” Machines consuming the exit code can distinguish policy rejections from syntax errors.

Error Offset Tracking

Every parser error includes a byte offset pointing to the source position of the violation:

func (p *parser) newError(msg string) *jcserr.Error {
    return jcserr.New(jcserr.InvalidGrammar, p.pos, msg)
}

For most errors, the offset is the parser’s current position, the byte where parsing failed. But for escape sequences and surrogate pairs, the offset points to the start of the escape that caused the problem, not the byte where the violation was detected:

if b == '\\' {
    escapeStart := p.pos   // Record position of the backslash
    p.pos++
    r, err := p.parseEscape(escapeStart)  // Pass source position
    // ...
}

For a lone high surrogate in "\uD800", the offset is 1 (the backslash). For a high surrogate followed by a non-low surrogate in "\uD800\u0041", the offset is 7 (the second backslash). This matters for tooling: an editor or diagnostic tool can highlight the exact source token that caused the rejection.

The offset semantics are stable across the failure taxonomy. Parse errors always report byte positions in the original input. CLI errors use offset -1 (not applicable). Bound violations report the start of the bounded element (e.g., the first byte of a too-long number token).

What Strictness Buys You

A lenient parser makes these implicit decisions:

• “Leading zeros are fine” → Different parsers may interpret 012 as octal 10 or decimal 12
• “Lone surrogates get replaced” → The canonical output of \uD800 is undefined
• “Duplicate keys use last value” → Or first value, depending on implementation
• “Trailing content is ignored” → {"a":1}extra silently becomes {"a":1}

A strict parser converts these implicit decisions into explicit rejections. The downstream consumer never has to wonder whether the input was ambiguous. If it parsed, it has exactly one interpretation. If it didn’t parse, the failure class and byte offset tell the consumer exactly what went wrong and where.

For infrastructure that depends on deterministic processing, this is the difference between “it works in my tests” and “it works because ambiguous input is structurally excluded.”

Strictness as Error Budget

There’s a useful way to think about parser strictness: it’s an error budget. A lenient parser spends its error budget on user convenience, accepting malformed input so users don’t have to fix their data. A strict parser spends its error budget on correctness guarantees, ensuring every accepted input has exactly one interpretation.

Neither is wrong. They serve different purposes. But when you build infrastructure that sits between systems, processing input from one machine and producing output consumed by another, spending the error budget on convenience is spending it on the wrong consumer. The machine downstream doesn’t benefit from leniency. It benefits from guarantees.

The parser code in the jcstoken package exists to provide one guarantee: if the parser returns a value, that value has a single, unambiguous, deterministic canonical representation. Every rejection (leading zeros, lone surrogates, duplicate keys, noncharacters, overflow, underflow, negative zero) eliminates a case where two consumers might disagree.

The implementation lives in the jcstoken package of json-canon, an RFC 8785 JSON canonicalization library. For callers who want a single parse-and-serialize call, jcs.Canonicalize(input) wraps this parser with the RFC 8785 serializer.

Revision History

Most software fails at the center: the core algorithm is wrong, the architecture doesn’t scale, the data model can’t represent the domain. Infrastructure software fails at the margins: the sort order is subtly wrong for one class of inputs, the error code changes in a patch release and breaks a CI pipeline, the upgrade removes a flag that a deployment script depends on.

These marginal failures are harder to prevent because they’re harder to see. They don’t cause test failures during development. They cause production incidents three months after a routine upgrade.

This article examines three categories of these decisions through the lens of a JSON canonicalization library: a correctness detail that affects sort order, a failure classification system that enables machine automation, and an ABI stability discipline that prevents upgrade emergencies. None of these individually is difficult. Together, they define whether downstream systems can depend on you without reservations.

Correctness Detail: UTF-16 vs UTF-8 Key Sorting

RFC 8785 (JSON Canonicalization Scheme) requires that object keys be sorted by lexicographic order of UTF-16 code units. That is not UTF-8 byte order, and it is not Unicode scalar-value order.

For most strings, these orderings are identical. They diverge when you have characters above U+FFFF (the upper boundary of the Basic Multilingual Plane). Here’s why.

Characters above U+FFFF require four bytes in UTF-8 but are encoded as surrogate pairs in UTF-16, as two 16-bit code units. The high surrogate falls in the range U+D800-U+DBFF. Consider two characters:

• U+10000 (LINEAR B SYLLABLE B008 A): UTF-16 encoding is D800 DC00 (surrogate pair). UTF-8 encoding is F0 90 80 80.
• U+E000 (first Private Use Area character): UTF-16 encoding is E000 (single code unit). UTF-8 encoding is EE 80 80.

In UTF-16 code-unit order: U+10000 (D800) < U+E000 (E000). The high surrogate D800 is numerically less than E000.

In UTF-8 byte order: U+E000 (EE) < U+10000 (F0). The three-byte UTF-8 prefix EE is numerically less than the four-byte prefix F0.

The two orderings disagree. RFC 8785 mandates UTF-16 code-unit order (see §3.2.3). Most canonicalization implementations get this wrong because they use byte-order comparison, which works for UTF-8 strings but produces a different order than the specification requires.

Why does RFC 8785 use UTF-16 code-unit order instead of the more natural (for modern systems) Unicode code-point order? Because JCS is defined for interoperability with ECMAScript string serialization rules, and ECMAScript strings are sequences of 16-bit code units. The required sort is a pure lexicographic code-unit comparison, not locale-aware collation (localeCompare is locale-sensitive and is not the JCS rule).

This means every implementation in a language with UTF-8 native strings (Go, Rust, Python 3) must explicitly convert to UTF-16 for comparison. Implementations that skip this conversion will produce correct output for the BMP (U+0000 to U+FFFF) and incorrect output for supplementary-plane characters. Since supplementary-plane characters are rare in practice (emoji, historic scripts, mathematical symbols), the bug is unlikely to surface in casual testing. It will surface when a production system encounters a key containing an emoji or a CJK Extension B character.

The implementation is compact:

func serializeObject(buf []byte, v *jcstoken.Value) ([]byte, error) {
    sorted := make([]sortableMember, len(v.Members))
    for i := range v.Members {
        sorted[i].member = v.Members[i]
        // Fast path: ASCII keys need no UTF-16 encoding since byte order
        // equals UTF-16 code-unit order for U+0000..U+007F.
        if !isASCII(v.Members[i].Key) {
            sorted[i].key16 = utf16.Encode([]rune(v.Members[i].Key))
        }
    }
    sort.Slice(sorted, func(i, j int) bool {
        return compareSortKeys(&sorted[i], &sorted[j]) < 0
    })
    // ... serialize sorted members
}

Non-ASCII keys are converted from Go’s native UTF-8 string to a []uint16 via utf16.Encode([]rune(key)). ASCII-only keys skip this conversion; byte order and UTF-16 code-unit order are identical for U+0000..U+007F. The compareSortKeys function handles both cases, falling back to compareUTF16Units when either key contains non-ASCII characters:

func compareSortKeys(a, b *sortableMember) int {
    if a.key16 == nil && b.key16 == nil {
        if a.member.Key < b.member.Key {
            return -1
        }
        if a.member.Key > b.member.Key {
            return 1
        }
        return 0
    }
    ak := a.key16
    if ak == nil {
        ak = utf16.Encode([]rune(a.member.Key))
    }
    bk := b.key16
    if bk == nil {
        bk = utf16.Encode([]rune(b.member.Key))
    }
    return compareUTF16Units(ak, bk)
}

When both keys are ASCII (key16 == nil), direct string comparison produces identical ordering, with no allocation needed. When either key contains non-ASCII characters, the function falls back to full UTF-16 code-unit comparison via compareUTF16Units. This is the only correct approach for RFC 8785 compliance. The canonical output for an object with keys U+10000 and U+E000 places U+10000 first, the opposite of what a naive byte-order sort would produce.

String Escaping: What Gets Escaped and Why

Canonicalization also requires deterministic string escaping. The rules are specific:

The solidus rule is worth noting. JSON allows \/ as a valid escape, and many serializers produce it. RFC 8785 specifies that solidus is not escaped in canonical output. This is a one-line decision in the implementation, but it means any test that compares canonical output byte-for-byte must agree on this point.

Failure Contracts: Designing a Taxonomy That Machines Can Depend On

Most error handling in CLI tools follows a pattern: print a message to stderr, exit non-zero. The message is for humans. The exit code is a vague signal, usually 1 for “something went wrong.”

This is insufficient for machine automation. When a CI pipeline runs a canonicalization check, it needs to distinguish between “the input was malformed JSON” (a build error) and “stdout couldn’t be written” (an infrastructure problem). The human can read the message. The script needs the exit code.

13 Classes, 3 Exit Codes

The failure taxonomy defines 13 failure classes mapped to 3 exit codes:

The exit code space is deliberately sparse. Codes 0, 2, and 10 leave room for future expansion without breaking existing scripts that check $?. Eleven classes share exit code 2 because they all represent the same decision for automation: “the input or invocation was wrong; don’t retry without changing something.” Exit code 1 is intentionally avoided; many shells and tools use 1 as a generic failure code, and conflating tool-specific failures with generic failure would reduce the signal.

The two exit-10 classes represent a fundamentally different situation: the tool itself encountered a problem. A CI pipeline should fail the build on exit 2 (bad input is a build error) but might alert ops on exit 10 (the build infrastructure has a problem).

Why 13 classes instead of 3? Because the exit code is a coarse signal for automation, while the class is a fine-grained signal for diagnostics. A script checks the exit code. A log aggregator or monitoring system can parse the class name from stderr. Having DUPLICATE_KEY as a distinct class from INVALID_GRAMMAR lets a monitoring dashboard show “we’re seeing a spike in duplicate-key rejections” without confusing it with syntax errors.

Root-Cause Classification

The classification principle is: classify by root cause, not by error origin.

The clearest example: a missing file path. The user runs jcs-canon canonicalize /nonexistent.json and the file doesn’t exist. This is an os.Open error. An I/O error. So it should be INTERNAL_IO, right?

No. It should be CLI_USAGE (exit 2).

The root cause is that the user supplied an invalid argument. The file path is user input, just like the JSON content. When the path cannot be opened, the problem is the same as providing an unknown flag or specifying two input files. The invocation is wrong.

INTERNAL_IO is reserved for failures after a valid I/O channel is established. If os.Open succeeds but a later Read fails due to a broken pipe, or if writing to stdout fails because the pipe was closed. Those are infrastructure problems. The channel was valid; something broke during operation.

This distinction matters for automation:

jcs-canon canonicalize input.json
case $? in
  0)  echo "Success" ;;
  2)  echo "Input or usage problem - fix the invocation" ;;
  10) echo "Infrastructure problem - investigate the environment" ;;
esac

The Structured Error Type

Every error in the system is a *jcserr.Error with four fields:

type Error struct {
    Class   FailureClass   // Stable category (determines exit code)
    Offset  int            // Source-byte position, or -1
    Message string         // Human-readable diagnostic
    Cause   error          // Underlying error (for wrapping)
}

The Class field is the stable contract. The Message field is not stable; its wording may change in minor releases. This is explicit policy: machines should switch on the class, not parse the message string. The Offset field gives byte-precise error positions for parse failures, enabling editors and diagnostic tools to highlight the exact location.

Offset Semantics

The offset field deserves its own discussion because it’s an area where most error reporting gets lazy.

For a simple parse error like a leading zero in {"n":01}, the offset points to the 0 at the start of the number token. Straightforward. But for errors inside escape sequences, the offset must point to the originating escape, not the byte where the violation was detected.

Consider the string "\uD800\u0041". This contains a high surrogate (U+D800) followed by a non-low-surrogate (U+0041). The parser detects the error when reading the second escape sequence, but the error offset points to byte 7 (the backslash of \u0041), not to byte 1 (the backslash of \uD800). This is because the violation is in the second escape: a high surrogate was followed by U+0041 instead of a low surrogate. Pointing to the first escape would be misleading: \uD800 isn’t inherently wrong; it’s wrong because of what follows.

Conversely, for a lone low surrogate like "\uDC00", the offset points to byte 1 (the backslash) because the low surrogate appearing without a preceding high surrogate is the violation.

These offset semantics are stable across the failure taxonomy and documented as part of the error contract. Changing where an offset points is a behavioral change, even if the class and message remain the same.

The error format is also deliberate:

jcserr: INVALID_GRAMMAR at byte 15: leading zero in number
jcserr: CLI_USAGE: read file "config.json": no such file or directory
jcserr: INTERNAL_IO: writing output: write: broken pipe

The class name always appears after jcserr:. Byte offsets appear only for parse-time errors (where they’re meaningful). The cause chain preserves the underlying error for debugging without losing the classification.

ABI Stability: Preventing Upgrade Emergencies

For a CLI tool consumed by scripts and CI pipelines, the interface is the product. Changing a flag name, moving output between stdout and stderr, or redefining an exit code breaks every downstream consumer. These breakages are invisible in your test suite and visible only in your users’ build failures.

Defining the Stable Surface

The stable ABI surface includes:

1. Command names: canonicalize, verify
2. Flag names and semantics: --quiet, --help, --version
3. Exit code mapping: 0/2/10 with defined semantics
4. Output stream placement: canonical data to stdout, diagnostics to stderr, verify result to stderr
5. Failure class names in stderr: The class name INVALID_GRAMMAR is stable; the surrounding message text is not
6. Canonical output bytes: identical input must produce identical stdout bytes

Notably absent from the stable surface: the exact wording of error messages and help text. These are explicitly non-stable, allowing improvements to diagnostics without a major version bump.

There is one subtlety here: failure class names that appear in stderr (e.g., INVALID_GRAMMAR in the error output) are stable, even though the surrounding message text is not. This lets machines extract the class name from stderr as a fallback when they can’t use the exit code alone, for instance when a script needs to distinguish between INVALID_GRAMMAR and DUPLICATE_KEY, both of which share exit code 2.

The stream placement policy is also part of the stable surface. The canonicalize command writes canonical bytes to stdout and diagnostics to stderr. The verify command writes nothing to stdout and writes ok\n to stderr on success (suppressible with --quiet). If a future version moved the verify result from stderr to stdout, every script that captures verify output would break. This is why stream placement is documented, tested, and governed by SemVer.

The Machine-Readable Contract

The ABI is documented in two forms. ABI.md is the human-readable specification. abi_manifest.json is the machine-readable contract:

{
  "abi_version": "1.0.0",
  "tool": "jcs-canon",
  "commands": {
    "canonicalize": {
      "stable": true,
      "flags": {
        "--quiet": {"short": "-q", "stable": true},
        "--help": {"short": "-h", "stable": true}
      },
      "stdout": "Canonical JSON bytes (on success)",
      "stderr": "Error diagnostics (on failure)",
      "exit_codes": [0, 2, 10]
    },
    "verify": {
      "stable": true,
      "flags": {
        "--quiet": {"short": "-q", "stable": true},
        "--help": {"short": "-h", "stable": true}
      },
      "stdout": "Empty (verify never writes to stdout)",
      "stderr": "'ok\\n' on success (unless --quiet), error diagnostics on failure",
      "exit_codes": [0, 2, 10]
    }
  },
  "exit_codes": {
    "0": {"class": "SUCCESS"},
    "2": {"class": "INPUT_REJECTION"},
    "10": {"class": "INTERNAL_ERROR"}
  },
  "failure_classes": [
    {"name": "INVALID_UTF8", "exit_code": 2},
    {"name": "INVALID_GRAMMAR", "exit_code": 2},
    {"name": "DUPLICATE_KEY", "exit_code": 2},
    ...
  ],
  "compatibility": {
    "policy": "Strict SemVer. Any change to items marked 'stable: true' requires major version bump.",
    "stderr_wording": "Non-stable. Diagnostic message text may change in minor/patch releases.",
    "exit_code_mapping": "Stable. Failure class → exit code mapping is frozen."
  }
}

This file serves two purposes. First, it’s a test fixture: CI validates that the tool’s actual behavior matches the manifest’s claims. Second, it’s a communication device for downstream consumers. A script author can read the manifest to understand what’s stable, what might change, and what exit codes to expect, without parsing prose documentation.

Why a Manifest Instead of Just Documentation

A reasonable question: why maintain both ABI.md (human-readable) and abi_manifest.json (machine-readable)? Isn’t the documentation sufficient?

Documentation drifts. When a developer adds a flag, they update the code and the tests, but they may forget to update the ABI documentation. The manifest is testable: CI can validate that the tool’s actual behavior matches the manifest’s claims. A test that parses abi_manifest.json and checks that jcs-canon verify --help exits 0, that jcs-canon verify produces ok\n on stderr for valid canonical input, and that error output goes to stderr, will catch ABI drift that documentation review would miss.

The manifest also serves as a communication device. A downstream consumer can curl the manifest from a release and programmatically determine what the stable surface looks like. This is more reliable than scraping a README.

Change Control

The SemVer rules are strict and non-negotiable:

• Patch releases (0.2.0 → 0.2.1): No behavior changes to stable surface items
• Minor releases (0.2.x → 0.3.0): May add new commands or flags, but existing behavior is preserved
• Major releases (0.x.x → 1.0.0): May change anything, with migration guidance

Any ABI-impacting change must update four artifacts in the same commit series: the implementation, abi_manifest.json, test assertions, and CHANGELOG.md. This co-update requirement prevents the manifest from drifting out of sync with the code. A test that validates the manifest against actual behavior catches drift that documentation alone would miss.

The Compound Effect

Each of these decisions is individually minor. None of them requires novel engineering. None of them is exciting. A project that omits all three still works. It parses JSON, it serializes JSON, it runs from the command line.

But the project that includes all three communicates something different to its consumers. It says: “We have thought about the cases you will encounter. The sort order is correct, not just close. The exit codes mean what we say they mean, and we won’t change them without a major version. The error you get for a missing file is classified by root cause, not by which system call failed.”

Infrastructure trust is not earned by one big decision. It’s earned by consistent attention to the decisions that are easy to defer and hard to fix later. The sort order that’s wrong by one comparison. The exit code that changes in a patch release. The error message that scripts parse because there’s no stable classification.

These are the margins where infrastructure fails.

Consider the alternative. A project that uses UTF-8 byte-order sorting passes 99.9% of real-world key comparisons correctly; supplementary-plane characters in JSON keys are rare. But the 0.1% failure is a silent correctness bug, not a crash. The canonical output is wrong for those inputs, and the consumer has no way to detect it without an independent reference implementation. This is the kind of bug that ships, is discovered months later, and then can’t be fixed without a breaking change because existing consumers depend on the (wrong) output.

A project with unstructured exit codes works fine for human use. But the first time someone writes a CI pipeline that retries on exit 1, they’ll retry on parse errors (which will never succeed) and on I/O errors (which might). Distinguishing these requires parsing stderr text, which breaks when the message wording changes in a patch release.

A project without an ABI manifest accumulates undocumented behavioral changes. Each one seems minor: renaming a flag, changing where help text goes, adding a new exit code. But each one is a potential breaking change for a downstream consumer who discovered the behavior empirically and depended on it.

Getting these details right is not the interesting part of the engineering. But it’s the part that determines whether your downstream can depend on you.

The implementation discussed here is from json-canon, an RFC 8785 JSON canonicalization library written in Go.

Revision History

For a JSON canonicalization library, determinism is the product. If the same input produces different bytes on Ubuntu vs Alpine, on x86_64 vs arm64, or on the third run vs the first run, the tool is broken, regardless of what the test suite says.

This article describes how to build an offline replay harness that produces executable evidence of determinism across distributions, kernels, and CPU architectures, and how to gate releases on that evidence. The approach is transferable to any project where output stability matters.

The Problem: “It Works on My Machine”

Unit tests prove that functions return correct values. Integration tests prove that components work together. Neither proves that the tool produces identical output across environments.

Consider what can differ between two Linux machines running the same Go binary:

• C library: glibc (Ubuntu, Debian, Fedora) vs musl (Alpine). Go is statically compiled, so this shouldn’t matter, but “shouldn’t” is not proof.
• Kernel version: syscall behavior, filesystem semantics, memory layout. Go’s runtime abstracts these, but abstractions can leak.
• CPU architecture: x86_64 vs arm64. Floating-point rounding, SIMD optimizations, and endianness. Go generates architecture-specific code.
• Runtime initialization: Go’s runtime performs memory allocation, goroutine scheduling, and GC initialization. Any of these could influence program behavior if the code contains latent non-determinism.

For a canonicalization tool, any of these differences producing a different output byte is a correctness failure. You can’t prove the absence of these failures by testing on one machine. You need to test on many machines and compare the results.

What Evidence Looks Like

The harness runs the tool against a fixed set of inputs on multiple nodes: different Linux distributions, container and VM execution modes, x86_64 and arm64. Each node runs the tool multiple times. Every run records SHA-256 digests of its output. At the end, a validation function checks that every digest matches.

The output is a JSON evidence bundle:

{
  "schema_version": "evidence.v1",
  "bundle_sha256": "5654748feaa65318...",
  "control_binary_sha256": "e0296e034d1440a4...",
  "source_git_commit": "da4a4ee6fcefc4f4...",
  "source_git_tag": "v0.2.1",
  "architecture": "x86_64",
  "hard_release_gate": true,
  "node_replays": [
    {
      "node_id": "alpine320-container",
      "distro": "alpine-3.20",
      "replay_index": 1,
      "case_count": 74,
      "passed": true,
      "canonical_sha256": "2818166c21e1b445...",
      "verify_sha256": "66d329b3bd829da5...",
      "failure_class_sha256": "af58643f979138da...",
      "exit_code_sha256": "73d91ef3f2fd6d70..."
    }
  ],
  "aggregate_canonical_sha256": "2818166c21e1b445...",
  "aggregate_verify_sha256": "66d329b3bd829da5..."
}

When all 60 replays (12 nodes times 5 replays each) produce identical digests, the evidence shows that canonical output is byte-stable for that specific binary across the tested matrix.

Proof Boundary: What This Evidence Proves

This matters for rigor. The evidence bundle does not claim:

• “all possible builds are deterministic,”
• “all Linux environments are covered,”
• or “future toolchains will preserve behavior automatically.”

It does claim:

• this exact control binary (by SHA-256),
• built from this exact source commit,
• produced identical output across this explicit replay matrix and replay count.

That distinction is the boundary between engineering evidence and overclaiming. Determinism is established for a concrete artifact under a declared environment envelope.

Test Bundles: Immutable Input Packages

The first requirement is immutable inputs. If the test data can change between runs, comparing digests is meaningless. The harness packages all inputs into a tar archive with properties that eliminate environmental variation.

Fixed Timestamps and Ownership

func writeBundleTarGz(path string, entries []bundleEntry) error {
    // ...
    fixed := time.Unix(0, 0).UTC()
    for _, e := range entries {
        hdr := &tar.Header{
            Name:    e.path,
            Mode:    e.mode,
            Size:    int64(len(e.data)),
            ModTime: fixed,   // Unix epoch: 1970-01-01T00:00:00Z
            Uid:     0,       // root
            Gid:     0,       // root
            Uname:   "root",
            Gname:   "root",
        }
        tw.WriteHeader(hdr)
        tw.Write(e.data)
    }
}

Every entry in the tar archive has its modification time set to the Unix epoch, its ownership set to root:root, and its permissions set by the code rather than the filesystem. This means the archive is byte-identical regardless of when, where, or by whom it was created.

Sorted Entries

Tar archives are ordered. If entries are added in filesystem order, the archive depends on the filesystem’s iteration behavior, which can vary between ext4 and xfs, between Linux kernel versions, and between NFS and local disk. The harness sorts entries lexicographically before writing:

sort.Slice(entries, func(i, j int) bool {
    return entries[i].path < entries[j].path
})

SHA-256 Binding

The bundle manifest records SHA-256 checksums for every component:

type BundleManifest struct {
    Version         string            `json:"version"`
    BinaryPath      string            `json:"binary_path"`
    BinarySHA256    string            `json:"binary_sha256"`
    WorkerPath      string            `json:"worker_path"`
    WorkerSHA256    string            `json:"worker_sha256"`
    MatrixPath      string            `json:"matrix_path"`
    MatrixSHA256    string            `json:"matrix_sha256"`
    ProfilePath     string            `json:"profile_path"`
    ProfileSHA256   string            `json:"profile_sha256"`
    VectorFiles     []string          `json:"vector_files"`
    VectorSHA256    map[string]string `json:"vector_sha256"`
    VectorSetSHA256 string            `json:"vector_set_sha256"`
}

The binary, worker, matrix, profile, and each vector file have independent checksums. The VectorSetSHA256 is a digest of all vector file checksums combined (sorted by path), so changing any single vector file changes the set digest.

The bundle archive itself also gets a SHA-256 checksum, which the evidence bundle records. This creates a chain: the evidence references the bundle by digest, the bundle references each component by digest, and the release gate validates all of these against the actual files on disk.

This chain has the same integrity property as certificate chains: modifying any component invalidates every layer above it. If someone edits a single vector file, its SHA-256 changes, which changes the vector set SHA-256, which changes the bundle manifest, which changes the bundle archive SHA-256, which no longer matches the evidence bundle’s recorded value. The release gate catches this at the top of the chain without needing to know which component was modified.

The Replay Matrix: Defining “Across Environments”

A matrix defines the nodes that must be tested:

{
  "architecture": "x86_64",
  "nodes": [
    {
      "id": "debian12-container",
      "mode": "container",
      "distro": "debian-12",
      "kernel_family": "host",
      "replays": 5
    },
    {
      "id": "alpine320-container",
      "mode": "container",
      "distro": "alpine-3.20",
      "kernel_family": "host",
      "replays": 5
    },
    {
      "id": "debian12-vm",
      "mode": "vm",
      "distro": "debian-12",
      "kernel_family": "distro",
      "replays": 5
    }
  ]
}

Each node has a mode (container or VM), a distribution, a kernel family, and a replay count. Container nodes share the host kernel but differ in userspace (glibc vs musl, different library versions). VM nodes run their own kernels.

The distinction matters: container-mode tests prove that userspace differences don’t affect output. VM-mode tests prove that kernel differences don’t affect output. Together, they cover the two main sources of environmental variation on Linux.

A typical x86_64 matrix includes 12 nodes: 6 container lanes (Debian 12, Ubuntu 22.04, Fedora 40, Rocky 9, Alpine 3.20, openSUSE) and 6 VM lanes (Debian, Fedora, Rocky, Ubuntu with GA kernel, Ubuntu with HWE kernel, and a legacy LTS kernel). Each runs 5 replays, for a total of 60 independent executions.

Environment Pinning and Reproducibility Boundaries

Matrix declarations define which lanes must run. Reproducibility additionally depends on how those lanes are provisioned over time. If a lane references a moving base image tag, the lane identity remains the same while underlying bits may drift.

For peer-review-grade replay reproducibility, pin lane substrates as immutable artifacts (for example: container image digests and checksummed VM base images/snapshots) and record those identifiers alongside evidence. Without that, evidence still proves parity for the observed run, but re-running months later may exercise different substrate bits.

A profile defines the policy for what constitutes a valid evidence bundle:

{
  "name": "maximal-offline-linux-x86_64",
  "required_suites": [
    "canonical-byte-stability",
    "verify-parity",
    "failure-class-parity",
    "bounds-limit-parity",
    "binary-identity",
    "env-independence",
    "evidence-completeness"
  ],
  "min_cold_replays": 5,
  "hard_release_gate": true,
  "evidence_required": true,
  "version": "v1"
}

The profile enforces that every required node runs at least 5 times, that the evidence includes all required test suites, and that the evidence is a hard release gate (not advisory).

Evidence Capture: What to Record

Each worker node runs the tool against every test vector and accumulates four independent digest streams:

1. Canonical digest: SHA-256 of all canonical output bytes, concatenated with vector IDs
2. Verify digest: SHA-256 of verify mode results (exit code, stdout, stderr per vector)
3. Failure class digest: SHA-256 of failure class tokens (“OK” or the class name) per vector
4. Exit code digest: SHA-256 of numeric exit codes per vector

These four streams capture different properties. The canonical digest proves byte-identical output. The verify digest proves that the verify command agrees. The failure class digest proves that error classification is stable. The exit code digest proves that the process-level interface is stable.

The digest accumulation works by concatenating structured records with delimiters, then computing a single SHA-256 over the entire stream. Each record includes the vector ID and the relevant output, separated by a unit separator (0x1F) and terminated by a newline. This produces a deterministic input to SHA-256 regardless of record ordering (vectors are processed in sorted order) or platform-specific line ending behavior.

Separating the four digest streams matters for diagnostics. If the canonical digest matches but the failure class digest doesn’t, the tool is producing the same output but classifying errors differently, which could indicate a failure taxonomy change that wasn’t intentional. If the exit code digest matches but the verify digest doesn’t, the tool exits correctly but produces different stderr text, which is acceptable if the stderr change is non-stable, but should be investigated.

Source Binding

The evidence bundle records the exact source state:

{
  "source_git_commit": "da4a4ee6fcefc4f43777c76e5235d824d249807c",
  "source_git_tag": "v0.2.1",
  "control_binary_sha256": "e0296e034d1440a4aad2a3620e5663d749c2007f..."
}

The git commit SHA pins the source code. The binary SHA-256 pins the compiled artifact. The git tag identifies the release. Together, these create an audit trail from evidence back to source code, with no ambiguity about which code produced the evidence.

Validation Logic: Detecting Drift

The validation function implements a simple invariant: all replays must produce identical digests.

func ValidateEvidenceBundle(e *EvidenceBundle, m *Matrix, p *Profile,
    opts EvidenceValidationOptions) error {

    // ... schema version, profile match, SHA-256 format checks ...

    var baseline *NodeRunEvidence
    for _, id := range requiredNodes {
        runs := byNode[id]
        wantReplays := requiredReplayCount(matrixByID[id], p)
        if len(runs) < wantReplays {
            return fmt.Errorf("node %s has %d replays, want at least %d",
                id, len(runs), wantReplays)
        }

        for _, run := range runs {
            if baseline == nil {
                r := run
                baseline = &r
                continue
            }
            if run.CanonicalSHA256 != baseline.CanonicalSHA256 {
                return fmt.Errorf("canonical digest drift at node %s replay %d",
                    run.NodeID, run.ReplayIndex)
            }
            if run.VerifySHA256 != baseline.VerifySHA256 {
                return fmt.Errorf("verify digest drift at node %s replay %d",
                    run.NodeID, run.ReplayIndex)
            }
            if run.FailureClassSHA256 != baseline.FailureClassSHA256 {
                return fmt.Errorf("failure-class digest drift at node %s replay %d",
                    run.NodeID, run.ReplayIndex)
            }
            if run.ExitCodeSHA256 != baseline.ExitCodeSHA256 {
                return fmt.Errorf("exit-code digest drift at node %s replay %d",
                    run.NodeID, run.ReplayIndex)
            }
        }
    }

    // Aggregate digests must match baseline
    if e.AggregateCanonical != baseline.CanonicalSHA256 {
        return fmt.Errorf("aggregate canonical digest mismatch")
    }
    // ... verify, failure-class, exit-code aggregates ...
}

The first replay becomes the baseline. Every subsequent replay (across all nodes, all distributions, all execution modes) is compared against this baseline. Any divergence is an immediate failure with a message identifying the exact node and replay index where drift was detected.

The aggregate digests provide a summary check: four SHA-256 values that represent the behavior of the entire run. If the aggregates match the baseline’s per-node digests, all nodes agreed.

What the Validation Checks

In addition to digest parity, the validation function enforces:

• Schema version: Must be evidence.v1 (enables future schema evolution)
• Profile match: Evidence profile name must match the policy profile
• SHA-256 format: All digest fields must be exactly 64 hex characters
• Git commit format: Must be exactly 40 hex characters
• Architecture match: Evidence architecture must match the matrix
• Artifact binding: Bundle, binary, matrix, and profile SHA-256s must match the actual files
• Replay coverage: Every required node must have at least the minimum replay count
• Replay contiguity: Replay indices must be 1, 2, 3, …, N (no gaps)
• Pass status: Every replay must be marked passed: true
• Suite coverage: Required test suites must match the profile exactly

Release Gating: The Go Test That Says No

The release gate is a standard Go test function. It loads the evidence bundle, the matrix, and the profile, then calls the validation function with expected SHA-256 values computed fresh from the actual artifacts:

func TestOfflineReplayEvidenceReleaseGate(t *testing.T) {
    evidencePath := os.Getenv("JCS_OFFLINE_EVIDENCE")
    if evidencePath == "" {
        t.Skip("set JCS_OFFLINE_EVIDENCE to validate offline evidence bundle")
    }

    bundlePath := os.Getenv("JCS_OFFLINE_BUNDLE")
    controlBinaryPath := os.Getenv("JCS_OFFLINE_CONTROL_BINARY")
    // ... load matrix, profile, evidence ...

    if err := replay.ValidateEvidenceBundle(evidence, matrix, profile,
        replay.EvidenceValidationOptions{
            ExpectedBundleSHA256:        mustFileSHA256(t, bundlePath),
            ExpectedControlBinarySHA256: mustFileSHA256(t, controlBinaryPath),
            ExpectedMatrixSHA256:        mustFileSHA256(t, matrixPath),
            ExpectedProfileSHA256:       mustFileSHA256(t, profilePath),
            ExpectedArchitecture:        matrix.Architecture,
            ExpectedSourceGitCommit:     os.Getenv("JCS_OFFLINE_EXPECTED_GIT_COMMIT"),
            ExpectedSourceGitTag:        os.Getenv("JCS_OFFLINE_EXPECTED_GIT_TAG"),
        }); err != nil {
        t.Fatalf("offline evidence gate failed: %v", err)
    }
}

The test is gated by an environment variable. In normal development, it’s skipped. During release, CI sets the variable and the test becomes a hard gate. The SHA-256 values are computed fresh from the files on disk; they’re not hardcoded. This means the test validates that the evidence bundle references the actual artifacts being released, not some previously valid set.

Environment Variable Binding

The release gate uses seven environment variables:

Most have sensible defaults: matrix and profile default to the repository’s standard files; bundle and control binary paths are derived from the evidence path’s directory. The evidence path has no default. It must be explicitly provided, which prevents accidental release without evidence.

The design as a go test function (rather than a standalone script) is intentional. It integrates with Go’s standard testing infrastructure: go test -v shows progress, -run selects specific gates, -count=1 disables caching. The test is part of the same codebase as the tool it validates, which means the validation logic is versioned alongside the evidence schema. And because it’s a Go test, it can import the same replay package that generates the evidence, ensuring the validation code and generation code share type definitions.

The t.Skip pattern (skip when the environment variable is absent, fail when it’s present but the evidence is invalid) means the gate is silent during normal development and enforced during release. Developers running go test ./... don’t see the offline gate. The release pipeline, which sets the environment variables, does.

Schema Versioning

The evidence schema is versioned independently from the tool version. The current schema is evidence.v1. The validation function checks the schema version as its first action:

if e.SchemaVersion != EvidenceSchemaVersion {
    return fmt.Errorf("unsupported schema_version %q", e.SchemaVersion)
}

This enables schema evolution without invalidating existing evidence. A future evidence.v2 could add new fields (node CPU architecture, memory constraints, filesystem type) without breaking the validation of v1 evidence bundles. The validation function would branch on the schema version and apply the appropriate checks for each.

The schema is also defined as a JSON Schema file (offline/schema/evidence.v1.json), enabling validation by external tools. Any system that consumes evidence bundles can validate them against the schema independently of the Go validation code.

Cross-Architecture Parity

The same harness runs independently for x86_64 and arm64, each with its own matrix, profile, and evidence bundle. The CI pipeline runs both and validates both independently.

Cross-architecture parity is not required to match; the aggregate digests between x86_64 and arm64 are compared separately. This is a deliberate design choice. Go’s runtime, floating-point behavior, and standard library may produce different intermediate results on different architectures. What matters is that each architecture is internally consistent: all x86_64 nodes agree with each other, and all arm64 nodes agree with each other.

If cross-architecture aggregate digests do match (which they do in practice for this tool, since the implementation uses pure integer arithmetic and explicit formatting), that’s additional confidence. But the harness doesn’t make it a requirement, because mandating it would create false failures if a future Go release changed architecture-specific behavior in a standard library function.

What Running 5+ Times Per Node Proves

The replay count (5 per node in the maximal profile) is not arbitrary. A single run proves the tool works. Multiple runs prove it works deterministically.

Non-determinism in software can come from several sources: uninitialized memory, map iteration order, concurrent goroutine scheduling, time-dependent behavior, or environment-sensitive code paths. A single run may happen to hit the “correct” ordering every time. Multiple cold runs (starting from a fresh process each time, with no warm caches) increase the probability of surfacing non-deterministic behavior.

Five cold replays is a pragmatic balance between coverage and execution time. Each replay runs the full vector suite from a fresh process invocation, exercising the tool’s startup path, parser initialization, and output formatting from scratch. If any of these paths contain non-deterministic behavior, five independent executions have a reasonable chance of producing divergent digests.

Why This Matters: Evidence as a First-Class Artifact

Most release processes treat testing as a gate: tests pass, the release ships. The evidence is a test log: ephemeral, human-consumed, not structured for machine validation.

Evidence-based release engineering treats evidence as a first-class artifact: versioned, checksummed, machine-readable, and committed to the repository alongside the code it validates. The evidence for v0.2.1 is available at the same commit as the v0.2.1 source code. Anyone can re-validate the release gate by running a single go test command with the evidence path.

This approach has three practical benefits:

1. Auditability: The evidence bundle is a complete record of what was tested, on what environments, at what time, from what source. There’s no ambiguity about whether the tests actually ran or what they covered.

2. Reproducibility: The bundle contains everything needed to reproduce the test: the binary, the vectors, the matrix. Re-running the harness with the same bundle should produce the same evidence (modulo wall-clock timestamps).

3. Trust: The SHA-256 chain from evidence to bundle to binary to source code means each layer’s integrity is independently verifiable. Tampering with any component breaks the chain.

The cost is real: maintaining the harness, running replays across multiple environments, committing evidence bundles to the repository. The evidence bundle for a single architecture is approximately 1,000 lines of JSON. The bundle archive contains the test binary, worker binary, all vector files, the matrix, and the profile. Running the full matrix takes minutes, not seconds.

For most projects, this is overkill. A well-written test suite with good coverage provides sufficient confidence for application software. But “sufficient confidence” and “proof” are different claims. When your README says “byte-deterministic output,” the evidence bundle makes that claim auditable. Anyone can examine the evidence, verify the SHA-256 chain, and confirm that 60 independent executions across 12 environments produced identical output.

For infrastructure that downstream systems depend on for correctness (not just convenience), this is the minimum required to make the claim credible.

The implementation described here is from json-canon, an RFC 8785 JSON canonicalization library written in Go.

Revision History