Float Tolerances in NumPy Testing

While writing tests for a few mathematical operations, I ran into a deceptively small question: what tolerance should I use when asserting equality between two floating-point results?

This was one of those questions where I started with a concrete annoyance in a test file and ended up reading about IEEE 754, machine epsilon, ULPs, subnormals, rtol, atol, and the slightly different defaults between np.allclose and np.testing.assert_allclose. Floating-point comparisons have that quality. The surface area looks tiny, and then the floor opens up a bit.

This is not meant to be a complete numerical analysis guide. It is more of a field note: the mental model I wish I had loaded into my head before tightening and loosening tolerances semi-randomly.

float32 and float64

IEEE 754 floating-point numbers allocate bits across three components: sign, exponent, and fraction.

The number of significant bits includes the implicit leading bit, so float32 has 24 significant bits rather than 23. To convert that rough precision into decimal digits:

So float32 gives you about 7 significant decimal digits. That could mean 1.234567 or 123.4567; it is about significant digits, not decimal places.

This is one of the first traps for my intuition. I tend to read 1e-7 as “very small” in an absolute sense, but for float32 it is also roughly the scale at which the representation itself starts losing distinct values near 1.0.

Machine epsilon and ULPs

Machine epsilon is the gap between 1.0 and the next representable number of a given floating-point type. For float32, the machine epsilon is:

In NumPy, I usually do not want to remember that number by hand:

import numpy as np

np.finfo(np.float32).eps
# 1.1920929e-07

The caveat is that this gap is local to values near 1.0. Floating-point numbers are spaced more sparsely as their magnitude grows, so the relevant “gap” depends on the value you are looking at. You can inspect the next representable value with np.nextafter:

x = np.float32(1_000_000_000)
np.nextafter(x, np.float32(np.inf), dtype=np.float32)
# 1000000060.0

That example is a little rude, but useful: at this magnitude, float32 cannot represent every nearby integer. The spacing is much larger than it was near 1.0.

This is where ULP, or Unit in the Last Place, becomes useful. ULP is the gap between adjacent representable floating-point values at a particular magnitude. Machine epsilon is basically the ULP near 1.0. For other values, NumPy gives a nice probe:

np.spacing(np.float32(1.0))
# 1.1920929e-07

np.spacing(np.float32(1000.0))
# 6.1035156e-05

The practical lesson is that “close enough” should usually scale with the magnitude of the values being compared. A tolerance that is sensible near 1.0 can be nonsense near 1e9.

The rtol and atol equation

The core comparison used by np.allclose and np.testing.assert_allclose is:

Stare at this equation for a few seconds and a few things fall out:

1. rtol dominates when desired is large.
2. atol dominates when desired is near zero.
3. When desired is exactly zero, the relative term vanishes.
4. There is no single default that works well for both large values and near-zero values.

This is also why the defaults can feel a bit surprising. As of the NumPy docs I checked, np.testing.assert_allclose defaults to rtol=1e-7 and atol=0, while np.allclose defaults to rtol=1e-5 and atol=1e-8.

That difference makes some historical and practical sense. assert_allclose is for tests, so it is stricter. But the strictness can be annoying when the value you expect is zero:

np.testing.assert_allclose(actual, 0.0)

With atol=0, this asks for exact closeness to zero except for the relative term, and the relative term is useless because desired == 0. In practice, if I am comparing against zero, I now try to write the absolute tolerance explicitly:

np.testing.assert_allclose(actual, 0.0, atol=1e-6, rtol=0)

Setting rtol=0 is not mathematically necessary when desired is exactly zero, but I like that it says what I mean: this is an absolute comparison.

A small table, with caveats

I am wary of giving recommended tolerances because the whole point is that the right tolerance depends on the dtype, operation, and scale of the result.

Still, these are the rough values I first reached for:

I would treat this table as a starting point, not as a rule. It is usually better to define tolerances in terms of the scale of the values and the machine epsilon of the dtype:

FLOAT32_EPS = np.finfo(np.float32).eps
FLOAT32_TOL = 10 * FLOAT32_EPS

Why 10 * eps? In the tests I was working on, a tolerance closer to 2 ULPs was too tight, while 10 ULPs was stable. I would not generalize that too far, but it is at least a more meaningful number than a random 1e-6 pasted into every assertion.

There is also some platform variance to keep in mind. Identical-looking floating-point operations can round differently depending on hardware and compiler choices, especially when SIMD or fused multiply-add enters the picture. I do not know this layer deeply enough to say much more, but I have seen enough “passes on my machine, fails in CI” numerical tests to leave a little room for it.

Subnormals and Hypothesis

Another trap I ran into was subnormal floats.

Normal floats have an implicit leading 1 in the significand:

value = 1.mantissa x 2^(exponent - bias)

Subnormal floats drop that leading 1:

value = 0.mantissa x 2^(1 - bias)

This lets float32 represent values smaller than the smallest normal positive number. NumPy exposes both:

np.finfo(np.float32).smallest_normal
# 1.1754944e-38

np.finfo(np.float32).smallest_subnormal
# 1e-45

This became relevant because Hypothesis can generate subnormal floats. A vector like this is technically non-zero:

np.array([1.2e-40, 0.0, 0.0], dtype=np.float32)

But for many geometric operations, it is basically a degenerate input wearing a tiny numerical disguise. If the behavior you care about is not “what happens to normalization under subnormal stress”, it can be cleaner to filter those inputs out at the strategy level:

from hypothesis import strategies as st
from hypothesis.extra.numpy import arrays

FLOAT32_NORM_FLOOR = np.finfo(np.float32).smallest_normal

float32_vector = arrays(
    dtype=np.float32,
    shape=3,
    elements=st.floats(min_value=-1e6, max_value=1e6, width=32),
)

nondegenerate_vectors = float32_vector.filter(
    lambda v: np.linalg.norm(v) >= FLOAT32_NORM_FLOOR
)

Two details matter here:

1. width=32 asks Hypothesis to generate values representable as float32, instead of generating wider floats and casting them down.
2. Filtering at smallest_normal removes exact zero and subnormal values while still leaving a very wide range of normal float32 values.

This is a good example of a broader testing habit: sometimes the right fix is not a looser tolerance. Sometimes the right fix is a clearer input domain.

Adaptive absolute tolerances

For vector operations, a fixed atol can be wrong in both directions.

If the result can range from near zero to very large, atol=1e-6 might be too tight for large vectors and too loose for small ones. A pattern I found more honest was:

FLOAT32_TOL = 10 * np.finfo(np.float32).eps

atol = max(FLOAT32_TOL * np.linalg.norm(v), FLOAT32_TOL)
np.testing.assert_allclose(actual, desired, atol=atol, rtol=0)

The first term scales with the magnitude of the result. The second term is a floor, so the test never demands more precision than the dtype can reasonably give near zero.

For operations whose error scales with two inputs, such as some projection-like calculations, the same idea can extend to both magnitudes:

atol = max(
    FLOAT32_TOL * np.linalg.norm(a) * np.linalg.norm(b),
    FLOAT32_TOL,
)

This is not a universal formula. It is a way of making the tolerance encode the scale of the operation, which is the part I care about.

A few testing habits I kept

The version of this note I originally wrote for coworkers had a lot of repo-specific commentary. For my own future reference, the distilled habits are:

1. Use np.testing.assert_allclose for most numerical assertions.
2. Set atol deliberately whenever the expected value may be zero.
3. Consider rtol=0 when the assertion is really an absolute near-zero check.
4. Avoid == and exact array equality for floating-point results unless bitwise identity is truly the thing being tested.
5. Prefer input filtering over tolerance gymnastics when generated test inputs are degenerate for the behavior under test.
6. Define tolerance constants from np.finfo(dtype).eps when possible, rather than scattering unexplained decimal constants across tests.

The bigger lesson is that tolerances are part of the test’s specification. They are not just a way to make a flaky assertion pass. A good tolerance says something about the dtype, the scale of the values, the operation being tested, and what kind of numerical error the code is allowed to have.

I still find this slightly annoying. But now it is the productive kind of annoying: the kind that leaves behind a better mental model and fewer mysterious CI failures.