There is a huge misconception here which really should be clarified in the title. Erlang has an "exactly equal" or strict equality operator, called =:=, which is different from just usual equality, which is ==. The usual equality operator will keep having +0.0 = -0.0, and floating point arithmetic will keep behaving as you would expect... It's that we no longer have +0.0 =:= -0.0, which is a very different thing (and frankly, makes sense - there should be some special operator that can differentiate between these two values).
Honestly - I rather dislike this change. I'd much rather have 0 & -0 be precisely equal to each other and have some obscure library function like `isNegativeZero` for the extremely rare cases they'd need to be differentiated. 0 and -0 being distinct values is honestly just an accident of convenience with how floats are stored - there are two bit patterns that result in a value of 0 but because they're different patterns we've assigned an unwarranted amount of distinction to them.
> 0 and -0 being distinct values is honestly just an accident of convenience with how floats are stored
That's not at all true. The inclusion of negative zero in ieee 754 was a deliberate design choice, and if it were considered desirable that zero should be unsigned, then the bit pattern assigned to -0 would not have been so assigned. See 'much ado about nothing's sign bit'[0], by one of the principal authors of ieee 754.
Whereas -0 and +0 are not operationally equivalent, operational equivalence is an important and useful relation; see the very pattern matching example linked from the OP[1].
the thread explains in the very first message that the existing semantics caused a subtle compiler bug in a way that makes it clear that you are not just wrong about the semantics of ieee-754 floating point but also about the pragmatics of how this affects real programs
> 0 and -0 being distinct values is honestly just an accident of convenience with how floats are stored
This is actually a valid distinction because these values can represent different values.
Floating point values are not numbers, but ranges of possible values: float -0 is a number in the range (-FLT_TRUE_MIN / 2, 0] and float +0 is a number in the range [0, FLT_TRUE_MIN).
IMO if I were to redesign IEEE 754 I would add a third float 0 value that always represents exactly 0 and redefine float -0 and float +0 as open limits, but that would probably make everything a lot more complicated.
"There are mathematical structures in which [dividing by zero] is defined. [...] However, such structures do not satisfy every ordinary rule of arithmetic (the field axioms)." https://wikipedia.org/wiki/Division_by_zero
You're familiar with the ordinary rules of arithmetic (ORA!) applied to the real numbers. However, there are many number systems that follow different rules.
The rules of arithmetic for floats, wraparound ints, elliptic curves, and asymptotic analysis are all different from ORA! (and each other) but they are useful in their own ways.
The reason division by zero can be a number is that floating-point numbers aren't individual numbers but ranges of numbers. So 0.0 isn't really the integer 0, it is the range of numbers between 0 and the smallest representable non-zero number. So division by 0.0 may be division by zero, or it may be division by a very small number, in which case it would approach infinity.
The same goes for the floating-point infinity: it doesn't represent the concept of infinity, it is a placeholder for every number between the largest representable number and infinity. That's how dividing a very large number by a very small number can result in infinity, a number which is really not a number.
This is the philosophy by which IEEE floating-point numbers were designed, and it's the explanation behind which negative zeroes and infinities make sense in floating point.
The way I find it easiest to reason about is by taking a graph of an asymptote and rounding it to the nearest units. You somehow need a way to say "there is an asymptote here!" even though you might not have all the units, and so you introduce infinities and negative zeroes to maintain as much precision as possible.
this isn't true. floats don't have semantics of internals. they have semantics of exact points that do math with rounding (otherwise the result of subtracting 2 floats would sometimes be more than 1 float)
the question of whether mathematical objects like 53 are more or less real than the historically contingent universe is indeed very interesting
however, it is irrelevant to the fact that a commenter was disagreeing because they had confused ieee-754 division with the division operation on so-called 'real' numbers
It is not about being very close. IEEE says "Comparisons shall ignore the sign of zero" and erlang (or rather CPU instructions erlang outputs for float == float) respects that
While =:= is not something mandated by IEEE. Erlang implements such an operator on floats and it decides to compare false for 0.0 & -0.0. So it probably compares bits/does integer comparisons on same data
The only other distinction between == and =:= is for integers and floats. Integers are not =:= to the corresponding float, but are ==, so it has nothing to do with closeness.
-0.0 is a weird dumb non mathematical value that IEEE 754 put in (for example the standardized value for sqrt(-0.0) is not even justifiable based on their choices). I think they didn't want 1/(1/-Inf) to be Inf, or something.
In the end think of erlang's "==" as "equal in value" and "=:=" as "equal in representation in memory".
Now if you really want to raise the hackles of someone using Erlang, should ask why integers and floats don't have a well ordering, and what you should do when you sort them.
i don't use Erlang (mainly Julia), but this change makes a lot of sense. in Julia, there is the == operator (value equality) and the === operator (value egality). 0.0 == -0.0,because they have the same "value". but they aren't identical (the bits are different), so !(0.0 === -0.0).
> there should be some special operator that can differentiate between these two values).
What for though?
I was expecting as I learned Rust that I would soon find the == operator (which is just the predicate PartialEq::eq) wasn't quite enough and need some === type operator (Rust doesn't have one), but I never did. Turns out actually in most cases equality is what I cared about.
The example for the compiler just seems like it wants a property that types don't necessarily even have (Equivalence), so my instinct would be that it's probably just broken and this is the tip of the iceberg or if the compiler does care about this property it should know floats don't have Equivalence and thus 0.0 == -0.0 isn't cause to perform this optimisation.
So far the only example I've been given of this operator is that it's what the compiler uses to decide equivalence, which is a situation where the correct answer was categorically no. The belief seems to be that this code is fine if we mistakenly unify cases where we think the IEEE floating point number had the same representation, and I don't buy that this is better than unifying -0.0 and 0.0 in the example, I think it'll be the exact same outcome.
i think the belief is rather that the compiler needs to reliably distinguish between -0 and +0 except where it explicitly invokes arithmetic comparison, and so the semantics of both =:= and unification should change to distinguish -0 from +0
i don't know what it would mean to not unify cases where the ieee floating point number had the same representation, assuming that by 'unify' you mean 'paattern-match) (erlang doesn't have full bidirectional unification where run-time data can contain variables which could be instantiated with constants from a pattern). what would the semantics of unification be then? no pattern could ever match a term containing a floating point number?
or perhaps you aren't aware that 'unify' means 'pattern-match' (prolog style) and we are discussing a pattern-matching bug, and by 'unify cases' you just mean 'remove one of two cases'. i think it's very likely that the compiler needs to remove redundant cases that fall out of its optimizer in order to get decent performance, and if the user puts them in their code it's probably a bug
> no pattern could ever match a term containing a floating point number?
No equivalence matching on floating point numbers seems like a much healthier approach, yes. I don't expect people to have huge problems with "x greater than 0.0" or "z less than 16.35" but "p is exactly 8.9" suggests muddled thinking.
The floats are very strange. While machine integers are already a bit weird compared to the natural numbers we learned in school, the floats are much stranger so I expect code which is trying to test equivalence on these values is going to keep running into trouble.
We know that humans, including human programmers, do not appreciate how weird floats really are, so I think telling programmers they just can't perform these matches will have a better outcome than confusing them by allowing this and then it rarely has totally unexpected consequences because a+epsilon was equivalent to b even though mathematically a+epsilon != b
you have to get rid of order comparisons too, though, or people will just replace a == b with a >= b && a <= b
also it seems like a compiler attempting to determine whether two pieces of code are equivalent (so it can throw one away) needs to be able to test whether two constants in them could ever produce different computational results; this is important for code-movement optimizations and for reunifying the profligate results of c++ template expansion
similarly, a dataflow framework like observablehq needs to be able to tell if an observable-variable update needs to propagate (because it could change downstream results) or not; for that purpose it needs to even be able to distinguish different nans. like the compiler, it needs the exact-bitwise-equality relation rather than ordinary arithmetic equality
where i think we agree is that floating-point arithmetic is probably a bad default for most people most of the time because it brings in all kinds of complexity most programmers don't even suspect
your comment reads to me as 'i don't understand floating point equality and therefore no human does so no human should have access to it'
but there are more things in heaven and earth than are dreamed of in your philosophy, tialaramex
My fear isn't about floating point equality but about equivalence. As we discussed repeatedly, nothing changed for equality, -0.0 = 0.0 just as -0 = 0 in integers. Equality works fine, though it may be surprising in some cases for floating point values because sometimes a != a and b + 1 == b.
"Exact-bitwise-equality" is yet a further different thing from equivalence. Is this what Erlang's =:= operator does? Erlang's documentation described it as "Exactly equal to" which is a very silly description (implying maybe == is approximately equal to), presumably there's formal documentation somewhere which explains what they actually meant but I didn't find it.
Presumably Exact-bitwise-equality is always defined in Erlang? In a language like Rust or C++ that's Undefined in lots of cases so this would be a terrible idea. I still think it's better not to prod this particular dragon, but you do you.
To be clear, it was added to the standard because there was user demand for it.
Zero is already special cased in IEEE 754 (along with the rest of the denormals, because of the implied leading mantissa bit), the implementation would be no harder if you just only had a single zero. However, back when the standard was being created, when a single zero value was proposed there was pushback from people in the user community who depended on knowing which side of the zero they underflowed on their own floating point implementations.
Many people think that negative zero is just an engineering artefact. This is not true, it is a feature that was asked for, debated, and then added...
... almost certainly because it was an engineering artefact in a previous system that someone managed to depend on, https://xkcd.com/1172/ style.
> Many people think that negative zero is just an engineering artefact. This is not true, it is a feature that was asked for, debated, and then added...
Allowing zero to be represented with two bit patterns was a feature that was advocated for to make bit operations easier - it was not decided that both zeroes should be distinguishable. This absolutely is an engineering artifact but you're quite correct that it was quite intentional.
These numbers are inequal in terms of storage - in real number terms zero has no sign and thus -0 and +0 are the same number. The source of their distinction is also important as it's just a pure point of convenience at the bit level - there are two separate bit patterns for 0 but both numbers are the same number.
You are overloading the word “real”. In terms of the mathematics of the reals, there is no difference. In terms of real-life pragmatics, 0 and -0 could be used to differentiate between different outcomes in a way that is sometimes useful.
Infinity and -0 are not in the real numbers, so the expression there makes no sense if you are thinking strictly in the real numbers. If you assign real number bounds to what the floating point numbers mean, the expressions make sense.
In floating point terms infinity tends to indicate overflow (any number that is too big to represent) and the 0's indicate underflows. So in more verbose terms,
1/(positive overflow) = (positive underflow)
While
-1/(positive overflow) = (negative underflow)
In this case, since the positive overflow isn't really infinity and the underflows aren't really 0, they are not equal. In practice, -0 and the 0 can both also arise from situations that produce 0 exactly, too, but this is not that case.
You may be thinking about how lim{x->inf}(1/x) = 0 = lim{x->inf}(-1/x), which is true. Infinity in floating point does not necessarily represent that limit, though, just any number that is too big.
You may also notice that the limit is not in the range of the functions inside the limit. For all real x, 1/x != 0
Floating point numbers don't represent all possible values, just a subset of values representable by a number of bits used. Therefore, in some cases it's reasonable to treat floating point numbers not as "exact point on a number scale", but rather a range between a number and the next possible representable number.
In the case of +0.0 and -0.0, they can be treated as values between zero and the smallest representable number (about 5.4E-079 for 32 bit floats). It isn't a very common use case since dealing with such small numbers isn't a very common thing, but it is definitely a possibility.
There are actually very clear and technical numerical analysis reasons for all of the weird stuff that happens in IEEE 754. The zero behavior, in particular, is because of this sort of thing.
if you want to say "zero approaching from negative one", fine, say that. but that IS NOT the same thing as -0. zero is just zero, it doesn't have a sign.
some people just want everything to fit in a nice neat box, and sometimes folks, life is not like that. this is an awful change, injecting meaning where there is no room for it.
-0 exists. It's just not a mathematical object. It's an IEEE754 signed-magnitude bit-pattern — one that is distinct from the IEEE754 signed-magnitude bit-pattern we call +0. Both of these bit-patterns encode the semantic meaning of the mathematical object 0; but that doesn't mean that the bit patterns are the same "thing."
Compare/contrast: the JSON document "0" vs the JSON document "0.0". Both encode the same type (JSON only has one Number type) and so the same mathematical object — which is, again, 0 — but these are distinct JSON documents.
And that's the whole point of this change. Erlang == is mathematical equality. Erlang =:= is (encoded bit-pattern) representational equality.
Which is often the kind of equality you care about, if you have canonicalization happening at parse time, and so you want to implicitly assert (usually for added efficiency!) that the values you got after parsing + mathing are not only mathematically equal to some value, but representationally equal to that value.
---
(If this irks you as a mathematician, you could model IEEE754 as its own mathematical group — a lattice, I think? — of which +0 and -0 would be members. Along with with some — but not all — base-2 rationals; and a random assortment of signalling and non-signalling NaNs.)
dealing with the same thing in rust, you can't use float in a lot of equality places where you can use ints, and they are moving to change match so you can't use floats in patterns there either.
A JSON document is itself canonically a string / stream of human-writable bytes. (I.e. the JSON specification specifies JSON documents canonically as a string-encoding grammar, not as a set of formal AST-node types like SGML/XML/etc.) When you mention a JSON document in text, you're mentioning an encoded string, so you put it in quotes.
Backticks to represent what you'd type at a REPL: if `"foo"` is a JSON document, then `foo` is the decoding of that JSON document (which will not itself be a JSON document, but rather a primitive or data structure in your programming language — one which happens to have a valid encoding as a JSON document.)
not sure what you hoped to accomplish with that. yes all documents are a byte array. this is not the same thing a the STRING type. this JSON document:
0
and this JSON document:
0.0
might compose a different stream of bytes, but by your own admission, they get parsed as the same type and value, 0. the only way to distinguish the difference in a parsed document, would be to present a string type as the input document:
["0", "0.0"]
but even then, you're only retaining difference in meaning after parsing, BECAUSE the values are strings, not numbers. so your comments have only proven that some byte arrays are different, and some resultant strings are different than others. so congrats?
JSON defines absolutely no semantics for numbers, only the syntax. Some implementations do handle 0 and 0.0 differently.
See also: the problems with stuffing a high magnitude integer like 2^60+7 into a JSON document, and then unpacking it in JS using the standard web browser API. The number can't be represented exactly with the standard JS number type, but it's a valid JSON document, and so the implementation has to make a choice.
> so your comments have only proven that some strings are different than others
Er... yes? What I'm trying to do here, is to provide an intuition for how IEEE754 bit patterns should be mentally modeled. They're byte-arrays that encode a semantic type, like a JSON document is a byte-array that encodes a semantic type.
And, just like `"0"` and `"0.0"` are different encodings of the same JSON-semantic Number, IEEE754 -0 and IEEE754 +0 are different encodings of the same mathematically-semantic number; namely, zero.
But, unlike with JSON — and I hope this should be obvious — there's no such thing as "decoding" IEEE754 to something else. The bit-patterns in IEEE754 just are... themselves. There's no more-canonical encoding to move them into that will erase the difference between IEEE754 -0 and +0, or between regular numbers and denormals, or between all the different NaNs. They're all distinct IEEE754 objects.
Which means that, unlike with JSON Numbers, where you just decode them to canonicalize them and can forget about the fact that there are multiple equally-canonical JSON encodings for any given decoded value... you can't just forget about there being multiple distinct IEEE754 objects with the same mathematical meaning.
You don't operate on JSON documents in their encoded form. But you do operate on IEEE754 bit-patterns in their encoded form. You don't turn them into "numbers" in some abstract sense, do math, and then turn them back into bit-patterns. They're bit patterns the whole time.
Every time the FPU does an IEEE754-math operation, it has to know what to do with those distinct equally-canonical encodings, applying logic to them such that they produce mathematically-semantically-correct — but potentially still bit-pattern distinct! — results. Your FPU isn't attempting to simulate mathematics over the field of the rationals (in the way that an arbitrary-precision decimal library would); it's doing specific IEEE754 operations that move an IEEE754-bit-pattern-typed register from containing a value representing one position on the IEEE754 lattice, to a value representing a different position on the IEEE754 lattice. Where some of those position-values of the lattice may be interpreted, by humans, to represent a mathematical number. (Non-uniquely, because -0 and denormals!)
There is no other domain to move the math into. You're stuck in encoded non-canonical bit-pattern land. And as such, it's helpful to be able to talk about the particular encoded non-canonical bit-patterns themselves, compare them for bit-pattern equality, and so forth. Because what those bit-patterns are can determine things like how long an operation takes, or how well a vector of said bit-patterns compresses, or whether external systems will be able to decode the results, or whether you'll still have any data at if you try to hard-cast your FP32 to an FP16.
(And don't get me started on the fact that computer "integers" aren't mathematical integers! At least IEEE754 is a standard, such that `float` means "position on the IEEE754 FP32 lattice" on every architecture targeted by programming languages with FP types, and when an arch doesn't have that support, it's shimmed in with a runtime IEEE754-correct FPU support library. The C machine-integer types — not just `int`, all of them — aren't even well-defined as being a position on a two's-complement ring vs a position on a signed-magnitude ring! Any program that ANDs two machine-integers together and expects a particular bit-pattern to result, is completely non-portable to many historical and esoteric architectures! Which comes up more than you'd think, because mathematical proof-assistants are effectively "esoteric architectures.")
> there's no such thing as "decoding" IEEE754 to something else.
Huh? You can totally take the raw bytes in an IEEE754 floating point number and parse them into something else (say, a decimal type) the same way you can take the raw bytes of a JSON document and parse it into a struct.
However, there are cases where comparing floating point numbers for equality works just fine. For example, if you have a variable 'v' and want to update it to a new value 'f', you can do 'if (v == f)' to check if the variable would change. Or if you have a sentinel value -1.0, it is perfectly ok to do 'if (a == -1.0)'
In general, if you look for a number 'f' in variable 'v', you can safely use equality as along as you expect 'v' to be set to 'f', i.e. the value of 'v' is not the result of a computation. This may sound trivial, but this is often omitted when discussing floating point comparison. If someone knows a scenario where the examples above would break, I would be curious to hear! (My main field is audio programming and code like this is omnipresent.)
cesaref's comment is not good advice, it's not advice at all, it's a snarky take on how floating point is trickier than it looks
almost invariably floating-point calculations end up in some kind of comparison; if they didn't we'd probably do the calculations in a galois field or something instead
even equality comparison of floats is useful; there are an abundance of seminumerical algorithms which use it correctly on the results of computations. a trivial example is computing an attractive fixpoint by iterating a function and comparing each hare result to the previous iteration and to a tortoise. you have to be especially careful about x87 extended precision here
ieee-754 requires bit-exact results for the five fundamental operations
here is an example where your 'safe' generalization is not safe:
#include <stdio.h>
int main()
{
float f = 0.7;
printf(f == 0.7 ? "ok\n" : "wtf\n");
return 0;
}
this prints 'wtf' on, for example, x86_64-linux-gnu
It's an easy mistake to make in C (and C++) because of their lackadaisical approach to type safety.
In Rust if we insist we want to compare an f32 (what C would call float) to an f64 (double), the language says it doesn't have a way to do that, highlighting our mistake. If we just try to compare an f32 with value 0.7 against 0.7, the type inference concludes 0.7 is also an f32 so it's equal, and if we do the same with an f64, this time by inference 0.7 is an f64 and so again it's equal.
Rust's behavior makes so much more sense! IMO, implicit conversions is one of C++ biggest weaknesses. Yes, it may save some keystrokes, but it can easily lead to subtle (or not so subtle) bugs.
I know there is a fair bit of division on this point but I've always liked type mixing, in languages where this is available, to handle signal values and preferred separate variables where it is not. I'd prefer to `if (v IS NULL)` a nullable float compared to checking a constant value - or else to use a guard value to test the validity i.e. `if x = FREQUENCY_SET // enter logic involving the frequency stored in v` - that approach is even better if you can use tuples/structs to glue the status and value fields together.
I am a quickly crustifying developer who was an enthusiatic value packer in their younger years but I've changed my tune to be less afraid of using more variables if they are clearer - even if that comes at a cost to data efficiency (and like, an extremely negligible cost in most situations). Though I don't have any objections to value packing for serialization - that's a fundamentally different domain.
Somewhere in the distant past a second grade me is staring at his math homework and fuming over the frustratingly ambiguous meaning of the minus sign. Is it an operator? A property? Both?
We could make the minus sign go away if we adopted a balanced base (e.g. balanced ternary), that has both negative and positive digits. That way, just looking at the digits would make evident if a number is positive or negative, without any need for a minus sign.
The problem arises with the way that this stuff is taught at a very elementary level, which then needs to be thrown away and relearned long after intuition and habits have formed around the old model. As an adult my reasoning is that it does indeed only do one thing: negation. it doesn't take anything away at all, if anything it actually adds information. Negating a four doesn't take it away, it just specifies a different (inverted) mapping of a given value with respect to zero. And this is still a very naive take, as someone with no background in number theory. But it's much more useful than say, Billy has four apples, he gives three to Mary, etc.
No it isn't. S-expressions are the simplest serialisation or trees. Trees are the simplest grammar that can encode mathematical expressions. C++ by comparison is as usable as doing cube roots using roman numerals.
> I've always held that the numbers themselves are perfectly precise. It's the operations that don't do what you expect.
I think that this is an oversimplification, and misses a key point.
Floating point numbers can be interpreted in (at least) two distinct ways.
In one use model, floating point numbers allow a sparse representation of selected points on the (extended) real number line. In this model, the numbers themselves are perfectly precise, and the basic operations do exactly what you'd expect -- but because the result of the expected real-number operation may not be one of the selected representable points, each operation is potentially paired with a second, implicit, rounding operation, and the consequences of the cumulative effects of this rounding operation take some thought.
This first model is often comfortable for those who have done low-level numerical program with sized integers, which are likewise a selected sparse representation of selected points on the integer line. While rounding comes up in fewer operations here, and is simpler, wrap-around becomes an issue.
However, a second model of floating point numbers is that each possible bit sequence represents an /interval/ on the (extended) real number line. In versions of this model, while the set of representable intervals is still very restricted, every point on the real number line falls into a representable interval, which is a property that sometimes eases (and sometimes seems to ease) analysis. In exchange, the operations are much more complex, and their limitations are much more obvious -- in particular, not only can no disjoint intervals be represented, but for any given point on the real line only one interval that contains it is representable -- there's no way to represent intervals of different sizes containing (some of) the same points.
This second model is what people are mostly thinking of when they say that floating point arithmetic is "not precise." In particular, both the inputs and outputs of operations have imprecise interpretations -- that is, the input interval contains infinitely many indistinguishable points, and the output interval contains infinitely many indistinguishable points, even if a real number equivalent of the problem would have a single number in the desired range.
When extended to true interval arithmetic, the second model can be very useful for understanding error propagation and uncertainty, but sadly this is rarely done.
the problem is the second interpretation is wrong. for example, exp(x) would give a higher value than it does since the average value of the exponent of a range is higher than the exponent of it's center.
Yep. For a sequence of operations to work on a range you do actually need to track both ends of the range, not just one of them. Otherwise what you end up implicitly doing is flipping lossily between a range interpretation and a point interpretation and back again.
As a followup to this comment, in Python you have the option of using `isclose()` which is present in both `numpy`[1] and the standard `math` [2] libraries. This has been quite helpful for me in comparing small probability values.
Since they're not changing ==, I wouldn't think =< and >= would change. Note, Erlang has a no arrow looking comparison rule, so <= is not a comparison operator; actually it's used in bitstring comprehensions, the bitstring version of list comprehensions which use <- [1]
We don't have fast hardware decimals, except on IBM mainframes. So, if you need fast financial calculations which are too complicated for integers (e.g. gasp division), you usually use decimally-normalized doubles and do it very very carefully.
This is a sad state of affairs, but nobody was fixing this in the last 30 years, so there's that.
Algorithmic trading, of course. High-frequency trading in particular. But any trading, in fact, when you run simulations for thousands of instruments and search among millions of parameters. Or when you implement an exchange, or route orders between exchanges according to their complicated criteria. Or when you do options pricing. Basically, everything in this field.
High frequency traders effectively charge money to provide liquidity.
But they provide far more than I need. I'm not worried about liquidity when I eventually sell those stocks, and if I could opt out of buying extra liquidity I definitely would.
If I have a medium size order, then even though they lower the spread they also front-run and limit how much I can buy at that price. So I'd rather have them not be there.
If I have a tiny order, then I don't care what the spread is within reasonable bounds, and I still don't want them to be there.
oh, well, if you're a speculator, often you really do lose money to hfts because they're better speculators than you, but it's unclear why anybody outside your immediate family should care about that
if you're an investor, otoh, the lower spread and greater liquidity means timely execution costs you less, not more. you aren't paying them for liquidity; they're paying you, or rather you're paying them, but much less than the spread you'd've paid an old-style open outcry market maker
(do you even remember markets before decimalization? minimum spread 12.5 cents)
of course you do need to execute intelligently; you can't just plop a million-dollar order in a hundred-million-a-day market and expect the market not to move against you
> of course you do need to execute intelligently; you can't just plop a million-dollar order in a hundred-million-a-day market and expect the market not to move against you
It's fine for the market to move against big orders, I just want the movement to, like, take one second. I don't want anyone to change their position in response to an order that hasn't even resolved.
> (do you even remember markets before decimalization? minimum spread 12.5 cents)
Well I'm not suggesting we undo that.
> if you're an investor, otoh, the lower spread and greater liquidity means timely execution costs you less, not more. you aren't paying them for liquidity; they're paying you, or rather you're paying them, but much less than the spread you'd've paid an old-style open outcry market maker
In this scenario I'm a long-term investor so the cost means nothing to me. So it's a matter of whether I want HFTs to profit, and I don't, because their actions are often not win-win. If HFT worked somewhat differently I wouldn't mind them the same way.
Do you just plan to save your money in dollar bills hidden inside a pillow until you retire? If you want to invest it in any way, you probably want to avoid floating point numbers for keeping track of it.
I would add generating large financial reports, enterprise billing etc. There is a huge business effect between for example 2 hours needed to calculate monthly invoices and 24 hours.
I'm assuming you mean equality comparison, rather than comparison in general!
but is there a better algebraic structure to use when thinking of floats? like, in terms of limits, or something that tracks significant digits and uncertainty? how do formal methods handle these?
There's more than one way to skin that cat, but interval arithmetic is a good and simple model. You can take the part of the real line which maps to the single float you are thinking about, and then look at the image of that set under the whatever functions you are thinking about.
> if you find yourself comparing floating point numbers, that's probably not what you want
Can you motivate this with an example for me? For instance, I think of games or location data encoded as FP; then, clearly, comparing them is a critical task to know questions like "what is closer" and so on.
They meant using the equal operator, because floating point is inexact and can produce different representations for the same number depending on how it was obtained. Greater and less than are fine. Equal is usually implemented by seeing if a number fits inside a tight range.
Every finite floating point value precisely represents an exact quantity. Arithmetic isn't lossless though. This isn't a special property of floats. Decimals behave the same way.
Second point: Combing FPs isn't lossless, across most/all arithmetic. Clearly true, the root of our discussion
Third point: decimal arithmetic is lossy too. Strongly disagree. The system of representation is what is lossy - floating point. Arithmetic (between two decimal numbers) is clearly not lossy in and of itself, only as implemented with computers.
Decimal is only lossless if you track an indefinitely-growing suffix to your number with a big bar over it to indicate "repeating". It's basically working with rational numbers but more painful.
Decimal numbers with any limit on digits, even a hundred, are lossy.
Decimal numbers as used by humans outside of computers have limits on digits, so they are lossy.
Maybe my point is uninteresting, but I feel you haven't really understood it.
Combining two FP numbers with arithmetic leads to possible errors that you can't represent with FP on every operation.
Combining two decimal numbers with non-division arithmetic never leads to that case. With division, sure, things are bad, but that's more of an exception than a rule to me.
This is because decimal numbers don't really come with any inherent limit on digits, and it's a bit strange to add a clause to the other's claims before making a counterargument.
Well in part I didn't understand your point because you didn't mention division as an exception.
But even then, multiplication causes an explosion in number length if you do it repeatedly.
When you specifically talk about numbers not being in computers, I think it's fair to talk about digit limits. Most real-world use of decimals is done with less precision than the 16 digits we default to in computers. Let alone growing to 50, 100, 200, etc as you keep multiplying numbers together to perform some kind of analysis. Nobody uses decimal like that. Real-world decimal is lossy for a large swath of multiplication.
I agree that if you're doing something like just adding numbers repeatedly in decimal, and those numbers have no repeating digits, then you have a nice purity of never losing precision. That's worth something. But on the other hand if you started with the same kind of numbers in floating point, let's say about 9 digits long, you could still add a million of them without losing precision.
And nobody has said anything about irrational numbers as you dismissed in your other comment.
So in summary: decimal division, usually lossy; decimal multiplication, usually lossy; decimal addition and subtraction, lossless but with the same kind of source numbers FP is usually lossless too
> Combining two decimal numbers with non-division arithmetic never leads to that case.
Most types are subject to overflow that has similar effects. Most of the FP error people encounter is actually division error. For example, the constant 0.1 contains an implicit division. The "tenths" place is defined by division by 10. I think that almost all perceptions of floating point lossiness come from this fact.
Likewise, bit-limited integers are lossy as they are incomplete representations of ℤ, and are not reciprocal with division. The point being that, just like floating-point representations of ℝ, division will always be lossy in the general case unless you choose a symbolic algebra instead of numeric.
that's slightly incoherent and reflective of the same confusion i identified; there is strictly speaking, no such thing as a decimal number, only a decimal numeral. normally the difference is subtle, but in this case it's the essence of the issue
'decimal' means 'base 10'
numbers, the abstract entities we do arithmetic on, are not decimal or in any other base; it is correct that arithmetic on them is not in any sense lossy
decimal is a system of representation that represents these abstract numbers as finite strings of digits; these are called 'numerals'. it can represent any integer, but only some fractions, those whose denominator has no prime factors other than 2 or 5. such fractions arise non-lossily as the result of the arithmetic operator of division when, for example, the dividend is 3. representing these in decimal requires rounding them, so decimal is lossy
binary floating point is lossy in the same way, with the additional limitation to only being able to represent fractions whose denominator is a power of 2, whose numerator does not have too many significant bits (53 most often), and which are not too positive, too negative, or too small
there are other systems of represention for numbers that are not lossy in this way. for example, improper fractions, mixed numbers, finite continued fractions, and decimal augmented with overbar to indicate repeated digits
IMHO That was poorly worded. What it should have said is "if you find yourself using equals to compare floating point numbers...". With Floating Point your comparisons should always be less than or greater than. Precision artifacts make the equals unreliable, and you should always be mindful of that when dealing with them.
Caveat: In the following, I use "comparison" to mean "check for equality".
Floating point numbers lose precision because binary arithmetic doesn't represent decimal in all cases ( 1/3 is an easy example). It's not hard to get into a situation where you're asking if a number is 0.0 but due to a precision error the number you have is 0.00000000000001 or whatever and it should have been zero.
If you're dealing with anything where precision is paramount (i.e. money or high precision machining) you should consider using something else that matches the real work precision you are trying to model.
You can, of course, use floats and test differences rather than strict equality (i.e. distance is < 0.01) and that's fine too... if you remember to do it, and account for the potential for small precision differences to propagate throughout your calculations and accumulate into larger errors.
> Floating point numbers lose precision because binary arithmetic doesn't represent decimal in all cases.
That’s not the reason. A counterexample is decimal arithmetic. That represents decimal in all cases, yet loses precision when doing calculations. Once you start doing division that applies even to arbitrarily sized decimal numbers.
The correct reason is because the subset of the reals that is representable as IEEE float isn’t closed under the operations people want to perform on them, including the basic ones of addition, multiplication and division.
Ideally, you'll want to do distance calculations in fixed point numbers (of desired resolution). Floating point works well as a first approximation, but unless you can make sure that you are not going to end up wandering in the weeds (and ensuring that required understanding of computational numerics), you should probably replace them with fixed point once you understand the problem and the solution and know the limits involved.
It's the hard task of trying to figure out the magnitude of the expected errors (which can accumulate), in the simplest case of a single operation you compare within epsilon distance :
Ok, sure, so it was just for a specific case of comparison.
Well, followup, if some error Epsilon can be introduced during any manipulations, when you check for strict inequalities do you check x < y + Epsilon? Does the language implicitly do it for you?
The language just gives you the direct comparison.
If you know numerics, then you can come up with the correct value for epsilon. But that is hard work. There are various things that the language could do for you, like use a dynamic amount of precision, or interval arithmetic where the error bounds are saved—but! Most of the time people just want the answer faster and with less memory used, which is what you get with bare floats. The people who make it their job to care about numerical accuracy can do it better than the language runtime would anyway.
Most of the problems are more easily solvable at a higher level anyway. Like, imagine Clippy saying to you, “It looks like you’re inverting a matrix. Are you sure that’s a good idea?”
This smells like the right way to think about things. Whenever thinking of real life measurements represented as real numbers, equality is always fuzzy — up to some resolution, and it would be better to make that explicit.
The `is` operator is checking object identity by comparing addresses. It is useless, although it may sometimes produce reasonable-looking results due to optimisations.
Python 3.11 gives me this:
>>> -0 is 0
<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
True
Because (a) -0 is an integer, and there is only one integer zero, (b) small numbers have only one instance in memory as an optimization.
It is 2023 and our tooling still encourages the same mistakes people were making 40 years ago. Can we really not have equality operators that do a comparison with a 1% tolerance or something as a sensible default equivalence instead of blatantly wrong bitwise comparison?
I would even be happy with a default set of compiler warnings or errors, which I don’t believe I have ever seen.
what people want varies, but often what people doing numerical programming want is to write their monomorphic code in infix syntax so they can more easily see bugs in it
they're already familiar with rounding errors
from that perspective you're suggesting taking a step backwards from fortran i toward assembly language
the erlang compiler bug presented is not an example of what you're talking about because you're talking about arithmetic, and the buggy comparison operator it's using is not an arithmetic comparison operator
Fortran would be an interesting case, because say C++ is polymorphic, and so are all dynamic programming languages with more than 1 type (x == y could be a string or float comparison)
But I would still say that floating point equality is a vanishingly rare operation
I can't think of any real use case for it, other than maybe some (bad, incomplete) unit tests that assert 1.0 == x and assert 1.0 == y
And that use case is perfectly served by float_equals(), and arguably served better if it has some options. Although probably abs(x - y) < eps is just as good, and you don't even need float_equals()
---
Can you show some real use cases for floating point equality in good, production code? (honest question)
It's similar to hashing floats, which Go and Python do allow. I made an honest request for examples of float hashing:
I didn't get any answers that appeared realistic. Some people said you might want to create a histogram of floats -- but that's obviously better served by bucketing floats, so you're hashing integers.
Another person said the same thing for quantizing to mesh.
One person suggested fraud detection for made-up values, but that was clearly addressed by converting the float to a bit pattern, and hashing the bit pattern.
For that case it seems pretty clearly OK to omit === on floats, since I use awk and R for simple stats and percentages, and that's likely what people would do with a shell.
Though as always I'm open to concrete counterexamples.
i gave some examples in https://news.ycombinator.com/item?id=35883963, though arguably those are not very concrete. also i pointed out there why unit tests that use exact comparison on floats are not necessarily bad: ieee-784 specifies bit-exact results for the five fundamental arithmetic operations, and if your compiler is introducing rounding errors, you want your tests to fail. (or introducing nonerrors, in the case where it's introducing fmas.)
even hashing is useful for memoization
i think probably you're asking in the wrong places
the lapack source might be a better place to look; we can probably suppose that if something is in lapack it's realistic and also not a bug, and numerical code doesn't get more 'good, production' than lapack
in dgelsx for example i find
ELSE IF( ANRM.EQ.ZERO ) THEN
*
* Matrix all zero. Return zero solution.
*
CALL DLASET( 'F', MAX( M, N ), NRHS, ZERO, ZERO, B, LDB )
and also (fairly dubious, at least to my uneducated eye)
* Determine RANK using incremental condition estimation
*
WORK( ISMIN ) = ONE
WORK( ISMAX ) = ONE
SMAX = ABS( A( 1, 1 ) )
SMIN = SMAX
IF( ABS( A( 1, 1 ) ).EQ.ZERO ) THEN
RANK = 0
CALL DLASET( 'F', MAX( M, N ), NRHS, ZERO, ZERO, B, LDB )
GO TO 100
ELSE
RANK = 1
END IF
of course float comparisons for ordering are much more common than float comparisons for equality, but there are numerous realistic examples of float comparisons for equality in lapack and, i think, in applied mathematics in general
but if you ask an audience of type theorists, sysadmins, and web developers, they won't know that; they aren't numerical analysts and probably haven't inverted a submatrix since sophomore math class, if ever
(there are 6550 files in lapack 3.9.0 of which i looked through 15 to find these examples, suggesting that there are on the order of 1000 realistic examples in there; eventually i'd probably find one that wasn't even comparing to zero)
plausibly instead of giving them floating point by default you should give them fixed point with, say, nine digits after the decimal point; that was the approach knuth took in tex, for example, to guarantee reproducibility, but it also gives human-predictable rounding. and in most cases it should be perfectly adequate for simple stats and percentages. as a bonus you get 64 bits of precision instead of 53
Thanks for actually going and looking! I would say that this supports the idea of float_equals(x, 0.0) or even float_equals_zero(x) :)
It wouldn't be too surprising to me if 90% of float equalities in real code are comparisons to 0.0 or 1.0 or -1.0 (that's what I meant in my original post, there was a typo)
The point about x >= 0 and x <= 0 is interesting. I suppose to be consistent, you would also remove those operators on floats, although for something awk-like, I don't think it matters either way.
---
The other point is that it looks like the Fortran code isn't actually using == or x.eq.zero polymorphically!
So the first point is that I would want to understand conceptually where float equality would be equal. You mentioned fixed points, although there I also suspect that abs(x - y) < eps is a better test in most situations, though certainly there could be counterexamples.
The second point is -- would any of those situations be polymorphic with int and float, or string and float? It depends on the language, but it seems vanishingly unlikely in most languages, and in particular in say in C++
I would probably go look at the equivalent Eigen code if it mattered with respect to Oil, but it probably doesn't
yeah, i also wrote a cube root routine using newton's method yesterday and came to the conclusion that |x - x'| < threshold was a better approach in that case. maybe in every case, maybe not
i agree that prefix notation is not as bad for these as i had thought before looking
i think there are plausible cases where your float might be polymorphic with double, complex, a vector, a matrix, or an element of a galois field, but i think it would only be polymorphic with int in cases like stopping change propagation when an observable value gets recomputed to the same value from new inputs, and in those cases what you need is exact bitwise equality, not arithmetic equality
an awk-like thing might be better with fixed-point, like tex and metafont; you could call the data type 'btc` and represent it as a 64-bit integer that's multiplied by 10*-8 for output
The problem with defining an “epsilon” (1% in your case) is that there is no value that would please everyone. For some, 1% will be fine, but others may need 0.0001%.
The solution is to either use decimal floats if they suit your need (and eat the performance penalty), or to use a linter that flags float comparisons by equality.
That’s bad enough, but it is not the problem with defining an “epsilon”, it’s a problem.
Another one is that you would lose transitivity on equality. With a 1% epsilon, you would have
1.000 == 1.005
1.005 == 1.010
1.010 == 1.015
but
1.000 != 1.015
You also would have to be careful on how to define that 1% error range. the naive "x is equal to all numbers between 0.99x and 1.01x" would mean 1.0 would be considered equal to 0.99, but 0.99 would not be considered equal to 1.0 (1.01 times 0.99 is less than 1.0)
You also lose that, if a == b and c == d it follows that a + c == b + d.
The behavior around zero also will go against intuition. If you consider 0.99 and 1.0 to be equal, do you really want 1.0E-100 and -1.0E-100 be different?
And then there's the question of how you use the epsilon - i.e. whether 1.2e-16 and 1e-20 are "within 1%". Sometimes those are very much different sizes, but if you're comparing sin(π) (i.e. the 1.2e-16) with some other almost-zero computation, they're very much within 1%.
> Can we really not have equality operators that do a comparison with a 1% tolerance or something as a sensible default equivalence instead of blatantly wrong bitwise comparison?
Then it wouldn't really be an equality comparison any more. For instance according to your rule 1.0 and 1.01 are the same number, but they clearly are not. But if you want to do this regularly you could create a special type and overload the equality operator (if your language supports it, otherwise you'll have to call the function directly) that points to 'boolean about_equal(float f1, float f2, float epsilon)' or something equivalent where epsilon is the amount that you would allow f1 and f2 to deviate from each other to still call them equal. And you could define epsilon as ((abs(f1) + abs(f2)) / 100) .
The bitwise comparison isn't 'blatantly wrong', it's the expected behavior in just about every programming language. I'm not aware of any language that has a native float operator that does this but as you can see you can usually simply add one yourself if you really need it. But this need rarely comes up and usually indicates that you are doing something wrong.
Negative zero is very, very annoying because it means that many floating point identities flies out of the window. x + 0.0 != x, x * 0.0 != 0.0, etc. It makes it much harder for the compiler to optimize arithmetic. Thus, at least some SIMD circuits do not follow ieee754 to the letter because it would result in performance degradations.
Those identities do hold though, as long as zero and negative zero compare equal, which according to ieee754 they should. Performance degredations happen in other places such as denormals.
This feels like a bad idea. Treating +0 and -0 as unequal because they have different bit patterns is no more correct than treating two NaNs as equal because they have the same bit pattern. Zero is zero.
In case of floating points, since they are an approximation, zero is not zero. Zero is a quantity approaching to zero. In fact dividing by zero with floats is entirely possible! (it leads to infinite, positive or negative depending on the sign).
For example, the quantity (try it in Python):
0.1 ** 100 / 1000**100 evaluates to 0.0
while the quantity
0.1 ** 100 / (-1000**100) evaluates to -0.0
It's clear that neither the two quantities are 0 (no division can produce a result that is exactly zero!) but they approach the zero. But one quantity is slightly less than zero, the other slightly more than zero.
Of course for integers -0 doesn't make sense (that is the reason why we invented 2 complement, to not have 2 "zeros")
Please refer back to "It's clear that neither the two quantities are 0 (no division can produce a result that is exactly zero!) but they approach the zero." because you just gave an example of how that rule is correct, not an exception.
no, its not. zero is zero, its nothing. "a quantity approaching to zero" is "a quantity approaching to zero". its easy to remember because zero is not a quantity, its a lack of quantity, by definition.
these are two distinct concepts, which some people mistake as the same item.
That sounds wrong to me. In IEEE floating point, 0.0 is not an approximation of zero, it is zero exactly. The binary value is literally all zero bits. Negative zero appears simply from the IEEE floating point format reserving a bit for the sign.
IEEE 754’s idea was that if you got a zero in a floating-point calculation (which, at the time, meant one with physical quantities), it’s probably underflow, not a legitimate result, so the FPU might as well try to give you at least a sign even if it can’t give you any significant digits. Though 754 says comparison must treat positive and negative zero as equal nevertheless, you have to deliberately choose to look at the sign (with e.g. C99’s copysign) to see the difference.
=:= is supposed to be "exactly equals". An example given in the docs[1]:
1> 1==1.0.
true
2> 1=:=1.0.
false
You can think of it as analogous to == vs === in JS, except that the "type safety" here really only refers to different numeric types (and in JS there is no such distinction)
This makes sense. They are different entities and you have to have a way to acknowledge that they are. Breaking strict equality seems a fitting way to do so.
In Geometry for Programmers, I reference a case when an automated test system caught a "zero-not-zero" bug, and I couldn't reproduce it with a unit test until I finally realized that the test system was comparing XMLs not numbers. In print, "-0." and "0." are, of course, different, although in runtime they pretend to be the same.
As someone who programmed assembly on a one's complement machine* 40 years ago, this discussion is interesting, from both the "haven't we learned anything" and the "makes sense to me" camps.
More interesting to me is how they are introducing this change, both in the previous OTP and next, and how they will arm people with tools to hopefully identify and understand the impact. I wonder how many folks will actually be impacted?
What about NaN? Does Erlang have NaN? How does Erlang compare two variables containing a NaN (or expressions evaluating to a NaN)? There are (2^24)-2 different NaNs in a 32bit IEEE 754 float, and there are (2^53)-2 different NaNs in a 64bit IEEE 754 double. The IEEE 754 standard says, that a NaN value should compare to any other number (including any other NaN) as not equal. Yes, the expression `x == x` could return `false`.
They are a way of approximating Real (the set not as an adjective) numbers, though. Real numbers are exact when they are equal to an integer or a rational, because, depending on how you construct them, either that's what the cut contains or that's where the series converges.
And also because that's what the most common spec demands.
floating point math is math like any other, just with a vastly different number system. There are valid arguments on both side if we should hide or expose those differences in the implementation, but there's always going to be pretty big implications (i.e. the compiler behaviour mentioned here)
Yes but this is floating point - +0.0 represents all numbers between 0 and the smallest +ve FP number, -0.0 represents all numbers between 0 and the smallest -ve FP number - both can represent actual 0 - remember these can be generated by underflow so those meanings can be important in context
One wonders how different the world would be if IEEE754 decided to use two's complement instead of sign-magnitude. Or alternatively, if two's complement was rare and almost all mainstream hardware (due to some historical reason with inertia) was sign-magnitude.
Hot take as a non-user of Erlang. There may be arguments for and against 0.0 =:= -0.0. You could probably do either. But I would be terrified of changing from one to another.
Yup - both of the bit patterns for 754 here were assigned the value of 0 and the standard instructed comparison operators to treat the two values as equal. The bit patterns for both -0.0 and +0.0 are assigned to the same real world value - there is no reason to distinguish them outside of circuits that actually execute floating point arithmetic.
it's important to distinguish them in programs written for those circuits and in compilers that transform those programs because they have significantly and intentionally different behavior
