Converting decimal to byte (little endian): what's wrong with 205?

Python Help
1

Hello everybody!

That’s my first message here, so I start by asking apologies for any mistakes.

To the problem…

I have a routine that converts decimal numbers to bytes (little endian) that is working Ok (or so I guessed…). It is like this:

_value = int(_amount * 100)
self.ssp_command.CommandData[1] = _value       & 0xff
self.ssp_command.CommandData[2] = (_value>>8)  & 0xff
self.ssp_command.CommandData[3] = (_value>>16) & 0xff
self.ssp_command.CommandData[4] = (_value>>24) & 0xff

It gets a value and multiply by 100 before converting. So:

For a value like “4.05 → 405”, it converts correctly to 95 01 00 00

For a value like “3.05 → 305”, it converts correctly to 31 01 00 00

but…

For a value like “2.05 → 205”, it converts wrongly to CC 00 00 00, which is 204; the correct is CD 00 00 00

Question: how is that possible?

I’m using Python 3.9.2 on a Raspberry Pi 4, with Debian 11 (kernel 6.1.21-v8+)

2

What does that have to do with bytes? Doesn’t int(2.05 * 100) give you 204 already?

1 Like
3

Don’t test two seperate conversions at the same time. What is going wrong here is the step _amount → _value, not _value → bytes. _value isn’t 205, it’s 204 because of how floating point numbers work.

1 Like
4

round would be better to use here than int, as floating-point errors shouldn’t be large enough to affect rounding to the nearest integer. (int itself does not round a floating-point number, but simply takes the integer portion, i.e. drops the fractional part.)

>>> round(2.05*100)  # rounds 204.99999999999997
205

Depending on your inputs, there is one thing to be aware of with round: it uses “banker’s rounding”, where a result is rounded to the nearest even integer if the argument is half-way in between:

>>> round(2.5)
2
>>> round(3.5)
4
3 Likes
5

Got it.

I fixed by using math.ceil() instead of int(). This gives the correct value and the rest is working fine.

Thanks for replying.
Regards.

6

So math.ceil(2.45 * 100) doesn’t give you 246?

2 Likes
7

Oh my…

You’re correct. My quick and dirty “solution” is flawed.

8

The right way is to create a system that preserves the necessary integer value the entire time. For example, by using the desired integer equivalent for _amount, and altering other calculations that use _amount to divide by 100.

2 Likes
9

At the end of the day, I just need to convert 2.05 to 205 and 2.45 to 245 and 1.3 to 130 and so on.

So I decided to “convert” the float value to string before the final transformation. Now it’s working fine.

I’m aware that’s not the pythonic or elegant way to solve the issue, but at least it’s a way to circumvent all the complexities of numerical analyses quickly.

I really appreciate your help.

10

What was wrong with @chepner’s suggestion of using round?

11

That’s what you said about the previous attempt, too.

2 Likes
12

It’s convoluted and probably not efficient, but at least it gives the correct answer:

>>> var = 2.05
>>> int(var * 100)
204
>>> from decimal import Decimal as D
>>> D(str(var))
Decimal('2.05')
>>> D(str(var)) * 100
Decimal('205.00')
>>> int(D(str(var)) * 100)
205
1 Like
13

The decimal module provides better ways to convert float to Decimal values than relying on str to produce floating-point literals.

# Demonstrate the problem with 2.05 in the first place
>>> decimal.Decimal.from_float(2.05)
Decimal('2.04999999999999982236431605997495353221893310546875')

# A solution that explicitly spells out the desired outcome.
>>> decimal.Context(prec=3).create_decimal_from_float(2.05)
Decimal('2.05')
2 Likes
14

@chepner How is it better? Please show an example where it is. If I change the number to 12.05, you produce 12.1, that’s worse.

15

I did; it doesn’t rely on an intermediate conversion to a str value. (Or for all I know, it does, and this details are hidden in the implementation of create_decimal_from_float. Point is, I’m not re-inventing the wheel.)

16

No you didn’t. You just got the same correct result. That’s not better, that’s just equally good. And for 12.05, yours is worse.

17

The point is, I am using a method specifically designed to convert a float value to a Decimal value.

The problem with 12.05 is that correct prec argument depends on the magnitude of the value to convert. Perhaps the quantize method is better:

>>> [Decimal(v).quantize(Decimal('0.01')) for v in [1.05, 2.05, 3.05, 4.05, 12.05]]
[Decimal('1.05'), Decimal('2.05'), Decimal('3.05'), Decimal('4.05'), Decimal('12.05')]

But my main goal here is not to solve the problem, but to point out that arbitrary string conversions are not the solution.

18

It’s a redundant alternative constructor, nothing special.

With your prec/quantize attempts it rather looks like you’re the one trying to reinvent the wheel of float’s string conversion.

How is the way via string not the solution? You still haven’t shown a case where it’s wrong.

19

I’m not trying to do string conversions at all. OP is trying to round (a multiple of) a float to an integer. Somebody else though converting a float to a string was a good way to do that.

I don’t have to show that it is wrong, because I’m not contesting a valid claim that it’s right in the first place. I’m pointing out that, if one chooses to use Decimal, there are methods already provided for handling float conversions that don’t involve converting it to a string in the first place.

20

The string conversion is just another way to get to decimal and round the value. Unclear why you have an issue with that and why you think your way is better.