New format specifiers for string formatting of floats with SI and IEC prefixes

[avylove (Avram Lubkin)]

@jagerber

4a. I think negative precisions should be supported. In this case the float will be rounded to the corresponding precision and then displayed with precision down to (and including) the ones place. e.g. 123.45 with precision = -1 would be displayed as 120.

Negative precision is an interesting concept. I think this could be very useful!

@jagerber

• 10. Shifted* engineering notation in which the float is displayed in base 10 with an exponent chosen so that it is an integer multiple of 3 and the mantissa satisfies 0.1 <= m < 100. However, if the float is 0 then the mantissa will also be equal to 0.

I think this would be better done through a “margin” field like it’s done in Prefixed. Then it is more flexible and can be applied more generally. For example, moving the boundary from 1000 to 100 would be a margin of -90, since 100 is 90% of 1000. In Prefixed this looks like f'{Float(123.456):%-90h}'.

@jagerber

17b. Current python float formatting supports left, right and center justification of a float within a field of an arbitrary fill character. I do not think this feature should be supported for strict float formatting. I consider left and right padding by arbitrary characters until the string is a certain size to be a string formatting feature, not a float formatting feature. If a user needs to do this can first format the float and then re-format that string so that it has the width they desire. However, this strategy doesn’t work to insert leading zeros between the sign and the top digit, so that specific feature makes sense for float formatting.

I disagree that the user should have to reformat the string, but I do agree that fill on the number and fill on the string are different operations and the current spec is lacking in this area. It would be great if there was a precision-like field for the left of the decimal.

@jagerber

19a. I do not think there should be an option for no space between the mantissa and the prefix. For the SI notation that would violate NIST standards for formatting numbers with units. I’m not sure about the IEC standards.

I’m not apposed to this. I almost always add the space for readability.

@jagerber

19b. I do not like the idea of supporting 123 Mi = 123 M. If that is supported then it’s ambiguous whether 123 M = 123e+06 or 123b+20. Users who want to do that will have to strip off the i themselves. See proposal 20 below.

I’m not apposed to this either, but as @pf_moore indicated, many people use a single capital letter. I agree this is ambiguous and I never know if they mean base-10 or base-2. It would probably be good if the language enforced the standards.

@jagerber

Strings expressing floats using prefix notation should be convertible back to floats. That is float(12.3 Ki) should be yield the float 12.3b+20

Yes, I did leave this off the original post, but Prefixed does do this and I think it’s important.

@jagerber

I’ve also used the .. separator to flag sig fig rather than precision mode.

I’m torn on this one. In one way it’s simple, but in another, it’s easy to accidentally include two periods when one was intended and not realize it.

@jagerber

I would say that existing formatting types such as f and g should be parsed exactly as they always have been.

Agreed. None of the existing behavior should break. Both for backwards compatibility and to ease people moving from other languages that also derive their format specification from C.

@jagerber

One idea is that rather then forcing this new formatting specification into the existing language, we could open up a new namespace, perhaps prepending with a colon (again inspired by prefixed) or something to indicate we want “scientific” float formatting. Then we would be free to design the format specification however we wish, unconstrained by the existing format. But actually, given the regex above, I don’t actually think such a measure is necessary.

This is an interesting idea! I think it would only make sense if we were to come up with a very functional, yet incompatible specification. I’m not even sure what that would look like, but it would be interesting to think about. Perhaps a new spec could address things like customization.

@steven.rumbalski

This seems like a lot of complexity added to support a minority of users. These seem like good ideas for an external package on PyPI, but not for inclusion in Python. How do other languages support these use cases?

I don’t think we’re talking a lot of complexity nor a minority of users. Just looking at the number of code examples, questions, and existing PyPI packages that exist indicate the current spec if lacking. Even if we implement a few changes it would reduce downstream complexity and duplication of code.

@jagerber

Next steps

What are next steps here? It seems there is decent appetite for “engineering formatted” floats. Is there enough appetite to actually try further steps beyond forum posts for this? I’ve never written a pep before but I think some of the material in this post and throughout the thread could be a start for the skeleton for a pep.

I think there is still some things to work out. The question is if it make sense to do that in this thread or in a working group. One thing is we should probably break things down into problems and proposed solutions. For example, “The existing format specification does not provide sufficient control of significant digits.” Then the proposals of a new flag or a change to the precision field such as supporting '..' with examples and pros and cons. We’ll need this for a PEP anyway, but I don’t think there is enough consensus yet for a PEP. However I haven’t really participated in this process before, so I’m not really sure.

[ChrisBarker-NOAA (Chris Barker)]

@Rosuav

it’s just trying to format numbers.

Absolutely

@Rosuav

As such, “significant digits” isn’t actually all that useful either - the most useful metric is “digits after the decimal place”,

That COMPLETELY depends on the use case. Sometimes you know the order of magnitude of your numbers, and sometimes you don’t.

And for any values that are very far off from O(1) number of digits after the decimal makes no sense either. Though the e formater works OK for that.

Anyway, on the the OP’s idea – my thoughts as a scientist/engineer:

+1 on the IEC formats – I, as I’m sure many others, have written hacked together versions of this many, many times. (and put it in a utility function that I then forget, and rewrite…). This would be really nice to “just have”

-1 on the SI prefixes – I am almost always using particular units for a reason, and want to stick with that, I can’t imagine when I’d want, e.g. grams to be converted to kg for me, and just for display. And if I did, the would I do:
f"{mass:.2h}g" ??

That’s just me, but I’d never use it.

+1 on engineering notation, e.g. like g but always a multiple of three exponent.

I’d use that.

[jagerber (Justin Gerber)]

Upon more thought, I’ve identified exactly what sig fig feature is missing in the built-in format specification. Including the details here in case anyone now or in the future is interested.

The built in formatting can always display numbers to a specified number of sig figs just using exponential mode directly. E.g. f'{123.456:.3e}' gives 1.235e+02. '.pe' will format to p+1 sig figs.

The built in formatting will also always display a number to a specified number of sig figs using the #.pg mode. The downsides are (1) the inclusion of the # means that mantissas with no fractional part will have hanging decimal points and (2) The user has no way to coerce fixed point mode.

More on point (2): specifically, according to the -4 <= exp < p rule (a) it is impossible to display floats less than 0.001 in fixed point mode while specifying sig figs and (b) It is impossible to get fixed point mode while specifying fewer sig figs than there are digits to the left of the decimal point.

Point number (b) says that it is impossible to pass any formatting string that will convert the float 123456 into the string '120000'. I imagine this was the intention behind requiring exp < p for fixed point mode. The original authors didn’t want the possibility that float format would look like it’s rounding (as opposed to just truncating) the floats passed in. Of course the float formatting DOES do rounding all the time, f'{1.77:.1f}' gives '1.8', but I can see the argument for not rounding “above the decimal point”.

So in other words, built in formatting is comfortable with sig figs as long as the least significant digit is at or below the decimal point.

Including sig fig formatting would certainly make it possible to pass a formatting specification that converts 123456 to '120000'.

[jagerber (Justin Gerber)]

Ok, I’ve been wanting scientific formatting for a long time and have been kicking around code to do what I want (and researching existing packages to see what is currently offered) for quite some time. I’ve finally taken some time to bundle the code a little bit more nicely and make it more flexible. Here’s the result:

GitHub - jagerber48/sciform: A package for formatting floats into scientific...

This code was heavily influenced by the prefixed package we’ve been discussing, as well as the uncertainties package which has functionality to format pairs of value/uncertainty floats into value/uncertainty strings like 84.23 +/- 0.08. I’ll also highlight the sigfig package (which uses the round function for sig fig rounding as well as string formatting) which has a stated goal to be included in stdlib.

See the readme for details about what features are offered. The notable features are:

• Explicit control over sig fig vs precision mode
• Engineering and shifted engineering notation mode
• Binary and Binary IEC modes
• Explicit exponent selection
• Prefix mode
• Flexible thousands, thousandths, and decimal separator configuration
• Global configuration overrides (including temporary override with a context manager)
• Formatting for value + uncertainty pairs (forthcoming)

To bring it back to the discussion at hand:
There is a section near the top of the readme where I describe the shortcomings of the built-in float formatting for scientific float formatting and there is a section near the bottom of the readme where I describe the incompatibilities between the design choices I took for sciform and the built-in formatting. This is basically a list of the features which I think, while they may be useful for generic string and maybe numeric formatting, they are not critical for scientific float formatting.

The minimal interesting changes to built in string formatting would be adding one or both of (1) floating point sig fig formatting and/or (2) engineering notation. I’m getting the sense from this thread that none of the features discussed in it will make it into stdlib/built-in at any point. Nonetheless, I think there’s still a next step to convince people which would be if someone put together minimal and backward compatible extensions to the built in formatting language which support one or both of the above features, especially handling/making decisions about some of the tricky edge cases. Perhaps with hard implementations in hand people could be convinced to add at least one of these features to built in.

@avylove I think what I described in the last paragraph would be pretty much exactly what prefixed did if prefixed continued to use numerical exponents (like e+02) and if prefixed didn’t handle binary scientific/IEC formatting. I wonder if it would be easy to fork prefixed and make something which is a more minimal extension of the built in formatting.

[mdickinson (Mark Dickinson)]

@jagerber

(2) The user has no way to coerce fixed point mode

I think this is the crux of it. I think of formatting as the composition of two separate operations: a rounding operation (whose target is some kind of intermediate decimal type), followed by a presentation operation. Those two operations are largely independent, but can’t be controlled independently: f format says “round to p places after the point, then present in non-scientific notation”, while e format says “round to p+1 significant figures, then present in scientific notation”. There’s no built-in way to mix and match (“round to p significant figures, then present in non-scientific notation”), even though it’s perfectly meaningful to do so. (As already mentioned, g mode is more of a do-what-I-mean-without-me-having-to-think-too-much mode, and has hard-coded choices that make it not particularly useful in situations where you want fine control.)

The more general problem here, which seems quite fundamental, is that (a) there are many many possible variations that people might want for formatting, but (b) there’s a limit to what can reasonably be encoded in the formatting mini-language without it becoming an unmaintainable and complex (not to mention unreadable) mess - IMO, the problem space is just too big to be able to be expressed clearly in the mini-language. The best that the mini-language can hope to do is to cover common use-cases.

Luckily, it’s easy to escape the restrictions of the mini-language by doing your own formatting directly, and I think f-strings make that option non-horrible to read:

from fancy_formatting import FancyFormatter

f5sig = FancyFormatter(
    significant_figures=5,
    presentation_mode="fixed",
    trim_trailing_zeros=False,
    sign_on_negative_zero=False,
    ...  # more options here
)

fruitiness: float = do_calculations_here()
print(f"Optimal fruitiness is {f5sig(fruitiness)}")