Please - int <-> Decimal conversions have exactly the same O() behavior as int <-> str, so it’s perfectly legitimate to wonder whether they’ll become limited too. I’m on the PSRT list too, so was aware of the str -> int issue from the moment it was first brought up there. Technical knowledge of algorithms in this area was essentially non-existent. I brought up that CPython’s int -> str was also quadratic-time, which seemed to come as a surprise even after knowing that str -> int was quadratic-time. For all I know, it’s coming as another surprise now that CPython’s int <-> Decimal are also quadratic-time.
Since it’s already been suggested that int(Decimal(str)) is a workaround for those who want to convert large strings to int without fiddling the new global knob, people who care have legitmate reason to wonder that too will become hobbled. It’s highly relevant to those looking to mitigate the mitigation .
Also, consider that if you make breaking changes like this, which is encountered in the wild, arent you essentialy Dos’ing your own programming language by releasing it at such short notice? I guess in most cases, when this is encountered in the wild on a production server, users will report spruious 500 errors, with seemingly no cause? Nothing changed in our project code? All we did was keep with best security practices and make sure python is the latest version.
How many times has this been exploited in the wild? Was it even being exploited? Does this bring light to the fact that systems running older versions of python are affected, in the context that previously it was not exploited? And im not saying that you should hide any vulnerability in the language from the community, but, could you have taken more time to consider more universal fixes which are backwards compatible? I.e. we can upgrade python and it be more secure without having to search every project that we manage for every instance of int().
remove final portion as it was in regards to an unrelated comment, due to the linear nature of these threads
It is. Off-list, David clarified that his post was not about the decimal discussion his post immediately followed - that’s just a consequence of that this UI displays messages in a linear most-recent-last order.
For what you talked about in your message before last, int(Decimal) takes time quadratic in the number of decimal digits “before the decimal point”, period. It doesn’t matter to that how the Decimal was constructed; e.g., doesn’t matter whether you’re used quantize().
But if there are no digits “after the decimal point”, there’s no need anyway to convert it to int just to convert it to a decimal string for display. str(Decimal) is linear-time regardless of whether the input has a fraction part, and if it doesn’t the output doesn’t contain a radix point either. In the latter case, converting to int first is pointless.
I see no reason why something that is explicit about the intent like int(Decimal(str)) would ever grow a restriction. I do recommend adding a code comment anytime you use that idiom though as it otherwise isn’t obvious to a reader why you’d bother tossing a Decimal in there.
The title of this thread tells a different story about what the specific topic is, notwithstanding that this change is being pronounced indisputable (and decree by fiat is not really going to allay the concerns specifically about the governance dimension of this).
Where then should the concerns about balancing security vs. all other needs (stability, transparency, …) be discussed? A new thread? On the SC issue tracker? Or will this continuously be flagged as “off topic”?
Please understand that all participants here care about python and its ecosystem a lot (otherwise why spend time on the discourse). The reactions are proportional to how unusual, rare or even unprecedented this change was. A healthy resolution would be possible through open communication; it’s going to be more difficult (and have longer-lasting consequences) if the principal reaction remains “deal with it”.
Not the spelling, but the underlying operation: converting a Decimal to int, or vice versa. The current limitations are about converting between int and str regardless of how they may be spelled.
For example, suppose people take the advice I’ve seen several times to write giant integers as hex strings instead. They’ll convert to int very quickly then, but if the app goes on to convert them to Decimal (perhaps to exploit that decimal has much faster * and / for large operands), the quadratic-time spelling is just Decimal(the_int), and when they want to move back to int at the end there’s the quadratic-time int(the_decimal_result).
If decimal became very popular across web apps, we can bet too that libraries would spring up that convert to Decimal for internal reasons that users aren’t even aware of.
Then we’d be back to “we can’t possibly expect users to catch this on their own -it’s all over the place - we have to make slow operations impossible to perform by default”.
So my view remains that int <-> Decimal are unlikely to get hobbled simply because they’re little used in “server farm” apps & libraries now. That may change over time.
Yes this is true, but many flagged comments were about semver, or other mildly related topics, which is truly irrelevant to the subject and could be discussed elsewhere should the community choose to do so. Not only that, but it adds unnecessary length to the thread which is very important to many of us here, i personally only arrived here for this specific discussion and would wholeheartedly appreciate that the topic remain about the subject, rather than single line responses like this thread was some kind of messenger service.
On 5 May 2020, a message was sent to the private PSRT mailing list about a “takes forever” json.loads() example involving loading a large integer. An hour later, a PSRT member replied saying that they had been informed about the same kind of vulnerability just a few hours earlier.
I don’t know whether it’s ever been exploited “in the wild”. It was kept secret because so very many libraries are vulnerable and attacks are so very easy to construct.
But that’s all water under the bridge. Secrecy is defensible in this case, and so is the mitigation they picked. People can argue about both of those - but it’s futile.
The Moving Finger writes; and, having writ,
Moves on: nor all thy Piety nor Wit
Shall lure it back to cancel half a Line,
Nor all thy Tears wash out a Word of it.
Since Decimal is being discussed quite a bit here in the thread, I just wanted to point out that the conversion of base-10 strings to Decimal instances is only O(n) if the _decimal module that is written in C is used. However, import decimal will automatically and transparently fall back to the pure Python implementation if _decimal is not available, and _pydecimal.py has completely different complexities: all the operations are implemented by converting the decimal string to a regular Python int object (O(n^2) as we’ve all been learning) and then converting the result back to a str (again O(n^2)).
Now the C implementation should be available ~always (except on PyPy ), but if not, the effect is pretty bad, if the application tries to use Decimal for arbitrary precision arithmetic with efficient conversion back and forth to str. I’m not sure whether I am suggesting anything here, should the fallback to _pydecimal.py produce a warning?
_pydecimal never actually gets used by anything in CPython so it no longer working in various situations with a limit enabled was considered a non-problem as there is no scenario in which the C _decimal module is not present. _pydecimal only exists as a reference / prototyping implementation within CPython at this point.
To be clear if anyone is thinking of using int(Decimal(str)) that approach is much more susceptible to an unarguable DOS vulnerability. Only a few bytes of input is needed:
int(Decimal('1e1000000'))
For me that takes 80 seconds. Add more zeros as you please but bear in mind:
Save your work etc first because you might end up needing a hard power off.
Ctrl-C probably won’t work. Make sure you’re ready to kill the process.
My point wasnt so much about a workaround, but rather about real world implementations of the decimal package, in any case it seems like its susceptible all round.
But that does beg the wider question which is, did you [all] give ANY condideration to those of us using decimal and whether there could be a potentially more suitable and universal solution?
Perhaps in my use case, which is that of django, im being a bit facetious because django enforces a max field length, but then i wonder where even this is susceptible due to the fact you still require user input which could be malicious, and could be specifically designed to stop your server workijg as normal.
That being said, a guarantee that python wont be affected by this is the short-term is most certainly and unequivocally going to be the smarter option, even if the long-term solution is discussing a better solution at great length in the community.
If I’m not mistaken, the best known complexity is O(n (log n)²), which I don’t think is near-quadratic.
I wonder: Was that a lack of knowledge about the best known complexity, or a lack of understanding or a disagreement about it not being near-quadratic? Or am I misinterpreting what was said there?
Until now, my part in this ended soon after the first reports, about 2 years ago. I was the “number guy” with canned knowledge of why things behaved as they did, what was possible in theory, the best that had been achieved in the real world, and the prospects for making various changes to Python’s conversion algorithms.
So I can’t say what was done outside the PSRT mailing list traffic. But we should certainly presume good will.
Did they give consideration to decimal users? Sure. Nothing about the decimal module is affected by this change. As already explained, _pydecimal.py is affected, but that Python module isn’t used at all by CPython anymore - and the mitigation here is to the CPython implementation, not to the Python language.
For all the words in this message thread so far, I haven’t seen a less intrusive/disruptive plausible mitigation. Turning off this change entirely is very easy, and in multiple ways (envar; cmd line; or set the limit to 0 programmatically via a new sys function). It’s clear that plenty of thought went into it.
Personally, I would have made it opt-in rather than opt-out. But that’s me. I don’t, e.g., have much use for Unicode either. I’m old and in the way .
Or if by “more universal” you’re disappointed by that, say, nothing was done to mitigate the int(Decimal('1e1000000')) example, that’s just way out of scope. The goal wasn’t to stop users from ever doing anything that takes more than a millisecond. It was to stop str <-> int timing traps that are ubiquitous in apps and libraries in very wide use on servers today.
Just misinterpreting. Near quadratic and subquadratic and efficient are all vague words to avoid trying to make a specific exact algorithm O claim because those would be argued over. The main point is that it is worse than NlogN.
So instead you see arguments over interpretation of the words instead of possibly over how many decimal points of precision put on an exponent representing Karatsuba or that faster things than that exist or that these points may be irrelevant within the size ranges we decided matter.
If we state this in terms of easy to understand lines of code to maintain, that’d be a more accurate representation of our actual motivation.
.
), but if not, the effect is pretty bad, if the application tries to use 
.