Problem that appeared in version 3 with the help facility

Python Help
, ,
1

Not sure if this is the right place to ask this question, but if not, perhaps someone could point me to the correct place…

I have a published package (msmath) that runs in both Python2 and Python3. I do the following (after pip install msmath):

from msmath.ffield import *
F=ffield(2**256)
help(F)

In Python2, this shows the documentation for the newly created class as expected, but in Python3 it hangs. What is apparently going on is that the Python3 help facility is trying to execute some code (in particular the code that implements the generator property for this ffield) that is destined to take semi-infinitely long in this case. (It’s trying to factor 2**256-1.) What has changed between Python2 and Python3, and is there a way to avoid this problem?

2

It would help a lot to have the source code for this msmath module. I went looking on GitHub and found this:

Is that the one? It looks plausible but I’d appreciate confirmation.

3

Yes, that’s it. I thought there was a way to find the source from the package itself. It is a published package (installable with pip).

4

Thanks. There probably is, but you also didn’t say that you got it with “pip install msmath” so I didn’t want to assume. Either way, now we have the code and can explore.

It’s doing some, err, very interesting things. I’m still trying to figure out all the details of what it does.

5

Per your feedback, I’ve edited the initial post to include the fact that it’s a published package loadable with ’ pip install msmath ’

Thanks.

1 Like
6

So, what I’ve figured out so far:

  • The massive slowdown happens while help() is trying to process F.generator (which is documented as potentially taking a long time). If anyone wants to further explore this, note that ffield(8) will give you a useful point of comparison here as it is fast.
  • The actual result from F.generator does not seem to appear in the output anywhere.
  • The slowdown happens inside inspect.classify_class_attrs
  • classify_class_attrs does its work by, among other things, querying the attribute - thus triggering the property.

I’ll be honest, this is almost certainly not the only place where something in the stdlib assumes that a property can be queried quickly and efficiently. The best solution would be an API change, eg having F.is_generator() as a method call; but that’s a backward incompatible change. I can’t think of any backward-compatible fixes at the moment, though, so I’m going to leave these notes here for the next person and hope that someone smarter than me can figure out a solution.

7

Thanks. Python2 doesn’t have this problem and I’m not sure why Python3 is querying the property if it’s not going to use it for anything.

8

It’s trying to figure out what all the different attributes are. I don’t have a Python 2 installed any more, so I can’t compare, but try on a smaller one and see if there’s more information in Py3’s help() than in Py2’s. I suspect that the section on properties might be absent in Py2.

9

OK, I did try what you suggested, using F=ffield(8), doing help(F) in each interpreter. Other than Py3 including a lot of " from msmath.ffield", and a couple of " if defined", and this difference:

< class GF2^3_11(__builtin__.object)
---
> class GF2^3_11(builtins.object)
> | GF2^3_11(x)

Py3, but not Py2, has the line “Readonly properties defined here:” but doesn’t list any.

10

That’s probably what’s triggering it; but it isn’t able to find any that fit the description, since the class is doing strange things.

11

If you define __dir__ on the class to omit the expensive property, you should prevent the hang (since pydoc should only be querying the public attributes).

2 Likes
12

Thanks! That seems to work.

2 Likes