Prevent .pyc creation when shadowing another module?

This is an utterly insane idea but hear me out here.

There’s a fairly common problem that follows broadly this pattern:

  1. Novice learns about cool new feature like turtle or flask
  2. Novice gets excited and starts playing around! Creates a file called turtle.py for experimentation.
  3. Circular import! Oh no. This isn’t working.
  4. Rename turtle.py to turtle_fun.py
  5. Wait, why are we still having problems??

The issue here is that there’s still a turtle.pyc caused by the circular import, and built from the errant file. (I’ve seen this from turtle, flask, and various others, but it could be any name and not necessarily from the stdlib.)

So here’s a crazy thought: Would it be possible to have current-directory imports not save bytecode if they shadow an existing module? This could be overridden with a variant of the -B option or PYTHONDONTWRITEBYTECODE, but making a default that’s less likely to shoot people in the foot would help people with recovering from a very common error.

1 Like

A temporary workaround might be to have a tool that finds modules that are shadowing other modules, so if you did py find_shadow.py flask it would tell you that a local flask.py or flask.pyc was shadowing an installed flask. An addition might be an option to delete or rename the said errant file(s).

That would potentially be of value, although as a third-party tool, it would be less useful to novices (the main target of this feature), so it’d need to be in the stdlib - making it nearly as high a bar to clear as an actual language/interpreter feature.

But whichever way it ends up being implemented, I would like to see a day when you can tell people an easy fix. My ideal is something that only requires the original .py file to be renamed and the problem’s solved; a second-best would be “rename the file, then type this command”.

Wouldn’t it be easier to ignore the pyc file if there is no corresponding py file? No other change in behavior, or new flags.

2 Likes

Tempting though it might be to once-and-for-all terminate the questions of “how can I conceal my source code while distributing the compiled version”, I don’t think that sort of backward-incompatible change is going to fly :slight_smile:

Although… maybe there is an exception for the __pycache__ directory. I’m not sure. But since I still see people having trouble with this, I suspect that the residual pyc file is still causing issues.

How recently have you seen this happen? As far as I can tell, ever since Python 3.2 and PEP 3147 (the __pycache__ directory scheme for cached pyc files), you have to really go out of your way to make a pyc file loadable without the .py file there.

If I create and import foo.py, it creates __pycache__/foo.cpython-313.pyc. If I delete foo.py and try to import foo, I get ModuleNotFoundError, even though __pycache__ is still there. In order to be able to import foo from the pyc file without the .py file present, I have to manually move ./__pycache__/foo.cpython-313.pyc to ./foo.pyc. Are novices really doing this regularly?

Back before PEP 3147, I always thought that including SourcelessFileLoader (the one that is able to load from .pyc without a .py) in the default loaders was a mistake; it should be available as an advanced feature for those who need it, but not enabled by default, because of the problems it can cause. (Those problems aren’t limited to module shadowing cases; I used to see it cause heisenbugs when a regular non-shadowing module was deleted from e.g. a source control repo, and then some other user would pull the latest from that repo but still have a .pyc sitting around for the deleted module.) But as far as I can see, the whole issue has been moot ever since PEP 3147.

If there really are scenarios I’m missing where this can still happen to unsuspecting users, I would much rather find a deprecation path to simply removing SourcelessFileLoader from the default loaders entirely (and document how to enable it), rather than try to detect module shadowing and magically do something different in that case.

4 Likes

I still see it periodically. Most recently an unconfirmed case this week where someone had a flask.py and then renamed it, but still had the same problem (unconfirmed in that the OP hasn’t yet responded since being advised to remove any pyc). If it turns out that this was due to Python 2.7 being used, then I withdraw the idea.

1 Like

In that particular case, my suspicion is that that user was still seeing an ImportError on the first line of their script, but it had a different cause (they were doing from flask import flask when it should have been from flask import Flask), and they didn’t notice that the error message contents had changed (hence the “still get the same error” report).

For reference, that thread is Trying to run a webapp . Judging from the latest replies, @mdickinson is right.

Ah, cool. I still see this sort of thing periodically, but it’s good to know that it isn’t from the same old cause any more.

I got quite spoiled by pyclean, which is installed on Debian-based distros by default: pyclean · PyPI

1 Like