I’ve got a situation where app.py imports now() from my_lib.utils and my_lib.utils imports now() from shared_lib.utils.[1]
This isn’t the only function that ended up being indirectly imported.
Ideally I’d like a linter tool that I can run occasionally that will turn all indirect imports into direct imports.
The reason it’s become an issue is because my pylance settings got changed in a way I don’t fully understand and can’t revert, and now pylance started labeling all indirect imports as reportPrivateImportUsage. I’m in 2 minds whether pylance is right. On the one hand, indirect imports are unnecessarily indirect, which is bad. On the other hand, I do prefer being able to refactor my_lib without having to change all the files that import from it.
This happened because now was initially defined in my_lib, but then I discovered I wanted to use it in a project that uses shared_lib but doesn’t have access to my_lib. I hope these details don’t really matter. ↩︎
I’m not aware of any tool that can rewrite imports for you as such but it might be easier for you to actually fix the underlaying reason pyright is complaining.
Normally in python there is no way to know if an import is meant to be used by a module internally or it is meant to be the “canonical”/”preferred” way to import a name. Since most imports are of the former kind that is the default assumption and that you can use it from that module is a side-effect.
The main way to tell a typechecker that an imported name is actually intended as a re-export for others to import from this module you can use the redundant from shared_lib.utils import now as now in mylib.utils to mark that you want that function to be considered as public api rather than an implementation detail.
It’s good practice to define an __all__ after your imports. That way you are explicitly declaring what your module is exporting. Otherwise you’re saying that everything in the module namespace (including imports and excluding names with an _ prefix) is part of that modules public API.
You can also pretty easily check against that with a script later if you want to make sure you aren’t indirectly importing anything.
Generally, I would recommend not doing star imports, but importing each thing separately, as a module. This doesn’t eliminate the issue you’re talking about, but it does make it more obvious. Consider:
from time import *
def print_time():
print(ctime())
vs
import time
def print_time():
print(time.ctime())
If you have these in another file, say “utils.py”, and import that into another script, you end up with:
from utils import *
print_time()
current_time = ctime()
It’s much easier to write a tool that checks for “utils.time.ctime()” (since utils.py imports time, so you can easily check for that) than to detect things that happen to be being star-imported from it.
That is not really true. Type checkers complain about private (_<name>) imports in both cases (e.g. pylance reportPrivateUsage, MyPy doesn’t complain), wether a __all__ is present or not.
__all__ is of then used to define the public API of some module (.pyi files are helpful in those cases too), but their intended usage is for the Python runtime to know which names get imported when doing from mod import *.
Apart from that, __all__ is just all non _ prefixed names by default.
Even when re-exporting private names, as long as their names are private, type checkers usually complain.
My preferred way to avoid this is to rename my_lib/utils.py to my_lib/_utils.py then create a new my_lib/utils.py containing from ._utils import functions, people, should, actually, get, from, here. Then my_lib.utils.now() will no longer exist. The underscore prefix in my_lib._utils.now() is a well known signal to tools and IDEs that its private and best ignored.
I mostly just use __all__ as a sanity check for myself (e.g. put the name in it when it’s implemented). It’s also useful when you’re exporting tons of names into a higher level module and using a from submodule import *.
You can also check if something is being indirectly imported by doing a 'func_name’ in module.__all__ assuming you have an __all__ defined.
I’m not looking to avoid this though, but to fix it.
My expectation is that there will be more imports that get moved from my_lib to shared_lib in the future.
I’ve already got a tool that’s able to detect indirect imports, and after such a change there suddenly are a lot of indirect imports.
What I’m looking for is a tool that’s able to fix them, so that it becomes a single line I can run from the terminal instead of having to go into all the files that the linter detects a problem in and adjust them manually.
but I get you’re saying this is a way to make autocomplete suggest the correct version for future development (?)
Unless you explicitly del all after defining it (which might work, idk) or setting it as something that’s not a collection of strings, this should work. As I said, iirc modern Python versions automatically add a __all__ with all non private attributes. This might however still make tooling (linters, maybe type checkers) complain, which would be my main concern.
Using it as a sanity check however is somewhat ‘smart’, although I would just use a to-do list (markdown or such) and use that, which should be easier to keep track of.
This part did. It doesn’t necessarily mean you’re doing from shared_lib.utils import *, as you could be explicitly naming the things you’re importing, but if that is what you’re doing, there should be no difficulty tracking down the imports by name. That’s why I guessed that you were star importing.
In my experience, most linters respect a defined __all__. I only define it because I tend to import names instead of modules so I can more easily tell from imports what the code in that file is doing and make inline type hints less verbose. It also keeps me from accidentally using the same name for something that might be available in an imported module.
Because modern Python automatically exports all ‘non-private’ names, this can lead to some confusing indirect imports.
At the end of the day this is entirely a design decision though. There’s multiple valid ways to avoid accidental indirects, and the most important thing is to come up with a method and stick to it.
Right. Based on the reactions my Original Post was unclear. Though I still don’t see how I should have written it instead.
I don’t have any issue tracking the indirect imports. As you say, due to the niceness of python import declarations (in absence of * imports), all imports can be tracked easily.
The issue is there are 20 files where I need to change from my_lib.utils import foo, now, bar into from my_lib.utils import foo, bar; from shared_lib.utils import now. The list of from imports is never quite the same. And sometimes I used from my_lib import utils; utils.now().
I can modify 20 files, but it doesn’t stop there, because utils.now isn’t the only function that has this issue, and there are other functions or classes that might cause the same issue in the future. That’s why I’m looking for a cli tool.
You could have avoided my personal confusion by posting the exact code you were using, but ultimately it isn’t a hugely significant distinction.
Ah! Well, this sounds like something where you may want to consider an AST parsing script. Python gives you some really cool tools and they’re a ton of fun to play with. I’m totally not addicted to parsers and stuff, I can stop any time… ahem. Anyway. You can make yourself a Python script that reads in each of the programs in some directory (you probably are reasonably familiar with walking a tree and doing something on each file, if not there’s plenty of tutorials out there), and then for each one, you do something like this:
module = ast.parse(file_contents)
class Visitor(ast.NodeWalker):
def visit(self, node):
if the node is an Import node:
check the source
check the list of imports
oh it seems you are getting a thing from the wrong place
print("Possible problem in this file")
Visitor().visit(module)
This will recursively walk through all the nodes in the tree, so you can find imports that might be inside functions or buried anywhere, and it won’t be broken by formatting changes. This is a really effective way to track these things down.
You can then use this to either make changes as a batch, or to flag things as errors as part of a verification pass that you run regularly (eg as part of CI/CD). And you could even do the same sort of parsing (or maybe just import utils and check its __all_) to build your list of problems.
I think that should do what you want. If it doesn’t, it’ll still be enormous fun to play with! (I’m totally not projecting my own personal appreciationg for AST walking. The funness of this is absolute and universal. This is a fact.)
I wrote a tool for this at work recently; I can’t share it but it wasn’t too hard. libcst is the best tool for this job, as it lets you rewrite a file while keeping formatting the same.
