Make ad-hoc command line execution friendlier with options to import a module and output the value of an expression

Ideas
1

The frequent requests of adding a utility CLI to certain modules such as Add a minimal CLI functionality to urllib.request appear to usually stem from the fact that the import statement has to be separated from the statement that actually performs a desired task by either a semicolon or a newline, making the command line unnecessarily ugly especially when embedded in a configuration document:

"healthCheck": {
    "command": [
        "CMD-SHELL",
        "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:8080/health')\""
    ],
    "interval": 30,
    "timeout": 5,
    "retries": 3,
    "startPeriod": 10
}

I think we can add two repeatable options, -M <comma-delimited module paths> and -F <module path>, which will be equivalent to writing import <comma-delimited module paths> and from <module path> import *, respectively, when preparing the namespace before the code in the -c option runs, so:

python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')"

can be rewritten as:

python -M urllib.request -c "urllib.request.urlopen('http://localhost:8080/health')"

or in many cases more concisely as:

python -F urllib.request -c "urlopen('http://localhost:8080/health')"

Also, oftentimes we just need to output the value of an expression such as random.randrange(3):

python -c 'import random; print(random.randrange(3))'

We can make such usage easier with a -p <expression> option (as in “print this expression”) so that the above can be rewritten as:

python -F random -p 'randrange(3)'
2

What would it be like if it were something like this?

python -m run 'random.randrange(3)'

and it knew to import whatever was necessary and then print the result of that expression? This can be done with no infrastructure in the Python executable, everything would live in the run module - and, added bonus, it could be something developed externally to Python, with the potential to become part of the stdlib if that were to be beneficial.

1 Like
3

Sounds great, but if run has to exist as an external dependency I would just use the perfectly good fire instead.

This proposal is meant to make Python CLI friendlier in barebone environments.

4

This would be very convenient if it would be supported directly by Python. I wouldn’t mind -c behaving like that, but due to backwards compatibility concerns a new option like -C would probably be better.

5

Right. What @Rosuav suggested would be great as a built-in command line option, which can be implemented with a proxy builtins mapping equivalent to:

from collections import UserDict

class AutoImport(UserDict):
    def __getitem__(self, name):
        try:
            return super().__getitem__(name)
        except KeyError:
            return __import__(name)

def run(source):
    return eval(source, {'__builtins__': AutoImport(vars(__builtins__))})

if __name__ == '__main__':
    import sys
    print(run(sys.argv[1]))
6

You’d need something a bit smarter to handle submodules. That one fails if you need to do a import foo.bar.

~> xdg-paste | python - 'urllib.request.urlopen("http://127.0.0.1")'
Traceback (most recent call last):
  File "<stdin>", line 15, in <module>
  File "<stdin>", line 11, in run
  File "<string>", line 1, in <module>
AttributeError: module 'urllib' has no attribute 'request'
7

If I understood the proposal by @Rosuav and the suggestion of a way to implement it by @blhsing then quite a few simple command line programs would run without extra verbiage, albeit a tad slower.

I am wondering about cases it would not do what is expected.

  • Obviously, it would fail when the function is not one of the built-in modules and that is fine. But what if the builtins are changed in some version of python so it is no longer found, or perhaps the order is changed and a different function with the same name is found instead?
  • I also wonder if the proposal will handle multiple items like random.randrange including nested versions where such a function has arguments with other such functions.
  • And, as @bwoodsend pointed out, some modules are nested albeit I am not sure if builtins lists them properly so it works.

I remember a somewhat related idea I have seen. It is a loose attempt to take lots of the popular UNIX utilities (now more ubiquitous in places like LINUX and elsewhere) and rewrite many of them as simple python scripts. Often, you can write a one-liner to replace a common version of something like cat, more/less, cp, head/tail, ls, cut, and so on.

Clearly, something like this can often be done well, even as a one-liner, if you have a module that does it and the module does not need to be explicitly imported.

And, to be very obvious, there are many utilities someone like Chris might want to implement, often on the fly for one-time used, that would benefit.

8

Yes, this can be worked around by wrapping the dynamically imported module in a proxy module that delegates attribute access back to a proxy mapping:

from types import ModuleType
from collections import UserDict

class AutoImport(UserDict):
    def __getitem__(self, name):
        try:
            return super().__getitem__(name)
        except KeyError:
            class _ProxyModule(ModuleType):
                def __getattribute__(self, name):
                    return namespace[name]
            if (self_name := self['__name__']) == 'builtins':
                module = __import__(name)
            else:
                module = getattr(__import__(f"{self_name}.{name}"), name)
            namespace = AutoImport(vars(module))
            return _ProxyModule(name)

def run(source):
    return eval(source, {'__builtins__': AutoImport(vars(__builtins__))})

if __name__ == '__main__':
    import sys
    print(run(sys.argv[1]))
2 Likes
9

Now I’m wondering if the regular ModuleType class could behave this way. It would be nice to never again have to explain why import urllib.requests is required but import os.path is redundant.

1 Like
10

Good idea. I don’t see any real downside to making ModuleType always attempt to import a missing attribute as a module, as I can’t imagine any good use case for any existing code to use the produced AttributeError to do anything useful other than to also perform an import on demand.

It just feels far more intuitive to be able to access request from urllib once urllib is imported, knowing that urllib.request is importable.

11

Isn’t this essentially lazy imports? At least, it seems very similar, and I suspect the reasons it might not work would be related.

edit: by “this” I mean generalizing the import behavior to all modules, not the main topic of this thread.

12

It’s similar but different. Lazy import is about importing a module only when an attribute of the module is accessed, whereas what we’re talking about here is for a sub-module to be imported when it is accessed as an attribute of its parent module.

But you’re right that the warning in the documentation of importlib.util.LazyLoader may indeed apply here:

For projects where startup time is critical, this class allows for potentially minimizing the cost of loading a module if it is never used. For projects where startup time is not essential then use of this class is heavily discouraged due to error messages created during loading being postponed and thus occurring out of context.

13

READERS DIGEST CONDENSED VERSION: Other ideas that could perhaps allow the functionality.

If I understand it, the need being expressed here is to call python as a sort-of one-liner to do some code on the command line but without explicitly doing a needed import statement.

Some of the suggestions look interesting and I wondered about others.

  • I see a PYTHONSTARTUP environmental variable which can be set to a filename to be run before any code. Could you have one or more files set to import what you need.

  • Another idea, on UNIX-like environments you could use things like a shell-script that reads a “HERE” document of any number of lines. It might look like this:

python args <<!
import blah
blah.func(args)
!

Obviously this allows fairly large scripts and no changes to python as it basically is asking Python to receive a program on <stdin>.

  • Another silly idea is to load everything if the command line asks with some new argument, and then you can reference everything easily enough. Clearly, not a cheap solution. But for commonly used modules, maybe you could have some that load just one named module or a smaller subset often used together.

  • Yet another idea does not currently work on python but can work fine in languages like R with delayed evaluation. If you wanted to run mod.func(args) you could ask to run avail(mod, mod.func(args) which would first import mod and then return whatever mod.fun(args) returns when evaluated. Without that feature, the only kludge includes providing character strings and running eval which tends to be frowned on!

Again, not arguing against other solutions and would love to know if some of these ideas are just plain wrong!