Adding support for `wsgi_scripts` entrypoint?

Packaging
1

pbr has long had support for a wsgi_scripts entry point. This is the equivalent of the console_scripts entrypoint but for WSGI application functions and it allows us to avoid using scripts (which has plenty of well known issues). The expectation is that the function you point to must return a function or callable that matches application function signature defined in PEP 333. For example:

setup.cfg:

[metadata]
name = foo
# ...

[entrypoints]
wsgi_scripts =
    foo-api = foo.wsgi:init_application

foo/wsgi.py:

def application(environ, start_response): ...

def init_application():
    return application

When installing this, you’ll end up with a script like the following:

#!/usr/bin/python3.11
#PBR Generated from 'wsgi_scripts'

import threading

from foo.wsgi import init_application

if __name__ == "__main__":
    # <snipped irrelevant debug-style application setup>
else:
    application = None
    app_lock = threading.Lock()

    with app_lock:
        if application is None:
            application = init_application()

At the moment, this is all built on setuptool’s easy_install which is deprecated (presumably for removal). I’ve been looking at how we can migrate away from easy_install but have found that management of easy_install is now handled by the installer, pip for the most part. As such, I suspect a long-term solution to this would likely need to live in pip. I imagine this is broadly useful, assuming virtually all Python-based web applications are deployed with WSGI and a more than few of them are being packaged and installed.

My question is this: would this make sense as something to support in pip and potentially any other installers where it makes sense? If so, what would be the best place to start? A PEP? A PR against pip? Something else? I have a few other open questions (like what do we do about ASGI) but those could be resolved once I have a place to start writing things down.

2

I don’t understand what the proposal is here (sorry, I don’t know how the pbr functionality you mention works).

You can define whatever entry points you need in a project. The only way the “console entry points” are special is that installers are expected to make executable commands that run those entry points. For a WSGI application, the generated script you show uses wsgiref.simple_server - that’s not likely to be the right choice for a production app, and so it certainly wouldn’t be something we’d want to bake into installers. On the other hand, I’d imagine it would be reasonably straightforward to make that script generic, by loading an entry point using importlib.metadata, where the library to load (foo) is an argument to the script. Something like:

parser.add_argument('--app', help='WSGI app to run')
appname = args.app
entry_points = importlib.metadata.entry_points(group="wsgi_scripts", name=f"{appname}-api")
init_application = entry_points[0].load()

That could be done without needing any sort of standardisation, using the same metadata as you already declare.

3

Yeah, ignore the CLI fallback mode (all the stuff inside the if __name__ == "__main__": block): that’s definitely not something I’d bake into installers. I’ll snip that from the original post.

I guess my main ask is how would one distribute a script for a WSGI application without relying on setuptools’ scripts feature? afaict, this feature is considered legacy and isn’t support by the likes of poetry or flit so I’m not sure what folks are using, if anything, here.

4

As a console entry point? Just write the script and include it in your project, and create an entry point that runs it.

If you’re asking for something to generate that script for you, then installers won’t do that, as it’s specific to WSGI applications, not a general requirement. Maybe a build backend (or build backend plugin - tools like PDM and hatch support plugins) could do that for you?

5

If I’m not mistaken, that wouldn’t do much for you. That would generate a script wrapped in if __name__ == '__main__': which wouldn’t work with any WSGI server, e.g. we’ll get something like the following in the scripts path (seen by sysconfig.get_path("scripts")):

#!/tmp/venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from foo.wsgi import init_application
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(init_application())

When what we actually want is the following, again in the scripts path:

#!/tmp/venv/bin/python
# -*- coding: utf-8 -*-
from foo.wsgi import init_application
init_application()

(assuming you strip away all of the debug stuff, which really isn’t relevant)

Okay, that’s probably my answer. Given how important WSGI is the Python ecosystem I figured this would be analogous to the console_scripts entrypoint and broadly applicable. If that’s not the case though so be it.

Thanks for the feedback!

1 Like
6

I am not sure if I understood the use case correctly, but isn’t it possible to just write a regular function for that and expose it via entry-points?

If I am not mistaken, for ASGI, you can do something like:

# pyproject.toml
[project]
name = "helloworld"
version = "42"
dependencies = ["uvicorn"]
scripts = {run-helloworld = "helloworld.mod:run"}

# src/helloworld/mod.py

async def app(scope, receive, send):
    await send({
        "type": "http.response.start",
        "status": 200,
        "headers": [[b"content-type", b"text/plain"]],
    })
    await send({"type": "http.response.body", "body": b"Hello, world!"})


def run():
    import uvicorn
    uvicorn.run(f"{__name__}:app", host="127.0.0.1", port=5000, log_level="info")

You can run this with run-helloworld in bash.

So I imagine that it would also be possible to do something similar for WSGI, it just depends on the WSGI server of your choice exposing a Python API…

7

As a maintainer of a WSGI application framework, I’m not clear how this proposal would work. There’s no standard for how to run a WSGI server, or how a WSGI application callable is prepared, so I’m not clear what sort of script you expect to be generated by installers. (This all applies to ASGI too.)

If you just mean that you want a standard way to specify “this is the WSGI callable provided by this wheel” you’d be missing the fact that a factory function is often used to generate the actual WSGI callable along with extra configuration arguments, which could not be described by an entry point alone.

If you need a replacement for the old setuptools concept of scripts, then using the modern project.scripts would be appropriate without any new spec.

1 Like
8

i can shed some light on the context here

pbr is the build system we use for openstack.
pbr wraps setuptools and delegates most of the heavy lifting to setup tools.

setuptools wraps distutiles and both setuptools and distutils supprot extendtions.

pbr uses the extendtion functionatlity to defien a new entrypoint wsgi_scripts which uses a factory function to generate a standardised wsgi script for all openstack rest api.

each openstack project that has a reset api uses the wsgi_script entrypoint

here is an example of how that is used in the nova project

console_scripts =
    nova-api = nova.cmd.api:main
    nova-api-metadata = nova.cmd.api_metadata:main
    nova-api-os-compute = nova.cmd.api_os_compute:main
    nova-compute = nova.cmd.compute:main
    nova-conductor = nova.cmd.conductor:main
    nova-manage = nova.cmd.manage:main
    nova-novncproxy = nova.cmd.novncproxy:main
    nova-policy = nova.cmd.policy:main
    nova-rootwrap = oslo_rootwrap.cmd:main
    nova-rootwrap-daemon = oslo_rootwrap.cmd:daemon
    nova-scheduler = nova.cmd.scheduler:main
    nova-serialproxy = nova.cmd.serialproxy:main
    nova-spicehtml5proxy = nova.cmd.spicehtml5proxy:main
    nova-status = nova.cmd.status:main
wsgi_scripts =
    nova-api-wsgi = nova.api.openstack.compute.wsgi:init_application
    nova-metadata-wsgi = nova.api.metadata.wsgi:init_application

as you can see the syntax is identical to console scripts and the behavior is also the same

the only thing that differs is the waht the script is generated in pbr

pbr uses a diffent template for the content of the generated script but its installed in exactly the same way and in the same location as the generated console scripts.

this has not been a problem until rechent version fo pip which started moving to pep 517/660 support

recently we moved our ci system to install openstack in a singel global virtual env instead of
doing “sudo pip install -e ” that is to prepare for the upcoming changes
to debian/ubuntu where that will be blocked

in doning that we discuoved it did not work on centos 9 stream
so for centos we left it install globally instead of in a vnev.

i started looking into why that was failing 2 weeks ago and discover that it was related to teh recent release fo pip which removed calling setup.py as a fallback if pyproject.toml nont found.
instead new version of pip invent calls to setuptools internally when a setup.py is present and that does not use pbr. as a result our wsgi_scripts entry point is not processsd and the rest apis cannnot start as the wsgi script are not found.

the obvious soltion seamed to just be add pyporject.toml to all the openstack repos
which i started here.

review.opendev.org/q/topic:pip-23.1-support

pbr already support building with pep-517 after all so that shold be a small change.

that where teh prblems begin becasue that almsot worked perfectly exception for one edge case.

pbr did not supprot pep 660 editabel wheels that was fine excpte for the fact
we use tox for unit/functional test. for reasons we run your py39 tests on ubuntu 20.04 which has py38 as its default interperter. thats fine ubunut hase py39 avaiable as a package so we just insall it.

adding a pyproject.toml file is enough to enable isoloaded build
with tox that means that the wheel is build in a seperate tox venv “.pkg” and that venv uses you systems base python in the case py38

we are building universal wheels so that should not be a probelm excpet when that wheel is installed into the py39 venv pip hard fails saying that the backend (pbr) does not supprot editable mode.

this only happens when using a different .pgk python veriosn for the test venv.

so again the fix seams simple lets just add support to pbr for pep 660 so that it can supprot editabel_wheels

so we did that and we did a 6.0.0 release which fixed our unit tests.

the change in pbr just delegates to setuptools to do the heavy lifting

this si where the problems start.

neither setup tools or pip know about wsgi_scripts

setup.py develop is not being called anymore so pbr cannot hook that to install the wsgi scripts ourslevs
and the console_script setup.py install_scripts command is not being used either.

the problem stems form teh fact that pep-660 did not standardise supprot for script or datafiles

it gave special status to console_scripts and gui_scripts and did notn provide a way to execute other script handelers.

so for openstack to supprot pyproject.toml we need a way to adress this gap as this si a blocker for use to consinute adding supprot for this going forward.

im not sure thtat adding wsgi_script is the best way to do that and i woudl prefer a more generic way to refernce a handler function that will be used to render the templsate with a standard set of args.

if anyone has adviace as to which of the three standardised function in pep660 could be abused in the interim to generate the wsgi scripts

build-editable, get-requires-for-build-editabl, or prepare-metadata-for-build-editable

i could try and make pbr do somethign fomr there but the issue is not really with the wheel building but the installation.

building a non editable wheel with pbr does include the scripts in teh wheel
the problem is we now have no generic way to install them.
in the ediable case they are not generated at all because the hooks we were using are nolonger used.

9

apprently you can only have 2 links per message so here is a copy of my orginal message with links to code
https://paste.openstack.org/show/822305/

10

I won’t say @SeanMooney put it succinctly (:wink:), but they likely have hit the nail on the head. PEP-660 appears to have left no mechanism for distributing WSGI scripts (or any other kind of script that isn’t a console or GUI script). uWSGI has the --wsgi-file opt / UWSGI_WSGI_FILE env var (plus --file / UWSGI_FILE aliases), while mod_wsgi has the WSGIScriptAlias config option. Those expect to point to a wsgi.py file (name not important) that contains an application object. I note that frameworks like Flask and Django document how to create these wsgi.py files but give no indication about how to distribute them or install them into $PATH.

What is the expectation here, with the removal of the distutil-derived scripts mechanism? Is no one else distributing e.g. myapp-wsgi executables that can be used with these frameworks?

11

There’s no mechanism in the wheel standard for distributing automatically created scripts other than console entry points. This has been true for many years now, it’s not new with PEP 660. Individual build backends like setuptools can offer ways to generate such scripts, but as they can’t be stored in wheels, you would have to invoke the backend directly to generate them.

What has changed is that pip has finally stopped[1] special-casing setuptools and invoking its interface directly. That gives you a problem, as you say, because you were (maybe unknowingly) relying on a non-standard interface in the install process.

Your options at this point are probably:

  1. Modify your application (or your build backend) to include the WSGI boilerplate in a function that can be called as a normal console entry point script. When I suggested this before, you said “that’s probbaly my answer”. I’m not sure why you have changed your mind.
  2. Use a custom install process, maybe by asking your users to run a post-install function after installing the wheel of your project. Pip won’t run a post-install automatically, (a) because there’s no standard for doing so, and (b) because that invalidates the promise that installing a wheel cannot run any untrusted code.
  3. Propose a new standard, adding support for other types of executables that can be created by installers when installing a wheel. This will be a very hard sell, because once we start adding application-specific types of support, where do we stop? And while WSGI is an important type of application, it’s still just one type.

  1. or at least is in the process of stopping ↩︎

12

Sorry but why are WSGI scripts entrypoints needed?

I’ve used a couple frameworks and always ran a WSGI application server command, with a specification of a WSGI entry-point function in its config or parameters.

1 Like
13

Yeah as I said above this doesn’t make sense given how WSGI web applications would actually be deployed. We could maybe help you figure out what does make sense, but it’s hard to understand the concrete problem you’re trying to solve. A minimal reproducible example, without PBR, OpenStack, etc, only a “hello world” WSGI application, could help demonstrate the problem you want to solve.

14

@davidism I posted a reproducer here. Hopefully that helps explain what I’m missing/asking for. Effectively we had a mechanism for dumping a script that contained a WSGI application callable within it. This script would be installed in bin, wherever that is (depending on how and where you’ve installed your package). That’s allows for a well known path that we can point to with e.g. mod_wsgi’s WSGIScriptAlias That can be then specified We no longer have the ability to do this nor mark any other arbitrary scripts as something that should be placed in bin.

To @merwok’s point, I’m aware that many WSGI servers allow you to define the location of this callable using a Python path instead (iirc, gunicorn only supports this configuration mechanism). There’s a long history of relying on the script based approach within OpenStack though and replacing that across the various deployment frameworks is going to cause much wailing and gnashing of teeth for effectively zero benefit

1 Like
15

I don’t see anything there that requires a wsgi-script. You would want to configurat a WSGI server (and all servers are different), but you’re packaging a WSGI application. That deployment configuration would be somewhere else, such as a Docker file.

16

That’s my point though: I don’t want to run a WSGI server. I want to give the user a WSGI script installed on $PATH that contains an application callable in it (and maybe some code for running a debug WSGI server).

17

Then define a normal console-script that runs the application exactly how you want. What would the wsgi-script be for?

(Oops, I edited my previous post by mistake.)

18

I’m clearly doing an awful job explaining what we have and why we want it :see_no_evil: Sorry. We can probably close the discussion down. I now have confirmation there’s no way to inject arbitrary scripts into wheels and have some idea why (thanks, @pf_moore). It seems we’re just going to have to drop this feature from pbr and get the deployment tooling folks to either start creating and installing their own WSGI scripts or to start using Python paths (path.to.module:application).

To @pf_moore’s point about why I’d changed my mind, I didn’t spot that your two points were related and had read the second point as “why don’t you implement wsgi-scripts-like functionality into your own build backend” and had then realised this couldn’t work due to the wheel thing.

1 Like
19

After reading the whole thread a few times, it’s still not really clear to me what you’re looking for…

IIUC, you want a way to install Python files (a) in a location that’s on $PATH because you have something that searches them on $PATH, and (b) that are not meant to be run, instead they are imported and some special attribute (application) is looked up on them.

Is that correct? Or am I misunderstanding completely? It’s pretty weird to put something on $PATH if you don’t want to run it…

Anyway: the current standards do have support for installing arbitrary scripts in bin/. You just put them in the scripts/ subdirectory of the .data directory inside the wheel. See the spec and Pradyun’s reply on the previous thread. What is not supported is generating these scripts at install time. You must generate them at build time and include them in the wheel. If your generated “script” depends on the project metadata but doesn’t depend on the user environment, then you can use that. I don’t know much about setuptools, so I can’t tell you if it has an easy way to use this feature of the wheel format.

2 Likes
20

The current pbr wsgi script we generate can be used either as a command line executable or as a wsgi script that is imported.

i was on PTO yesterday but shortly after my original post i read over the wheel spec while cooking dinner and i settled on trying the approach suggested by jean i the most recent post.

when we build non editable wheels we do in fact pre-generate the wsgi_scripts and add them to the scripts data dir in the wheel.

we should be able to reopen the editable wheel produced by setup tools and append the generated scripts to that. im going to hack on our build-editable implementation to do that this morning https://github.com/openstack/pbr/blob/master/pbr/build.py#L82-L91

we have looked at also taking thte console_script approch but i think part of what is being missed in that case is we wanted to avoid code duplication. openstack is a collection of diffent software project that work togheter and we and orginal create the wsgi_scripts entryp point so that we could provide a consitent approch to generating a script with the same functionality for our diffent proejcts.

we could just provide a lib for everyone to use to get similar code reuse but its a non trivial amount of work when you 34 project all replying on this and efectivly “getting it for free” from the build system they are currently useing.

assuming i mange to fix this via the build_editable hook point by just using

ill comment back to let people know.