A lot of the things that would make this easier are probably good to have anyway for other reasons, like testability, general robustness, or clearer code. See command-scoped global temp dirs as an example that stands on its own, but would also facilitate in-process usage.
A stream of thoughtful, standalone, small PRs making improvements would be very welcome regardless of how anyone feels about in-process usage.
I’m going to just reiterate the point that the direction we’ve been most success in making a “pip api”, is by not making a “pip api” at all, and reimplementing (or ripping out wholesale) functionality that pip has or needs into it’s own library, and then retrofitting that into pip as the mechanism pip uses. That allows this new library to provide an actually good API for performing some task, while pip itself shrinks into a smaller surface area, which makes cleaning up the global state and such a much simpler thing to do.
I do find Chris suggestion quite practical, adding one entry point which mimics CLI, mainly getting arguments which are translated the same way the CLI does. The only contract offered is that this would match the CLI call, nothing else. The side effects of in-process usage will be mentioned in the doc, making clear that this works in YOLO mode.
In time, we may want to add extra API entry points for more specific behaviours but that should be based on how pip evolves.
The another library approach is not really going to every fly. Nobody wants to add an extra dependency for installing dependencies. It creates a chicken and the egg problem where you cannot use pip-api module and you are forced to use pip internals to bootstrap it in order to have an api to install other dependencies. Makes no sense to me. Only something shipped with python would work.
I think that in this area only evolutionary approaches have changes to succeed, where we may gradual and minimal changed towards the directions we want.
I am afraid that while documentation is welcomed its existence does not magically creates an API.
Will all respect, but I was a core I would have blocked it by asking to add a test that validates the runpy can really do that. The only way to implement a reliable API is to add tests. In fact tests are the only contracts an API can really provide, documentation is like a promise made during a political campaign.
Would anyone be against a PR that would add such test?
I personally do not find the runpy very nice-looking but it clearly is a very good start.
I also have another question related to future. I see extensive use of “unsupported” and “unsafe”. We do all understand that evolution does not happen over night and there are lots of broken-use-case and that it may take many years to reach a “supported” status. Still, why not using “experimental” term.
There is a big difference between telling people something is not production ready and discouraging them to do something. API will never happen if we do not start with something and slowly improve it.
If someone wants to fix one of the issues related to in-process use, they will be welcomed to raise PRs, with tests. The current test suites should assure that their changes do not break other random bits. It is a long process but it needs to starts somewhere.
A test is only needed to confirm supported behaviour. You seem to be missing the fact that this is still completely unsupported. As a pip core developer, I disagree with you that any test was needed here. And so does @pradyunsg (who reviewed the change and approved it), I assume.