Pip V22 install --report in Json to get requirements

piscvau · August 6, 2022, 3:46pm

I am using pip V22 to install a few packages and I would like to get from the json report (pip install --report -) just the version number and the requirements so as to keep this in a requirements file.

I am not familiar at all with json so please forgive me.
I tried json.loads(report_stdout) and I get error
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

the stdout comes from a subprocess. I tried utf-8 enconding or binary and I always ge the same error!..

sbidoul · August 6, 2022, 4:21pm

Hi,

For historical reasons, pip install logs to stdout. So you need to use --quiet when sending the json report to stout.

piscvau · August 6, 2022, 4:30pm

Thank you for you answer. I looked into the pip V22 documentation and it says that the option --report - sends the report to stdout:
–report
Generate a JSON file describing what pip did to install the provided requirements. Can be used in combination with --dry-run and --ignore-installed to ‘resolve’ the requirements. When - is used as file name it writes to stdout.

from pip install - pip documentation v22.2.2

and I do not see the --quiet option. DO we talk about the same release?

sbidoul · August 6, 2022, 4:55pm

Yes we do. I take note to improve the docs for the next release.

piscvau · August 6, 2022, 7:14pm

Very strange; if I do include --quiet --report - and --dry-run , I do get a json report but it contains only the report about the pip package, not the report about the installed packages or would be installed packages!..

sbidoul · August 6, 2022, 8:31pm

That would be strange indeed. If you think there is a bug, please open an issue on the pip repo with a detailed reproducer.

piscvau · August 6, 2022, 8:33pm

Yes but I am using subprocess to call pip install from a python script. Is this a deviant use for pip?
Also I used the debug function of my IDE .
If you confirm this not a deviant use I will produce a reproducer

sbidoul · August 6, 2022, 8:47pm

Invoking pip as a subprocess is perfectly legitimate.

piscvau · August 7, 2022, 10:47am

Thanks for you help. No bug it was a misunderstanding from my side.

sbaack · August 8, 2022, 8:30am

Hi @piscvau, I developed a tool that is doing what you’re asking for I think: Using pip install --report to create requirements files: GitHub - sbaack/iso-freeze: Pin requirements using new `pip install --report`

You can check this file to see how I turned the report into JSON and extracted relevant information. Hope this helps.

piscvau · August 8, 2022, 10:33am

yes exactlly this is what I was looking for Thank you so much.
I just looked into the idea of creating an sqlite database using inspect and sqlitebiter . but sqlitebiter does not like the output of inspect see here incompatibility between pip V22 json output and sqlitebiter · Issue #88 · thombashi/sqlitebiter · GitHub.
Anyhow your proposal is a very nice shortcut.
May I suggest that your tool be referenced into the pip documentation . It would have save me a lot of time. I did google to find an existing package but did not hit yours!..

sbaack · August 9, 2022, 10:06am

Glad it’s useful! Not sure if this tool should be mentioned in the docs. At this point, it’s experimental and for a serious project people should use pip-tools which accomplishes the same without relying on pip install --report (or they could use tools like poetry or pdm with their own lockfile formats).

CAM-Gerlach · August 30, 2022, 3:44am

Somewhat related re: pip programmatic invocation

github.com/pypa/pip

Having a runtime dependency on pip: make it explicit whether this is supported or not

opened 08:01AM - 23 Jul 22 UTC

closed 10:24AM - 30 Aug 22 UTC

pfmoore

type: feature request

### What's the problem this feature will solve? See https://discuss.python.org/…t/pip-plans-to-introduce-an-alternative-zipapp-deployment-method/17431/22 for context. Basically, people are suggesting that if we distribute a zipapp version of pip and as a result, environments without pip become more common, they will add pip as a runtime dependency to force the environment to contain pip. While locating a copy of pip to run programmatically is a non-trivial problem (this is one of the reasons we're hesitating over making an official zipapp), if projects start to add a dependency on pip, this will block the option of finding other, less disruptive, ways of achieving pip-less environments[^1]. In addition, pip has no programmatic API, and the normal use of declaring a dependency is for your code to call a library from Python. People don't declare a dependency on `bzr` so they can run `bzr` commands using `subprocess`, any more than they do for git. Why should pip be different? [^1]: Which I believe are important - `py -m venv` takes 6 seconds on my machine, whereas `py -m venv --without-pip` takes under 0.1 second. ### Describe the solution you'd like Explicitly state in the documentation, alongside the statement that pip has no supported Python API, that we don't support including pip in a project's runtime dependencies. ### Alternative Solutions Leave our position unstated and implicit. ### Additional context Some of the alternative approaches to relying on pip being in the environment include: 1. Projects assume the existence of a `pip` command, and run that (the same as they would for something like `git`). This is a reversion of our existing advice that people should run pip via `subprocess.run([sys.executable, "-m", "pip"])` which would be frustrating churn for our users. Otherwise, though, it's exactly what we'd expect people to do for any other command line tool like `git`. 2. Projects add a configuration option to allow the user to state how pip should be invoked (with the default remaining `python -m pip`, at least for now). Not all projects have a configuration system, so this might be a bigger change than is ideal. 3. Projects try to auto-discover a suitable pip command. This is likely both costly and error prone. So I think that relying on pip being in the environment may need to remain the norm for now. And that may well impact the advisability of shipping an official zipapp. However, I still don't think that projects adding a dependency on pip is a good idea. ### Code of Conduct - [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).

and