Is there any standard way to use a package from a private pypi in build-system requires

I’ve run into an edge case where we want to run some internal code generators as a part of the build process. Under normal circumstances we’d just use a build script in the package’s repo and add the dependencies to [build-system] requires = [...].

But we’re now looking to move the content of the build script into its own package to avoid having to copy-paste a complex script between multiple repos. What I can’t find is a standardised way to reference a private PyPi repo in build-system.

I can see others have run into a similar problem with Poetry here. But the solution doesn’t help us much as we’re not exclusively using poetry and the solution is basically to have customised poetry installed on every workstation.

Am I missing something?

1 Like

Is telling everyone at work to run:

pip config set global.extra-index-url https://custom/pypi/simple

once when they set up their machine an option?

1 Like

Probably no; the irritating wrinkle here is that there’s an undefined number of these private repos for administrative reasons. The ideal situation would be if there was a way to put that info into pyproject.toml

They get added and occasionally decommissioned / merged as organisational structure shifts.

Looks like I’m not the first on this. Looks like a gap in PEP517

1 Like

Is there any standard way to use a package from a private pypi in build-system requires?

Yes and no.

Yes, in that you can use a custom Python package index (a mirror, proxy) that is configured to deliver the correct packages, which should be interoperable as long as the Python package index implements the standard specifications.

No, in that you can not write this down in a standardized and interoperable manner (i.e. in a standardized section of pyproject.toml). The closest features I could find are those, maybe one of those might work somehow, untested by me:

1 Like

As a work around, I wonder if there is a build backend that can, itself, install other packages. :thinking:

The typical way I’ve seen his done is to setup an internal pypi mirror and add your private packages to it.

Either passthrough requests for public packages upstream or manually replicate down packages deemed safe to your private local pypi.

I suppose if I really was in your situation then I’d just forget all about build-system.requires, put my build dependencies in a build-requirements.txt (which may contain --extra-index-url=https://... options) then use the --no-isolation flag to build.

pip install -r build-requirements.txt
build --wheel --no-isolation

If you still want some flavour of build isolation, you can just about run build under pipx.

pipx run --pip-args="-r$PWD/build-requirements.txt" build --wheel --no-isolation