Is there a method to package relative URL wheels? how to create pre-built binary builds for speedy platform installs?

The use case is basically, bundle up a repo’s dependencies, send it out to a gemfury pypi, so that when i go to download the thing from furyi, it unzips and is pre-built for the OS that i am using ( ubuntu 20.04.3 ), no compiling needed.

I thought that maybe trying to package relative url wheels, with hacky setup.py

I was trying to understand “Direct references”: PEP 440 – Version Identification and Dependency Specification | peps.python.org

But, it wasn’t clear to me from what I could find what the way forward is to package up a bunch of pre built platform specific wheels. ( looks like it’s not possible )

pip install wheel
python setup.py bdist_wheel
pip wheel -r requirements.txt
zip --exclude "./venv_3_9_10/*" --exclude ".git/*" -r ../dependencies.zip ./
cd ../
curl --progress-bar --verbose -F package=@dependencies.zip https://${FURY_PUSH_TOKEN}@push.fury.io/somespam/

> /home/jm/pycharm_projects/somespam/v3/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py(495)make_requirement_from_spec()
-> return self._make_requirement_from_install_req(ireq, requested_extras)

(Pdb) l
490           specifier: str,
491           comes_from: InstallRequirement,
492           requested_extras: Iterable[str] = (),
493       ) -> Optional[Requirement]:
494           ireq = self._make_install_req_from_spec(specifier, comes_from)
495  ->         return self._make_requirement_from_install_req(ireq, requested_extras)
496   
497       def make_requires_python_requirement(
498           self, specifier: Optional[SpecifierSet]
499       ) -> Optional[Requirement]:
500           if self._ignore_requires_python or specifier is None:

(Pdb) specifier
'aiven-client@ file:///tmp/pip-install-enrqy_za/dependencies_b08b9414cf1b484c997d8ceeef5018a1/wheels/aiven_client-2.14.8-py3-none-any.whl'

(Pdb) comes_from
<InstallRequirement object: dependencies==1.1 in ./v3/lib/python3.9/site-packages (from -r /home/jm/pycharm_projects/somespam/requirements.txt (line 2)) editable=False>

Any suggestions?

More context:

I’ve looked at pip, asaman, cibuildwheel, flit, setuptools, etc, but there’s just a lot to go through, and I can never quite tell if the packaging/methods are deprecated, going to be deprecated, and what not.

Related to pips vendoring:

Other things looked at:

Hi @allen-munsch, I am under the impression that PyPI currently does not accept packages with direct references for dependencies.

You can still use them locally or use another package index.

Meanwhile you might want to consider vendoring · PyPI?

1 Like

In terms of pip you might have some luck if you use git/http URLs instead of file, but again I think that PyPI will not allow you to publish those packages.

1 Like

@abravalheri Thanks a bunch for sharing that info. I took a look at the pip specific method of vendoring, and found that there were quite a few files to patch because of the import spam.double_spam style imports. Also, a bunch of warnings, since a few *.so files were being vendored as well.

Still looking to see if there is a simpler way.

Hi James, another tool that I can mention for applications is shiv: shiv 🔪 — shiv documentation
(obs: shiv is useful for distributing applications, not libraries. I also don’t know how shiv deals with binary files)

1 Like

FWIW, I’ll flag that vendoring is not really meant to be used by anyone except pip. If someone files an issue on vendoring that doesn’t affect pip’s use of it, those issues are going to be closed as wontfix.

Is there a method to package relative URL wheels?

No, if you’re working with specific wheels, you’ll need to use a deployment strategy that encompasses distributing those wheels.

Why are you trying to bundle up all the dependencies into a thing uploaded to a Python Package index? I’ll suggest reading Overview of Python Packaging - Python Packaging User Guide and looking into which “level” of packaging you really want.

1 Like