Is it possible to make a package (server) that can install itself or another package (client) from source in another environment/machine?

Python Help
1

Hi, I’m working on a project where I need to develop a Python package that acts as a server that can dynamically start Docker containers and install a client within them. So i need access to the source code of the client somewhere in the package.

Just cloning the client’s Git repository or using pre-built wheels for installation is not possible, because this could lead to version mismatches between the client and the server if i am currently working on the package. I don’t want to have to push my code everytime i do a change to the client.

I was thinking of adding the sdist/wheel/tarball of the package directly as a package data file, so the server can copy it to the client and install it. But i don’t know if i can do that, I believe that scripting within pip install commands or pyproject.toml files is not supported now, as it was with setup.py.

My project structure is like this so far :

  • pyproject.toml
  • src
    • client
    • server

Also, in an ideal world, if i am working in editing mode (pip install -e), the client source code would be also dynamically updated, and i wouldn’t have to reinstall everytime i run the code and start a new container.

How can i achieve that ?

2

Your server code, at run time, can invoke pip. I must not be understanding what the blocker is, I do not see a problem here.

1 Like
3

Let’s say i am working on my code, so i am ahead of my branch pushed somewhere with git. Now i want to test it or debug it.

  • First i install the package
  • I start the main code
  • I launch a new docker container, connect to it
  • Now i want to install a server on it
  • Since my codebase contain both the server and client, i want my package to be able to install itself inside the docker container, and start in server mode

I can call pip from inside the docker container, but on what ? I don’t want to use a remote repo, i want to have the code that i modified locally and installed. How do i define at installation time where to store a copy of the source code of the package so i can copy it inside the docker later and perform the installation again at runtime ?

4

You should be able build a wheel for your client and server, transfer it into your docker, and then install that.

5

Maybe it would be helpful to decompose the task in smaller parts…

What do you mean by that?

It is possible to copy local code inside the Docker image. It is also possible to mount host volume into docker image at runtime.

Maybe what is blocking you is Docker knowledge, rather than Python knowledge. Could it be?

If we exclude Docket completely, are you able to achieve what you want on you host’s local file system? What does it look like?

6

The first thing you need is a way to get files into your container. Some options include:

  • docker cp / kubectl cp
  • Roundtrip through Github / internet
  • Have a server running inside the container that can receive a payload
    (Note you’ll also want to think through security considerations)

Once you have the files, it should be pretty easy to install your third party dependencies using pip or whatever and then restart your server.

You might also want to have your client and server do some sort of handshake that establishes they have compatible versions.

7

Thanks for the answers. Initially i wanted to install using pip+wheels, to let it manage the python path and the dependencies. But i ended up just copying the source code of the module with this function and copy with docker cp, and i use the code without installation.

def find_module_path(module):
    return Path(module.__file__).parent