Run python without installing python environment by converting python interpreter and third-party packages into dynamic-link .so library

Normally in a linux server, you could run a python script only after you install the necessary python environment, and the python executation is highly dependent on the versions of any third-party packages, which brings a lot of troubles if you have to deploy a python-based project on any linux server. Here I fixed and removed the python environment dependencies of deploying a python-based project, so that you can run python codes on any linux server, even without python software installed.

This tech plan involves two linux servers (one with a python3 environment for preparation called the preparation server , the other without any python environment for deployment, called the deployment server ).

The basic idea of our tech plan: Use gcc and Cython to convert all .py files to executable files or .so dynamic libraries, among which start-engine script is transformed to executable file and all the middle imported files are converted to .so dynamic libraries. Then copy all already structured python head files folder and dynamic link libraries folder, and combine all above into a brand new large folder. Finally upload this final folder onto the deployment server for deployment. You could directly run the executable file to start the engine and you don’t have to do any compile mission on the deployment server .

The specific operations could be classfied as preparation phase and deployment phase. Let’s describe them separately:

The Preparation Phase (at preparation server ) consists of 2 steps as below:

Step 1 : Prepare all the head files and dynamic libraries. The procedures are as below (you only have to operate this once ever, and you could use the folder already packaged on any deployment mission later on):
a. We have to prepare all dynamic libraries for python original interpreter and third-party packages in advance. We need to prepare them well on preparation server . The conda version of python environment could bring some convenience to this job, since it already contains lots of third-party packages the original python doesn’t have. For those packages conda doesn’t include, you could always use command python3 -m pip install packages or download the wheel file to install them. Currenly I already installed tensorflow, pytorch, keras, pyspark, sklearn, kafka, pymysql, cx_Oracle, tornado, requests, dask, elasticsearch, simply because I’m working on deep learning and big data computation. If there is any packaged that is not frequently used but used in your project, you can download them as well.
b. Please keep in mind Cython is mandotary because you have to convert python script into .c file, and you could embed the python interpreter into the generated .c file with the --embed argument.
c. All successfully installed third-party packaged are automatically saved in /root/anaconda3/lib/python3.6/site-packages. As long as you make a link to libpython3.6.so in /root/anaconda3/lib, you can fulfil the calling for third-party packages in site-packages while running the executable file.
d. Keep well the /root/anaconda3/lib (dynamic libraried folder) and /root/anaconda3/include/python3.6m (head files foler). You could directly use them in step 2.

Step 2 : Wrap all python scripts into .c files or .so dynamic files. Operate everytime when you have a deployment mission.
Here we have 2 different situations (single-script situation, namely only one python script for your project and multiple-scripts situation, namely multiple python scripts for your projejct).

SIngle-script situation procedures:
a. Copy the /root/anaconda3/lib dynamic libraries folder and /root/anaconda3/include/python3.6m head files folder generated from Step 1 into the folder where single-script is:
Command: cp /root/anaconda3/lib /root/anaconda3/include/python3.6m .
b. Wrap the single-script into .c file:
Command: cython test.py -3 --embed
c. Move the final dynamic libraries file libpython3.6m.so.1.0 in ./lib to /usr/lib directory:
Command: mv ./lib/libpython3.6m.so.1.0 /usr/lib
d. Run command ldconfig
e. Compile and link .c file into the executable file:
Command: gcc -I ./python3.6m -L ./lib -lpython3.6m test.c -o test.out
f. You don’t have to do any operation for all the rest files

Multiple-scripts situation procedures:
a. Copy the /root/anaconda3/lib dynamic libraries folder and /root/anaconda3/include/python3.6m head files folder generated from Step 1 into the folder where multiple-scripts are:
Command: cp /root/anaconda3/lib /root/anaconda3/include/python3.6m .
b. Wrap the start_engine script into .c file:
Command: cython start_engine.py -3 --embed
c. Wrap all the middle-imported scripts into the .so dynamic libraries. Note that the name of the dynamic libraries have to be exactly the same as the original middle-imported scripts except from the suffix. Operate the following commands for each middle-imported scripts:
Command: cython xxxxxx.py -3 --embed
Command: gcc xxxxxx.c -I ./python3.6m -fPIC -shared -o xxxxxx.so
d. Move the final dynamic libraries file libpython3.6m.so.1.0 in ./lib to /usr/lib directory:
Command: mv ./lib/libpython3.6m.so.1.0 /usr/lib
e. Run command ldconfig
f. Comlile and link the .c file generated from start_engine script into the executable file:
Command: gcc -I ./python3.6m -L ./lib -lpython3.6m test.c -o test.out
g. You don’t have to do any operation for all the rest files

After the Preparation Phase is over, you should have a folder containing lib (dynamic libraries foler), python3.6m (head files folder), executable file converted from start_engine file, .so dynamic libraries converted from middle-imported files (only in multiple-scripts situation).

The Deployment Phase (at deployment server ):

  1. Upload the folder generated from the Preparation Phase into the deployment server
  2. Delete all the python scripts lefted
  3. Modify the configure file based on your need
  4. Move the final dynamic libraries file libpython3.6m.so.1.0 in ./lib to /usr/lib directory:
    Command: mv ./lib/libpython3.6m.so.1.0 /usr/lib
  5. Run command ldconfig
  6. Run executable file ./test,out, DONE!!!
    image2021-9-1_18-30-221

Anyone interested in this topic?

1 Like