I’m still new to learning Python. I have Python 3.9 on Windows 10. I have not learned yet how to make modules.
I’m making a custom module crutil.py which has many functions that I will use in many programs. My crutil.py will be used by many programs in many directories. So it should not go in the same directory as my main program.
Where is the proper place to put crutil.py?
Is it ok if I put just crutil.py in %PYTHONHOME%/lib?
I don’t need to install it via pip. It will not be released to the public.
Is it required that I make a new directory for my file under %PYTHONHOME%/lib? Like crutil/crutil.py. Because the docs mention I have to make an empty file __init__.py in that directory so the module is treated as a module.
This crutil.py module will eventually be used on the MS Azure ecosystem which appears to update Python automatically every so often. I don’t know how often.
Why are you using the very old python 3.9? Using 3.12, or 3.11, would be better.
As for where you put your library modules, I would not recommend using PYTHONHOME as that is controlled by the python installation.
You could create your own folder for your python modules and add it to PYTHONPATH. That way when you upgrade python you will not lose access to your modules.
The tutorial I’m doing is using 3.9. And I found some things the author did which did not work in 3.12. So I’m keeping it to 3.9 for the duration of the tutorial I’m doing.
My recommendation is to create a virtual environment (you can use the same one for each of the “many programs” that will use crutil.py), and use Pip to do an editable install of crutil.py into that virtual environment. This should be as simple as navigating to the folder where you put that file, and doing pip install -e . - assuming Pip is up to date. You may want to have a separate folder for your crutil.py, or at least make sure it doesn’t include other.py files that you don’t want to “install” this way
An “editable” install essentially means that Pip will write files into the site-packages of the virtual environment, but instead of actually copying in the code, it will set up metadata that the import system will use to “redirect” to the original folder instead. It’s “editable” because, due to this setup, changes that you make to the original crutil.py will therefore be automatically reflected the next time the code is imported (remember that import uses a cache, so you may have to restart Python or at least explicitly remove the sys.modules entry).
No, this suggestion makes no sense at all, for multiple reasons.
First off, the PYTHONHOME environment variable is usually not set at all. It’s only used if needed to override the default Python behaviour.
Second, despite what the documentation says, there would only be a lib subdirectory like that on Linux-based systems; but writing %PYTHONHOME% implies you’re on Windows.
Most importantly: the purpose of PYTHONHOME is to change where Python looks for the standard library. It’s emphatically not a way to tell Python about additional places where your code might be.
You might be thinking of PYTHONPATH instead. You can set this as you like to set additional sys.path entries on startup. It’s a hack, but it does save you from being forced to learn about packaging. You don’t need to make any subdirectories within that, and any subdirectories that are there don’t have special meaning.
You don’t need to use PyPI, or publish your project anywhere, to use Pip. It can install from local wheels, and it can make “editable” installs of a project folder.
No, Pip can install individual .py files as modules, as well as installing folders as packages. However, the tutorials out there are mostly oriented towards packages, and there are organizational advantages to that as well.
You’ve either found something very out of date (this includes cargo-cult advice on Stack Overflow that may reflect an outdated understanding even if it was written recently), or are misunderstanding it severely. Empty __init__.py files have not been required in order to create packages since Python 3.3. The presence of an __init__.py file (it does not need to be empty, and there are good reasons to put certain things in it) simply causes the package to be a regular package rather than a namespace package, which a) has implications how build tools automatically discover packages from your code base; b) allows for merging in other code located somewhere else into the same package; c) has some subtle implications for how the import machinery searches for imports.
Then you should definitely learn about packaging anyway. In fact, you should learn about packaging just on principle; it’s a fundamental skill.
Even if you don’t make it public, it can still be convenient to turn your script into an installable package. Doing so has many advantages:
Easy to access and use from other programs.
Integrates well in virtual environments.
You don’t need to worry about where the script is. Python’s packaging mechanisms take care of the details for you.
If you version control your script (you should), multiple different versions can coexist peacefully in different virtual environments.
A minimal package can be a directory containing just your script and a pyproject.toml file:
crutil/
├── crutil.py
└── pyproject.toml
pyproject.toml can be as simple as:
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "crutil"
description = "description goes here"
version = "0.1.0"
You can then easily install your new package wherever you need it by pointing pip at this directory. You can also easily build a wheel with python -m build, should you need one (requires build to be installed).
Thanks for the detailed explanation. Your writing style really helps me learn. My brain just “clicks” with your style.
I will give your instructions a try. As I start developing and practicing routines my crutil.py file will change a lot as I add functions to it.
One of the functions I want to add is an easier one-line way to do a regex find and return the found string. I will try some things myself, I don’t want to drift this thread.
(remember that import uses a cache, so you may have to restart Python or at least explicitly remove the sys.modules entry).
I don’t use the python shell much, I run my program from a Windows cmd window. Just to clarify, if my program ends, and I’m not in the python shell, that is the same as python ending. I.e. when I run my program again, the crutils.py cache is updated again with the latest version of crutils.py?
Regarding my tutorial and multiple python versions
I’m finding out that the video tutorial I’m doing was made as far back as 2018 as some questions say “6 years ago”. Since then it appears the author has updated only some chapters, and added others, and he is using several versions of python through the whole tutorial, depending on when that chapter was updated. Some versions he used were: 3.6, 3.9 (he said it was required to make PyQt6 work), and 3.11.
That should explain some of his outdated instructions.
He also starts out using Pycharm but goes on to use Visual Studio Code, which works better as he shows how to make web apps.
Each time python starts its sys.modules cache is empty.
As it executes code modules are imported and remembered in its sys.modules cache.
In full disclosure there is a cache of compiler modules on disk.
Python will load the already compiled code from disk if the compiled code is up to date. These are the files in the __pycache__ folder.
As new modules are loaded that have not been compiled, or the compiled version does mot match the source code, then python will save the compiled version to disk.
