Python Module madness

I have been coding since the 1970s, but I am new to Python. I found the Python tutorial too fundamental, so I skipped it and began coding with peeks at examples. I found the coding very easy but the built-in capabilities of the “as installed” system seem to be aimed at just fundamentals. Very soon, I found myself spending 90% of my time dealing with modules, sorting out “ifs” and “buts” and chains of modules that needed other modules, trying to use them, etc, to get capabilities I would have thought fundamental in the “as installed” version or at least available with a maths module and a graphics module. Dealing with Python modules needs a tutorial. (If I have missed a tutorial, please point me at it.)

It’s hard to say anything concrete without understanding what issues you’ve encountered. What exactly do you mean by “dealing with” modules? What tasks are you hoping to complete with them? What happens that you aren’t expecting?

“The” tutorial (I assume you mean the one in the documentation) is quite comprehensive and has an entire chapter on modules, so you could do worse than to swallow your pride and start back there.

2 Likes

It’s ok to have rushed ahead: Python is intuitive (up to a point) and fun to learn. It is perhaps time to circle back on the tutorial or the documentation.

The standard library that comes with an implementation of Python is quite rich but never enough. Installing more modules in the place Python will find them naturally used to be difficult. There have been several rounds of improvement: if it is continually difficult you have probably found the wrong way (or an old one).

pip makes getting most modules you need effortless. venv (or the virtualenv command) creates a sort of project-local installation of Python, which means what you install there does not affect other uses of Python. I remember thinking once these were advanced tools I’d learn later but they make things so much easier I wouldn’t start a project now any other way.

The main pitfall people experience, especially if they have several versions of Python installed (or have tried several times to re-install it), is that they find themselves using the pip that belongs to one installation and the python that belongs to another, and wonder why the module they just installed still isn’t found by their code. For this reason the recommended way to use them is:

python -m venv ...
# here you "activate" your virtual environment (platform dependent)
python -m pip ...

Then you know you are installing for the Python you use to run code. Both take a --help option, and then there are tutorials that will guide you.

There are a lot of options but I’ve never had to use 90% of them. A full understanding of the import subsystem is difficult to gain. Most people get by with a working knowledge of how to use it in common cases.

For what it’s worth, the tutorial doesn’t actually cover Pip. It’s about organizing one’s own code into modules and packages (for local use, not distribution) and the standard library. I wanted to check first whether OP’s issue is about these fundamentals, or about setting up third-party code. Your post preemptively covers the most important points for the latter, so thank you.

True. Tutorials that do are easily found, though … Install packages in a virtual environment using pip and venv - Python Packaging User Guide

Doing it via python -m seemed worth mentioning: when I first learned, it was normal to use the command-line tools. On these forums I have learned (and I know you’re aware) what a mess one can get into.

I was trying to do some simple image work, i.e., load images and work on pixels using the mathematical algorithms I developed. With other languages, this simply involved a graphics library and a maths library. With Python, I was confronted with a long list of modules that I felt were inconsistent in integration and operation.
I searched through the tutorials and guides, perhaps not exhaustively, to discover how to find specific “needles in the module haystacks” and get them working.
It took a whole day, but I eventually got some pathetic GWBasic-level graphics coded. Then, confronted with more advanced needs and more module searching, I gave up. I told my son, who had been taught Python formally but superficially, “Python is not easy for advanced programming. Everything worthwhile involves adding an endless list of modules and finding badly referenced syntax.”
That was a few months ago. I have only been drawn back here by a desperate need to develop some 3D printing code to integrate with other Python code. I am dreading the prospect, hence the message here.

I didn’t abandon the tutorials; I tried to find solutions to my problems through them. I found numerous mentions of “pip” but no references to its use. In the end, I just guessed my way through it. “venv” doesn’t even ring any bells right now.
I’m sorry, but that bit of code you show needs a lot more explanation. Can you point me to references to save a search?
I have had Python on systems in the past, but this computer’s installation was a clean first time.
At the time, I was writing code from scratch, but I hope to integrate third-party code in later projects. (That reminds me: are there Python modules that handle STL?)

Just a note that some of us don’t “activate” our venvs. It’s a
convenience that makes your current shell use the venv first (and
therefore get the venv Python).

But you don’t need to. Just running the “python” executable from inside
the venv causes that Python to use the venv to find modules. Eg:

 /path/to/venv/bin/python -m pip ...

will install to the venv reliably.

If you’re (the OP) new to Python, the “activate” thing is very
convenient though.

I looked in the documentation tutorial and I vaguely recall a mention of modules. I guess that is how I got into using them. The main problem was finding what I wanted, which modules to use, and how to use them. Moreover, the modules I loaded required a lot of adding more modules, and then there were “Tweedle Dee / Tweedle Dum” choices. I would add some modules, but that module didn’t work for some reason, so I had to unadd it.
I expected to simply load a well-documented graphics module and a well-documented maths module. If they existed I never found them.

I was trying to do some simple image work, i.e., load images and work
on pixels using the mathematical algorithms I developed. With other
languages, this simply involved a graphics library and a maths library.
With Python, I was confronted with a long list of modules that I felt
were inconsistent in integration and operation.

A very short list of useful third party modules in this space:

Pillow: a library for doing simply things with images.

Numpy: for doing maths with lots of numbers (eg series and DataFrames,
which are essentially tables of numbers like you’d see in a
spreadsheet).

It took a whole day, but I eventually got some pathetic GWBasic-level
graphics coded. Then, confronted with more advanced needs and more
module searching, I gave up.

Maybe you could describe what’s you’re trying to do more specificly.
“maths” is a very broad topic, and so is “graphics”.

I told my son, who had been taught Python formally but superficially,
"Python is not easy for advanced programming.

I think you’ll find people here who would disagree on this.

Everything worthwhile involves adding an endless list of modules

Many things require using some third party modules. As does almost every
modern system.

and finding badly referenced syntax."

3.12.3 Documentation is pretty good for the language spec and stdlib
modules. I use it (or my locally downloaded copy) all the time.

We can probably offer less vague advice if you have more specific
questions.

Well, there IS a well-documented math module. Is that not the one you wanted? Were you looking for something different, and if so, WHAT were you looking for? Part of the point of a module system is that you are NOT locked to just a single graphics module and a single maths module.

I’ve used GW-BASIC. It is horrifically restrictive, precisely because there is no module system. The only way to do graphics is the way the language provides. The only way to do mathematical calculations i the way the language provides. Do you like that sort of restriction? Then you can have that in Python too - just ignore PyPI and pip and venvs, and stick to the standard library. This is already far far richer than GW-BASIC ever provided, and you don’t need to go elsewhere.

But when what you’re looking for isn’t there, what do you do? Do you go “okay then, I guess I can’t do that”, or do you look for something additional? It’s not part of the standard library, so you need something third party. Guess what? There are multiple modules available, for almost anything you could look for. When you want graphics, do you want image manipulation, or a GUI? When you want maths, are you lookg for symbolic evaluation, matrix manipulation, or arbitrary precision integers? Oh wait, that last one is actually part of the core language… which GW-BASIC not only didn’t have as core, it wouldn’t even let you import from elsewhere.

Flailing around is seldom going to get what you want, but also, complaining because there’s more options isn’t exactly productive. You are ALWAYS welcome to throw away those options and stick to the stdlib, and you’re still going to be better off than a GW-BASIC programmer. By accepting PyPI, you inherently and by definition accept that there are now choices, and you have to make them.

With Python, you choose one package for reading and writing image files, and almost everything worth looking at will use NumPy as a backend for representing the image in memory (and it will be automatically installed as a dependency, if you don’t have it already). You’ll get a NumPy array (a type implemented by that package, which relies on C bindings for performance (the data is represented as a contiguous C array of native types, rather than as a possibly-nested list of Python’s immutable, arbitrary-size integer objects). Many of these libraries also handle video formats “for free”; you just get an array with one extra dimension (representing time). Sometimes they try to offer “user-friendly” ways to manipulate images for common tasks; but if you really want to do number crunching on pixel values, you should do that at the NumPy level.

I’ve used and can recommend imageio in this category. It handles a lot of image and video formats (including transparently treating animated GIFs as “video”). The “standard” is PIL (called pillow on PyPI); that one’s more in the “user-friendly” category and makes the conversion to a NumPy array less obvious (but still simple).

NumPy has a ton of functionality in it, so learning to use it well can be daunting. However, the documentation is quite comprehensive. Depending on your needs, SciPy builds on top of NumPy to support more advanced math. I don’t know what you’re looking for in a “math module” exactly, but there’s already one in the standard library. But if your manipulating images, it’s not very interesting to be able to e.g. compute the factorial of 300,000 (takes about a second on my 10-year-old hardware); you presumably are more interested in looping to apply a calculation to a multi-dimensional grid. That’s NumPy’s forte.

1 Like

Thanks, Cameron. It has been months since I abandoned Python, and I am going to try to get back into it as soon as need overcomes repulsion. So, what you have put here makes no sense to me right now. I have no idea how to even apply what you provided without going back to the ruin I left behind.
“venvs” will require a search, but I guess that is Pythonese jargon for a shell environment. I may or may not have activated it. Nevertheless, I don’t think I had that sort of problem.

BTW, I am very grateful for the kind responses to my plaintive call for help. You guys are very helpful. Thank you.

A “venv” is a virtual environment generally created by the standard library venv module (or by some other tool that wraps it). You would almost certainly know if you used or activated one (unless you’re using certain IDEs that manage one for you and don’t communicate this fact very well). It’s “virtual” in the sense that it’s not doing any sandboxing or anything like that; it’s held together with symlinks (or copies of certain files, depending) and glue, I mean, environment variables.

Virtual environments are used for setting up a separate place to install packages, so that they don’t interfere with packages for other projects (it’s also, IMO, the easiest way to make sure you have full access rights for the directory where the packages are installed). “Activating” a virtual environment means running a script that is automatically created as part of the environment; this script sets some environment variables so that e.g. the python command will use that environment, and so that you can see a visual indicator in your terminal prompt that the environment is active. Check out this guide:

1 Like

I got into both Pillow and definitely Numpy. I think it was Numpy that gave me problems, but I guess after time past, I will have to return to the scene of the crime to get more specific. I vaguely remember having to load more modules associated with Numpy. I also had problems with finding syntax.
I can’t get too specific about the graphics and maths because of confidentiality requirements. I can say they are available in C++, Delphi, Fortran std graphics and math libraries, and embarrassingly native in QB64 without even adding a library!
I really hope to regret and rescind what I said to my son.
Thanks for the Doc.Link I will get into that ASAP.

Sorry Chris, I only mentioned GWBasic to describe the sort of graphics level I got to before I gave up. I am sure Python can do better. I did not mean to upset you or anyone. I guess I could have been more diplomatic but I am under pressure and stressed.
I think I have answered your other questions elsewhere.
Thanks for the maths module link I will check that.
As for adding modules and libraries vs. built-in functions, I have programmed in everything from machine code loaded via switches on a computer I built myself, to languages so new they haven’t got names yet. I have reached the point where I just shrug my shoulders and accept philosophical issues like loading libraries.
I am at a point where I am expected to and usually do pick up programming languages quickly, and it galls me just to have to ask for help to work in something as established as Python.
I will try to be more considerate here in future.

Apology accepted :slight_smile: It’s worth remembering, though, that we are ALL dumb the moment we step outside of our areas where we’re smart. It is kinda galling to think that you’re out of your depth, and I definitely get that, but we all face this and we all need to know when it’s okay to read through the tutorial.

And yes, I have been in that situation. I thought I could do this based on my pre-existing knowledge, and I couldn’t. And then regretted not starting with the tutorial in the first place. But it’s never too late to go back, read the tutorial, and slot all the information into your brain as “yes, that’s exactly what I expect” or “that’s spelled differently but exactly what I know” or “oh, this is one place where Python is different”. Notably, this sort of expression means something very different in Python to, say, C:

if 5 < x < 10:

And the first time you see it, it’s fine to go “wait, what? That doesn’t make sense!”. That’s why we have lots and lots of great documentation :slight_smile:

1 Like

Wow thanks, Karl, there is a lot of real help there, my courage is coming back. :blush:
I do recall some of those modules you mentioned, and I will get back into it.

Thanks again Karl. It sort of rings a bell now.