I've messed up my python installs very badly

Hi, I’ve had long running issues dealing with the environment of Python (especially with multiple installs and virtual environments), but up until now I’ve always been able to find workarounds. However, it’s reached a tipping point where all of my installs are unusable in some way. I have asked in a couple other places, but I must have communicated poorly or caved to frustration, and so I wasn’t able to take away much from those. I thought that perhaps by asking here and exercising lots of self-control, I might be able to get a more proper insight about the situation from people who may be more dedicated.

I’m about to describe the situation in as good of detail as I can, however do note that it is a lot and as such it is likely I will inadvertently leave out important information. Please do ask me for this information if you notice, I will follow up with the information or another question asking for clarification.

First, let’s get the basics out of the way: I’m running Win10 on a PC with a couple of accounts (at setup of the OS I inadvertently created one account without realising, which may be causing some permission issues since the account I use is not that one (but it is an administrator)). I currently have Python versions 3.9.5 and 3.12.6 “installed”, but the installs have become tangled to the point where I can’t figure out what is going on.

The thing that pushed the issues over the edge was when I installed NodeJS a few weeks ago and selected to install the build tools, which included Chocolatey. The Chocolatey install (taking place within powershell) scanned for any installed versions of Python, and says it didn’t find any, which is the first sign of trouble. I had numerous working installs at the time (in C:/…
…/appdata/programs/python), so I don’t know what it’s going on about. Anyway, the install failed with “generic MSI error”, which helpfully explains to me that the environment is messed up (it was messed up quite badly before this point but at least things still worked), and that a probable cause is that Python is already installed (which is correct, but it’s hilarious because it claimed it was not installed). It then provided instructions to try and run the installer again and look at some log files (except the file path it points to does not exist), so I couldn’t progress on that front.

However, the effects of this failed install seem to have been severe. It installed half of a Python installation to C:\Python312, and the (previously working) 3.12.6 install was suddenly missing half of its critical files, including the Python exe itself.

I tried to repair the install, but it doesn’t repair anything, it still leaves behind two unusable Python installs (in case you were wondering, that’s how I ended up here. I decided to give the link in the installer a try after the “repair” option did nothing a few times).

So as a result, everything is on fire, including all the PyCharm virtual environments I had set up, and it has progressed so far that PyCharm seems to be making up installs out of thin air (it has some imaginary 3.11 and 3.13 and 3.10 interpreters in the venv settings), and one exe file is labelled as being somehow both 3.9, 3.10, and 3.12 at the same time in the venv options.

So it is clear to me that the most practical way forward is to somehow reset the Python installs on my system, although I am sure that this will break all the projects I’ve worked so hard on, as the places they expect files to be will be wrong once the reset happens. Additionally, the best reset options I’ve seen seem to be along the lines of “reinstall windows”, which is absolutely not an acceptable option for me.

Perhaps there is a way to perform some kind of “digital surgery” and find out where applications are looking for files, and where environment variables are pointing, and marry everything up; as well as cleaning the references to nonexistent installs and other such things that could cause problems in the future. It could also be possible that there is another way, but I am sure that people here would be the ones to know.

I’ve realised now that I haven’t actually mentioned the exact errors encountered, and I’ll try to do so here:

Upon running the install located in C:\python312:
it works, except with this warning:
Could not find platform independent libraries
As implied by this warning, libraries don’t work. I can’t run pip or import anything for that matter.

Upon trying to setup a new virtualenv in PyCharm:
“the SDK seems invalid” (among many other things, I recognise this may technically not be the best place for PyCharm but it would be a lifesaver if I finally got to the bottom of the nuance with virtual environments so that I don’t break everything every time I create one).

Upon running the previously working 3.12.6 install:
“No Python at C:.…” (expected, the exe is missing from that directory, along with loads of other things, and unfortunately I can’t list all those things since I don’t know every single file off the top of my head).

Another thing to note is that I’ve ran many machines with Python over the last few years, and this sort of problem always pops up every time something wants to install a version of python, or I try to upgrade or move projects or files between PCs, etc. etc. It always leads to a very bad system environment configuration, and the errors and quirks are never ending.

I’m willing to learn a lot and read docs or take courses to build intuition on the topics that are relevant to these issues, because I’d love to be able to one day resolve these issues myself, or better yet, have a proper intuition so that they don’t happen in the first place.

If you made it this far, I sincerely thank you, whether you can help or not. I’m giving it my all this time, and as I’m sure you can imagine, it takes a lot of effort to not cave into the frustration and take shortcuts with the description of the problem, or go on a rant.

I’m anticipating the resolution and troubleshooting process will be very long, but I am committed to getting to the bottom of this, so I will assist with any additional information that may help you guys to help me. Again, thank you so much for reading, and if you can help in any way, that would be much appreciated.

2 Likes

I would suggest you uninstall all the python versions.
I notice that at lease some of the installs are user install not system installs. This may be important.

You might also need to uninstall Chocolatey and nodejs.
You may need to do some clean up if uninstalling leaves python stufff behind.

Now install the python versions you require for ALL users.
Does that fix your issues?
For example py -0 should list all your ibstsll pythin versions.

1 Like

Sorry to hear about node via Chocolatey. It’s an extra step or two, but I’ve had no problems at all after using winget to install nvm to install node.

To clarify, It wasn’t via Chocolatey that the problems happened. The problems happened after installing NodeJS and selecting to install optional build tools, which failed at the Chocolatey install (i.e. actually installing Chocolatey, not merely using it to install NodeJS) because it didn’t detect my multiple python installations and tried to install Python while it was installed already.

Here was the output of py -0:

-V:3.12 * Python 3.12 (64-bit)
-V:3.9 Python 3.9 (64-bit)

(Sorry, I’m unsure how to make code blocks go over multiple lines, so I just did two code blocks here).

You use the </> button after selecting both lines.
Or you type it like this:

```
LIne 1 here
  and line 2 here
```

Ok. You now have both versions of python installed.
One step at a time add on the extra software you need and test that python is still working at each step.

Ah, I didn’t actually do anything to my installs yet. I have made no changes, not uninstalled anything yet, etc. This is because I am fairly sure that every single script I’ve ever made will stop functioning. Is there some kind of “surgery” that can be performed to swap out the python installs for good installs whilst keeping this aligned with what everything is expecting to be located at certain places, etc?

Can you say a bit more about why you think that uninstalling and reinstalling will break your scripts? I mean, it surely will cause them to stop working temporarily, but ideally you would just change some config options to point to the new Python, and they would begin working again. One possibility is to note the directories where Python is installed now, and then try to reinstall to the same paths, which may mitigate disruption somewhat.

You mentioned PyCharm. Are you running your scripts exclusively from within PyCharm, or are you running them from the command line, or a mix, or what?

I run from different places depending on my needs (IDLE/command line for single scripts, PyCharm for complicated projects). I must admit, I clearly don’t quite understand the virtual environments provided by PyCharm, I’ve had it explained to me on numerous occasions, but every time I make one, something unexpected happens.

As for why I think it will break everything, well it’s just a matter of experience. For example, I moved a project from one PC to another (within a PyCharm venv), and thought that if I just pointed to the correct Python, it might work. Unfortunately, no. It was total chaos, and that’s how my PyCharm reckons I have 3.10, 3.11, and 3.13 installed in the same executable file (I don’t have any copies of these installed anywhere), and that all the other python installs are now invalid in some way.

I think one thing to keep in mind is that you should not consider “re-install it all from scratch” to be a failure :frowning:

It’s simply what you need to do once in a while – the fact is that keeping a python installation or environment up to date by adding, removing, updating, etc. does sometimes end up with a mess, and tossing it and starting all over is OK.

The trick is to make sure you have a easy way to re-create the environment(s) you need: if you have requirements file for each environment you need, then it’s quick and easy to just start all over.

NOTE: you seem to have had some trouble with various systems’ (e.g. chocolaty) idea of the “system install” of Python.

If you want to have various versions of Python installed, and you want various different environments for different projects, you might want to consider conda – it manages separate environments for you, and python itself is also managed by conda, so you won’t get tangled up with other systems that want to manage Python.

I recommend miniforge as a way to get going:

Or you may want to give pixi a try:

It’s pretty new, but providing to be quite robust.

conda is supported by PyCharm (Configure a conda virtual environment | PyCharm Documentation) – I’m not sure how pixi and PyCharm get along.

2 Likes

Hmm. It’s always so daunting with these new things though. Of course I’ve said I am willing to learn, and I am, but it is a bit challenging given that everywhere I’ve asked has recommended loads of different third party systems to me, and I almost end up frozen with too many choices. In my opinion, more third party things like that just make the system more likely to break or have version discrepancies which is what I’m trying to avoid, so I have to be absolutely sure before pursuing something like that. I feel like I’d have to already be an expert in the packages recommended in order to use them without breaking everything all over again.

For example, I actually did try to install Conda for a uni project (it was recommended), but problems started up immediately as it battled with my pre-existing installs of Python on my laptop.

Additionally, I don’t consider reinstalling Python to be a failure, not at all. The part that’s a “failure” is that I have hundreds of scripts in my machine, and from experience I’m expecting every single one to break upon the reinstall (I actually may need instructions for how to perform the reinstall as well so that I don’t screw things up even worse than they are already). That is such a soul-crushing prospect that I’m genuinely scared of doing anything.

Perhaps a way to learn and decide for myself if I need third party tools like that and also how to decide which one to go for could be the way forward? I don’t know. I just don’t want to take big risks, but to be fair my system is messed up as it is so there’s little sense in keeping it as it is and doing nothing.

Is there any tips you guys can provide so that I can make informed decisions and get started and make progress?

Edit: I don’t know if it’s been picked up on, but it’s quite concerning that the python repair tool says “repair successful” to me with a straight face despite the fact that it did nothing, it just left two half-installs on my machine and somehow that counts as “repaired”. Is there a way to investigate how this is happening and why it thinks that two unusable instances of Python are, in fact, usable?

It only repairs the python install, it will not for example fix PyPI downloads.

I feel your pain. A few thoughts:

When I teach Python to newbies, I stick with the basics:

Install just one Python (the one you get from python.org).

Use that one for everything. pip install what you need.

Some folks suggest that you start out with virtual environments from the get-go, but I find that that creates a level of confusion right up front, that isn’t necessary.

If you are developing more than one complex application (like I am), then yes, virtual environments are a critical tool, but if you are writing a bunch of scripts, then maybe not so much.

I wrote:

note the " various versions of Python" and “various different environments for different projects”

Chances are you don’t need that :slight_smile:

That absolutely should not happen – it is highly unlikely that those scripts are all particularly dependent on a particular environment – you can probably run them all in the same environment (or could with a bit of tweaking, if some are old.).

If you are using PyCharm to run your scripts, then that config may break. Which leads me to my other point when teaching newbies:

I don’t start with a full-featured IDE like PyCharm — because it hides what it’s doing under the hood, so you don’t really know what’s going on. If you start out running your scripts at a command line, you have a much better idea what’s actually happening.

But in any case, PyCharm is a great tool, but you do want to spend some time understanding what it’s doing, and how.

My advice:

  1. uninstall / remove all the Python installs on your computer
  2. install Python 3.12 from python.org.

Fire up PyCharm, and see what happens – you will probably need to re-set what Python it’s pointing to when you click the “run” button, but that should be about it.

“pip install” the packages you need, as you need them.

I don’t know what kind of work you are doing, but the reason for virtual environments is when you have multiple applications that have conflicting requirements: e.g appA will only work with numpy < 2, and appB needs numpy > 2. [1]

But you used the term scripts – so I’m guessing you can keep your system pretty up to date with the latest packages, and simply tweak the older scripts when they go wrong – it probably won’t happen often.

If you do need/want to use “PyCharm environments” – take some time to understand how they work under the hood.

Finally: keep a “requirements.txt” file for your primary environment – when you need a new package, add it to the file, and then re-install everything from that file. That way, when it all goes to heck again (it might) then you can just re-install that file, and you’re good to go.

[1] I use the example for a reason – numpy recently released 2.*, and removed a lot of deprecated features, so there is some pain in the transition…

Well, the problem is it didn’t repair the python install at all. It only thinks it did.

Edit: Maybe I will be more clear. There’s the chocolatey half install, and the regular half install. It looks like the repair tool combs through the machine and finds that all the parts are technically present, but it doesn’t seem to know that they are not usable due to whatever weirdness goes on with them being spread out over two half installs. It definitely tries to do something, since it does spend a few seconds “repairing”, but in the end it doesn’t actually modify anything. I’ve also removed whatever the windows Uninstall tool removes, which left behind a little bit of one half install and most of the second half install. In either case, I don’t think I can action this any further other than say it did nothing, since I’ve removed a fair bit from my machine now.

Hi, I’ve started working on fixing some things. I used the Windows uninstall/remove tool to try and remove my installs. It left a lot behind, that I’m not sure I can cleanse, but anyway I decided to press on and download 3.13. I can get scripts working, but as you mentioned the PyCharm is very very confused.

My problem is, people seem to be suggesting that it should be easy to fix the confusion about PyCharm, but I’ve found it extremely difficult and am afraid to proceed without help. This stems a bit from there being numerous dialogs in PyCharm that look on the surface that they may be doing the same thing but they aren’t and its this nuance that I don’t get. For example, there’s “configure new interpreter”, “interpreter settings”, “add local interpreter” etc. I also want to clear up that I wouldn’t call myself a newbie as I’ve been using Python for probably 5-6 years now, but I’ve never understood environments because every time I breathe incorrectly when I’m working with them, the system goes absolutely nuts, and I haven’t found a good explanation yet (although I’m not good at looking for them), so I’m really just guessing when it comes to that.

To start, here are some specific questions (first about some definitions you used, just so that I know what you are talking about):

  1. When you say “primary environment”, are you referring to the environment used when I just run a python script from IDLE for example? I’m not sure how it works between users but I selected “install for all users” this time. Is this equivalent to whatever is specified in my PATH variable at a system level?
  2. What does an “other environment” actually refer to? Is it a folder, another Python exe, or something else? Is there a correlation between environments and the number of Python exe files, or can multiple environments use the same executable?

And here are some questions for what to do next:

  1. What do I do with my existing projects in PyCharm that used a virtual environment (whether they were configured properly to begin with is another thing entirely, odds are they aren’t). Edit: here’s a pic summing up the situation I face at the moment. https://imgur.com/cwnzHbq
  2. How do I properly configure these environments? I am really scared to create new ones due to not fully understanding the nuance with them. Previous help I’ve received seems to really gloss over many things, if it’s too much to explain can I have a pointer to relevant course/documentation that should explain it for me?
  3. You want me to take some time to understand how these environments work. Well, it’s one thing to say that, but another thing to tell me what I need to do to achieve this. What do I need to do to achieve a better understanding?
  4. You mentioned some kind of requirements.txt, that sounds like it could help, how do I maintain this and use this?

I’m also sure that some of these could be resolved by looking online, but I’ve recently grown extremely unconfident in my ability to find things and properly understand them when looking online, since it’s so hard to make sense of all that’s out there, and the conflicting answers that you get in different forums and places. I also think I’ve just built up an understanding that’s maybe got one or two fundamental things wrong, not by much, but enough to need a total reboot of how I understand it.

It is for that reason that I ask here, but do note that I’m also doing this quite cautiously because I’ve had extremely negative experiences doing so in other places in the past, so I really appreciate you guys for bearing with me so far!

1 Like

probably – but what we mean is what you get when you run python from the command line.

more specifically – when you install python, you get, well, python, and its stanard library, and when you do python -m pip install, stuff gets installed there.

in addition to that, there are various systems that create “environments”, or “vitual environments” – these create and isolated python system that you can pip install stuff into without chaning other things – so you can have e.g. two environments with different version of third party pacakge.

For you – I think your biggest issue is PyCharm – I’d go to their docs, and their must be a community of users somewhere. Sorry I can’t help you there.

  1. MAke sure the Python you jsut installed runs from the command line:

python

should get you a command prompt.

  1. Go back to PyCharm, and try to run a file – if it doesn’t work, try to set th iterpereter, and avoid anything with the word “environment” in it. At lsat for now – get it basically working, and then maybe you’ll want to take on environments…

Good luck.

1 Like

Well, my thing is that many of my projects have environments already. So I am unsure what to do there.
Edit: and yes, python does run fine from command line.

Oh this should be easy to fix. It probably won’t actually be easy, but that error just needs the python interpreter configuring in PyCharm.

Delete them, make new ones :slight_smile:

OK so, some things to understand:

  1. PyCharm will use a specific python interpreter for each project, this is configurable within PyCharm, the interpreter is chosen by providing a file path that points to the python interpreter (a python.exe file).
  2. A virtual environment will contain a python interpreter inside its Scripts folder.
  3. You can point PyCharm at the python interpreter inside a virtual environment.
  4. The python interpreter inside a virtual environment will be the version of python that was used to create it. They will not be changed in any way by uninstalling or reinstalling or upgrading other python installs (that’s partly why they are useful!).
  5. When installing dependencies using pip, use the pip from within a virtual environment in order to add the dependencies to that virtual environment. The virtual environment will not “see” (be able to import) dependencies installed using other pips.

As for how to set the python interpreter in PyCharm, read the manual (I don’t use PyCharm myself) :slight_smile:
It looks like here is a good place to start reading.

If any venv doesn’t work, write down what dependencies you installed into it, delete it, create a new one, and install the dependencies again (into the new venv). The python venv documentation makes it clear this is expected usage (without explaining why):

  • Considered as disposable – it should be simple to delete and recreate it from scratch. You don’t place any project code in the environment.
  • Not considered as movable or copyable – you just recreate the same environment in the target location.

And good luck! The hardest part of using python is managing the projects and dependencies, if you can do this, you’ll be grand.

1 Like

You can switch to using portable python builds

Also, importable modules can be controlled not through venv but through sys.path.
But this is another much more difficult question.

Not sure that is a thing on Windows.