Method to refresh os.environ

You don’t need refresh() to implement what you described. Just iterate on the “system” environment and override os.environ[key]=value, no?

Moreover, it’s outside the scope of the proposed refresh() method which only updates os.environ with changes made in the same process outside Python.

What I don’t get is: what if the process set the PATH environment variable? You will restore the default value from CreateEnvironmentBlock()?

Yes, or if you want to replace the environment do a clear() and update(). One could do the same with C environ, as I did via ctypes in my first version of create_environ(), except there’s no documented and supported way to access C environ directly via FFI, in contrast to WinAPI GetEnvironmentStringsW() and CreateEnvironmentBlock().

I never proposed using CreateEnvironmentBlock() in os.environ.refresh(), though I did suggest it would be possible to use GetEnvironmentStringsW() on Windows, synchronizing it with C _wenviron in the builtin function. However, I dismissed that suggestion on your issue, since it’s out of scope for CPython.

I brought up CreateEnvironmentBlock() here only to show an implementation on Windows of a create_environ() function that provides what the linked question seemed to require. Though I have no clue how to implement it on POSIX, in which some initial environment variables are added by a platform-dependent mechanism (e.g. PAM) and others get added by a set of shell scripts.

In relation to this, I did suggest that maybe refresh() could take a mapping to use that overrides the use of C environ. But it’s not hard to argue against that idea as being more of a replace() than a refresh().

I’m guessing this question is because you thought I was suggesting to use it in refresh(). More likely, one would selectively update a required environment variable from the latest value stored in the registry, not replace the entire environment. The more common use of CreateEnvironmentBlock() is to create a new environment for a child process.

I added a new os.environ.refresh() method to Python 3.14: commit.

3 Likes

I want to elaborate on why I think this isn’t a good idea, or at least a good name.

From my experience with beginners, it is not commonly understood, especially for new programmers who aren’t very tech-savvy otherwise, how environment variables work under the hood.

I think the following will be an extremely common scenario:

  1. A beginner will want to configure their application, and choose an environment variable.
  2. After a few weeks, they will decide they need to have this configuration value dynamic (change during program execution)
  3. Naturally, not understanding that environment variables are specific to the state the process was opened in, they will modify the environment variable. *This will be especially common for environment variables modified using Windows’ UI for this, which modifies things globally.
  4. After expecting this to work, and it won’t, they’ll look at os.environ and see the refresh method, and add a way to trigger that in their application, perhaps periodically.
  5. This will also not work.

I understand that this is documented, but:

  1. I’m afraid users won’t find this out from the documentation, but from writing os.environ." in their IDE and seeing refresh()`.
  2. Given the general lack of knowledge, at least from my personal experience, this needs to be A LOT more obvious in the documentation.
  3. I think 99%+ of the users looking for this feature are going to be looking for the out of process feature, as we’ve seen the in process update is relevant only in very niche situations.

As evidence of this lack of knowledge, I’ve provided earlier here the top google results to “refreshing environment variables”. These threads support the notion that this is what beginners think refreshing environment variables means (here are the links again): linux - How to reload environment variables in Python? - Stack Overflow and python - Force global variable to refresh - Stack Overflow

I want to reiterate this will be a much more common error on Windows, due to the global UI to update environment variables, different from how it is usually done on Linux.

Given all of this, I think this feature shouldn’t be exposed in such a public and obvious way. I think, at best, the name should explicitly mention this being from the same process. At worst, simply using a word other that is not refresh will also help a bit.

4 Likes

Note that updating the persistent value of an environment variable in the Windows registry does nothing directly to update environment variables globally in existing processes. The changes will be reflected in new environments created by CreateEnvironmentBlock(), but that’s not commonly used. Usually a process inherits the parent’s environment.

That said, the environment-variable editor does broadcast a WM_SETTINGCHANGE message with the parameter “Environment” to all top-level windows in the current session. To my knowledge, the only system application that listens for this message is the Explorer desktop shell. In response, Explorer reloads its environment from the registry, such that subsequent processes that it spawns receive updated environment variables. The CMD shell, on the other hand, doesn’t own a top-level window to get the “Environment” message, nor does it implement Explorer’s functionality to reload its environment. Processes spawned by CMD inherit its current environment.

I am aware of that. Like I said:

But I think that behavior is not known by everyone.

Generally, without formally learning environment varibles, for a new user clicking “OK” in the environment variable editor* and then running os.environ.refresh(), I would say it’s fair to expect the new value will appear. The problem is that it will not appear.

*not the registry editor, even though internally that’s what happens

I don’t follow how that’s a reasonable expectation, unless every time that they use the environment-variable editor, they’re in the habit of closing existing console sessions and starting a new console session with a new shell instance, or restarting their IDE. Otherwise they’ll learn straight away that updating the value in the environment-variable editor has no apparent effect. But really I can’t imagine not already knowing that before learning to program as an independent student. I think there needs to be a prerequisite of basic knowledge about processors, memory, disks, processes, threads, virtual memory, and filesystems – a high-level summary of computing 101, which can be learned in a day or two. Unless one is learning in a controlled environment that avoids the need to know anything about computers and operating systems, but then all of this would be carefully handled by the curriculum and one’s teacher(s).

I’m not saying most people will expect the environment variable to magically appear after hitting “OK”. I’m saying they’ll expect it to appear after a refresh function. That’s what I would assume if I didn’t know any better - that it’s cached when the process starts but refreshing it would retrieve the current value from the OS.

The problem is not just knowledge about environment variables, but knowledge on programmatic ways to refresh them. There is no reason for someone to assume refresh will only work for things inside the process but out of the interpreter. That’s incredibly specific.

Edit: As mentioned earlier, chocolatey uses refreshenv to mean fetch out-of-process updates.

If my IDE had a “refresh environment variables” button in it’s config, I would expect it to make an out-of-process update fetch. (Maybe using the CreateEnvironmentBlock call you mention earlier, or, if that works?)

Can you suggest a better documentation for the refresh() method?

Well, there are 1% of users like the reporter of this issue who only needs refresh() to get env changes made inside the same process. For example, using Tcl.

You’re correct that refresh() doesn’t cover all use cases. I don’t think that it makes it useless.

Sadly, this use case is very specific to Windows, I don’t see how we can provide a function working the same way on all operating systems, whereas refresh() is portable and works on all platforms (it’s confirmed by its tests).

Honestly, given the current situation - my opinion is that this simply doesn’t belong as a public API in the stdlib. We’ve only seen 1 user demonstrate a real-world use case for this, and I think it will be confusing for much more people, making it simply not worth it.

Looking at your PR, I noticed refresh works by simply wrapping _create_environ. I would feel much more comfortable about new users reaching out to this by accident if it stayed as a private _create_environ, and a refresh would be done using os.environ |= os._create_environ. How do you feel about that?

I am hoping other people, especially those not yet involved in this thread, could give their opinion on this :slight_smile: Maybe my experience is simply skewed and this is not going to be confusing. Maybe this is actually more useful than I think.

9 Likes

I think 1% of users are too small of a group to please with a public API. We should design APIs for the common case.

I share @yoavdw’s concerns about this API.

5 Likes

Agreed. Can I suggest that the addition gets rolled back and left as a PR until it’s gained a little more positive support? Adding this simply because no-one objected strongly enough to insist on it being reverted seems like the wrong default.

5 Likes

I disagree. We have a way to change an environment variable omitting the cache with os.putenv() and os.unsetenv(), but there is no way to read its value back. os.getenv() only returns the cached value. We should solve this problem in one way or other, and os.environ.refresh() is a solution.

Why not fix the inconsistency in os.{get,put,unset}env instead? If it’s important (and there’s little evidence that it is, given how long those functions have worked the way they do) why not either:

  1. Have putenv/unsetenv update the cache?
  2. Add a flag to getenv to bypass the cache?

Either of those seem like sensible alternatives. Re-reading the whole environment to refresh the cache feels like overkill for most situations.

1 Like

It is not enough. If os.getenv() omits the cache, you can get a new value. But you cannot know what new variables were set.

Good point. But is there an actual use case for this?

I’m not arging that this change doesn’t allow doing something that’s not possible at the moment. I’m simply arguing that what it allows isn’t important enough to justify the risk that people get confused because they expect it to refresh variables from the parent process. And actually, I’m not even arguing that - my point is simply that the change was made before there was any real consensus that it was a good idea, and as a result we’re now having to argue against the change, invalidating the normal “status quo wins” principle.

3 Likes

If they exit Python and restart it from the shell, then they’ll still have the old environment, and in that case it’s an entirely new process, not just a “refreshed” process. I don’ think it’s reasonable to assume that a refresh() method would go beyond that. Plus the method is clearly documented. You seem to be trying to protect a hypothetical person who in practice would just make a silly mistake from failing to read the documentation, failing to read the doc string, and making a bad assumption. They learn their lesson, maybe learn a bit about what environment variables actually are, and move on.

You’re mixing up apples and oranges. What “refresh” would mean in the context of a launcher, shell, or IDE is entirely different from a reasonable interpretation of what it means in the context of a language runtime within a process. If someone doesn’t understand that context, this is just one more notch in their belt toward actually understanding how things work. They have to learn this stuff eventually if they’re programming computers in an capacity for their profession or hobby.

2 Likes

Builtin putenv() and unsetenv() from the posix or nt module could easily be wrapped to modify the cached data in os.environ. I suggested that option in my first post in this thread, since at least closes the hole in the standard library.

Sorry, I did not know that this issue was controversial. The new feature looked quite strightforward to me. It may be not the ideal solution, but I do not see obvious flaws in it.

But it is not good to revert just made changes, then discuss them, then apply them again if the discussion ends with approving the initial plan. We need to have some consensus before reversion.

An alternate solution may be to make os.environ not caching the environment variable values, but calling corresponding OS API to read and write the actual values. This can harm performance, and this needs a way to obtain the list of all environment variables for iteration. But the user will not need to call refresh() manually. Is it good to you?