Datetime API: No atomic way to get "aware" current local datetime when moving between time zones

If I want to get a datetime instance that represents current local time and that is tagged with the current local timezone (i.e., an “aware” object), the recommended way to do this right now is as follows:

from datetime import datetime
datetime.now().astimezone()

Now, if this code was to be executed on a mobile device that is physically moving across time zones, it is possible for the system time zone to change during program execution. In such cases, we should expect to get the current local time in either the old or new timezone. However, because of two separate API calls (now() and astimezone()) that both rely on querying the local timezone, we can actually end up with nonsensical values that do not represent the current point of time in either time zone.

Here’s an example demonstrating the issue:

from datetime import datetime
import os, time

def settz(tz): # Should work on most Unix systems
	os.environ['TZ'] = tz
	time.tzset() # Applies change

print(f"Current time = {time.time()}")

# Let's run datetime.now() in ET
settz('US/Eastern')
dt = datetime.now()
print(f"now() time   = {dt.timestamp()}")

# Then assume we drive into the CT zone just before
# calling astimezone()
settz('US/Central')
dtz = dt.astimezone()
print(f"astimezone() = {dtz.timestamp()}")

Running it on my system right now:

Current time = 1717617045.704923
now() time   = 1717617045.705231
astimezone() = 1717620645.705231

Notice that the difference between the last two rows is 3600 seconds, or about one hour! It’s actually the wrong current time, regardless of which time zone the value dtz it represents.

The core problem is that the expression datetime.now().astimezone() requires at least two lookups* of the system local time zone: (1) the call to datetime.now() produces a naive instance by converting the system-generated time_t to a datetime instance using the system local time zone but without attaching a tzinfo object; (2) The astimezone() call again looks up the system local time zone to attach a tzinfo object containing the local offset from UTC. If the local time zone has changed between these two steps, we get a result that is not correct in either time zone.

Workaround?: One workaround I can think of for this is to first get a tzinfo object corresponding to the local time zone and then to call now() with that tzinfo:

local_tzinfo = datetime.now().astimezone().tzinfo # During a physical transition, will return at least one of the timezones encountered
datetime.now(tz=local_tzinfo) # Does not rely on localtime() at all

However, this is not intuitive. In fact, the code is probably not correct if the physical change of time zones also happens during a DST transition, since local_tzinfo could represent a timezone on the other side of DST as compared to the timezone followed locally now().

Ideal Solution: I think the datetime library should have an API that produces the current local “aware” datetime with just one atomic method call. Perhaps something like datetime.now(tz=timezone.local). Also the API documentation should mention what to expect when the system transitions through time zones so that users understand the nuances when developing for mobile platforms. Thoughts?

* CPython actually makes several more calls to the localtime() in order to compute the fold attribute correctly, which results in many more potential outcomes other than what I’ve shown above depending on when exactly the system changes time zones.

6 Likes

I’m impressed by the amount of effort you’ve put in to this. If there’s a straightforward solution then fantastic. Otherwise, unfortunately, I think the best thing to take away from it is your superb example of an edge case:

if this code was to be executed on a mobile device that is physically moving across time zones, it is possible for the system time zone to change during program execution

If someone else wants to support users with that requirement (does Python code run on avionics systems, or on ICBMs?), then go for it - fill your boots. Personally I’m happy to wait for the bug reports on that one. And if so recommend they use specialist date time tools, or connect to an NTP server and figure out a faster or more accurate datetime and timezone, themselves etc.

1 Like

Could you elaborate more on this?

Sure, I can elaborate.

Let’s say the current local time on my mobile device 2024-06-05 19:00:00 UTC-04:00 (that is, 7pm EDT) and I want to execute datetime.now().astimezone(). I am driving across a border and the system switches to CDT where it is 6pm. I might thus expect a timezone-aware now() to return 2024-06-05 18:00:00 UTC-05:00, which is perfectly fine. Both of these represent the same point in time (i.e., have the same UTC timestamp). However, it is also possible for me to get the value 2024-06-05 19:00:00 UTC-05:00 (7pm CDT), which is one hour in the future—it is not currently 7pm in CDT. This is because the first call to now() returns a naive datetime instance representing only 2024-06-05 19:00:00 (without attaching the UTC-04:00 offset) and then if the system timezone transition happens just before calling astimezone(), I will subsequently get a tzinfo representing UTC-05:00, which then gets attached to the datetime instance representing the hour value from the previous time zone. The final result is a timezone-aware datetime instance representing a point in time one hour in the future. Even if we try to convert it back to UTC-04:00 now, we will still get a wrong result (it would show 20:00:00 UTC-04:00, even though it is not currently 8pm EDT).

1 Like

I know almost nothing about ICBMs. I used to work on a submarine with SLBMs. We used “Zulu” (military for what was then GMT and is now UTC.)

3 Likes

How wide is this window? Are we talking “theoretical problem, unlikely to ever happen” or “this would become a constant annoyance to someone making a clock app” or “this is going to cause major problems for anyone who crosses timezones”?

That’s a race condition! You have to listen to the OS event for timezone changes if the OS supports it, or you can view the last timezone changes, e.g., in Ubuntu with ls -l /etc/localtime.

Being a race condition, nothing guarantees that the displayed time would be correct. Even if you manage to make now() timezone aware, the timezone could change just a fraction of a second after displaying the time. :man_shrugging:

3 Likes

It is indeed a race condition, and the question is whether the platform (in this case, the datetime API) supports a way to get a consistent view of the world as it was either before or after the racing event, without necessarily guaranteeing which one.

If we support a way to return a timezone-aware instance from now() directly, we can indeed ensure that we get a value that is either correct just before or just after the timezone transition (in my example above, either 19:00 UTC-04:00 or 18:00 UTC-05:00 which are both correct), but regardless it represents the current point in time. Even if the timezone changed just a fraction later, that’s totally fine. We would have something that represents the correct current time displayed in a timezone we just traveled from, no problem. Most operations we would perform with this value, such as scheduling an event or checking an expiration date will work just fine. The current issue is that it is possible to get values that represent other points in time (e.g., 19:00 UTC-05:00 in my example above, which represents a point of time in the future regardless of where you are located), which could result in unpredictable behavior of a client program.

3 Likes

Thanks. I think one simple solution would be to support something like datetime.now(tz=timezone.local), where timezone.local is not an actual tzinfo instance but rather a sentinel value that tells now() that we want an aware instance of current local time (this also avoids having tzinfo objects that change their behavior depending on location, as
@pganssle has written about). This would produce the same behavior as now(tz=None) followed by astimezone(), except it would not require querying the system local time multiple times.

Edit: I just came across a similar discussion topic which proposed the same API improvement but without going into the problems associated with moving across time zones. I don’t know if it is possible to link discussions (apologies, I’m new here).

4 Likes

Getting the current time in the specified timezone provides the correct time:

from datetime import datetime
import pytz
import os
import time

def settz(tz): # Should work on most Unix systems
    os.environ['TZ'] = tz
    time.tzset() # Applies change

# Function to get current time in the specified timezone
def get_current_time_in_tz(tz_name):
    tz = pytz.timezone(tz_name)
    return datetime.now(tz)

print(f"Current time (UTC) = {time.time()}")

# Let's run datetime.now() in ET
et_tz = 'US/Eastern'
settz(et_tz)
dt = get_current_time_in_tz(et_tz)
print(f"now() time   = {dt.timestamp()}")

# Then assume we drive into the CT zone just before calling astimezone()
ct_tz = 'US/Central'
settz(ct_tz)
ct_tz_info = pytz.timezone(ct_tz)
dtz = dt.astimezone(ct_tz_info)
print(f"astimezone() = {dtz.timestamp()}")

Result:

Current time (UTC) = 1717669401.4972122
now() time   = 1717669401.498265
astimezone() = 1717669401.498265
1 Like

Yes, true. Thanks for the snippet. The problem however is related to getting an aware-instance in local time. When developing an application, I don’t know what timezone the user may be in, so I can’t explicitly set a tzinfo value based on zone names. I only used the settz in the snippet in my OP to demonstrate how system time zones might change externally; in a real application, there would be no explicit reference to US/Eastern or US/Central. So, the programmer must rely on the datetime API to accurately get the current time—the main way to do this as per the API documentation is datetime.now().astimezone().

Similar to your suggestion, I posted a workaround in my OP which does use specific timezones by first querying the local time zone and then storing it, though that approach requires calling now() twice and querying the system local time at least 3 times, which IMO is a bit ugly and not intuitive.

The point of my original post is that the Python standard library does not have a single API method to get an aware-instance of current local time, which seems to me like it should be quite a common requirement.

2 Likes

As discussed in the other thread, getting timezone is possible, with its caveats, but working with time is not a purely technical challenge.

The rule I have used is to get the UTC time and convert to local timezone for presentation. I have not looked at how to do this with datetime calls only.

You can get the UTC time with time.time() and create a datetime object from that withe current timezone.

Is it still the case that if the system’s timezone changes glibc does not update its notion of timezone while a program is running?
So a user on a laptop crossing a timezone boundary will never this this issue, but face a different one?

3 Likes

This workaround is not really reliable. Time offset may change between now() calls due to change of daylight saving time state or due to some administrative rule. As a result you have a chance to get an object with combination of local time and offset inconsistent with your time zone. It is not the same as moving from one time zone to another.

Notice that astimezone() creates a fixed offset time zone, not a ZoneInfo object having history of time transitions taking into account DST rules. So tzinfo is valid only for the given timestamp.

The issue is that getting identifier for ZoneInfo and watching its changes is a platform-dependent task. It may be reading the /etc/localtime symlink and listening inotify events or using D-Bus interface. In the past timezone data may be stored in the /etc/timezone file, so historical data were accessible, but timezone identifier was unknown.

import time
ts = time.time()
time.localtime(ts).tm_gmtoff

Allows to get consistent values of local time and timezone offset, but tm_gmtoff is an extension and may be missed. You may try to compute difference of time.localtime(ts) and time.gmtime(ts).

I believe that unless it is expensive, datetime should provide a convenient way to get consistent time and tzinfo. Perhaps complications and pitfalls due to timezones should be more prominently stressed in the library docs.

P.S. A great collection of wisdom:
https://data.iana.org/time-zones/theory.html
“Theory and pragmatics of the tz code and data”

1 Like

I suspect I’m missing something, but if you use datetime.datetime.now(datetime.UTC), this seems to resolve the initial demonstration:

from datetime import datetime, UTC
import os, time

def settz(tz): # Should work on most Unix systems
  os.environ['TZ'] = tz
  time.tzset() # Applies change

print(f"Current time = {time.time()}")

# Let's run datetime.now() in ET
settz('US/Eastern')
dt = datetime.now(UTC)
print(f"now() time   = {dt.timestamp()}")

# Then assume we drive into the CT zone just before
# calling astimezone()
settz('US/Central')
dtz = dt.astimezone()
print(f"astimezone() = {dtz.timestamp()}")
Current time = 1717783092.9070656
now() time   = 1717783092.90712
astimezone() = 1717783092.90712

Is it just that I’m pushing the race inside datetime.now(), so you could run datetime.now(UTC) twice and get results off by an hour?

delta = abs(datetime.now(UTC) - datetime.now(UTC))
assert delta.total_seconds() < 1e-3  # boom
1 Like

If there is a race when asking for datetime.now(UTC) then that is a bug that should be fixed I’d claim.

As far as I know, that still requires two lookups of localtime: datetime.fromtimestamp(time.time()).astimezone() and can be affected by a timezone transition between the calls to fromtimestamp() (which depends on localtime to get a naive local datetime) and astimezone() (which also depends on localtime to create a tzinfo object).

I think what both these comments indicate is probably a much better workaround: datetime.now(UTC).astimezone() — this seems to require (at least in theory) only one lookup of system local time, and so should not be susceptible to the race condition. I don’t know if there is a race inside now(UTC) still, but that would be an implementation issue. From an API perspective, now(UTC) should not depend on local time zone.

1 Like

Maybe I’m missing something. but has it been established that it’s possible to get consistent time and tzinfo information at the C level, on all platforms we support?

This feels like an extremely rare race condition we’re talking about here, and while I don’t personally have a feel for how serious it is in practice, I’d assume that a bare minimum level would be “do the common operating systems provide facilities to address it?”

If there’s an OS-level capability, then exposing it to Python seems like a much more reasonable request that simply claiming that Python needs this capability anyway, to avoid a race condition which appears to be purely theoretical at the moment.

By design it is always possible. Windows, macOS and unix have the APIs.

You can always get UTC time where timezone is never needed on all OS.
You can always convert the UTC time to another timezone’s time on all OS.

Trivia: gettimeofday() (I forget the Windows function name) on all OS is one of the most optimised API calls. Slowing it by a few uS causes a measureable slow down of Windows and linux, etc at the user level.

The problem that the OP relates is that datetime.datetime.now() return local, wall clock, time. Now you have to back convert to get UTC and forward convert to get TZ of choice, two operations that require a timezone.

Question is how is now() implemented and I have not looked yet.

This is a deliberate theoretical race condition. It is deliberate because we are not storing timezone information; we depend on the system for the timezone. If the system timezone changes, who is to blame? Us, for not keeping a timezone-aware time. It is theoretical because there is very little chance for the timezone to change between two sequential calls.

If you need a timezone-aware datetime object, you should simply provide a timezone, e.g., UTC. Two sequential calls of datetime.now(timezone.utc) will not have issues related to timezone changes because UTC does not observe daylight saving time. However, other factors like system clock adjustments could still affect the results slightly, but they won’t drift by more than a small fraction of a second under normal circumstances.

I don’t see any bug in datetime.now().