I am processing data with lots of datetime(48bytes) information. I often subtract two datetimes to get a timedelta(40bytes), and then I use .total_seconds() for math stuff.
To make my program more memory efficient, I am grouping the data by date(32bytes). This way the timestamps (datetime) within each day can be converted to time(40bytes).
However, I was shocked to discover that you cannot subtract two time objects to get a timedelta. Instead you get this: “TypeError: unsupported operand type(s) for -: ‘datetime.time’ and ‘datetime.time’”
There doesn’t seem to be a time.total_seconds method for time math either. I will probably modify my code to use date and an int(28bytes) for seconds past midnight in lieu of time. It would be nice if time supported subtract though.
Thanks – Bob
P.S. As a C++ program I was surprised that float(24bytes) is smaller than int(28bytes). Very odd.
Don’t take sys.getsizeof() too seriously. Both of those take 48 bytes. What getsizeof() doesn’t know (or account for) is that all heap-allocated types are aligned to 16-byte address boundaries,
Right.
That’s right. Datetimes represent an unambiguous point in time. Time objects do not. What should, e.g. 1am-2am return? -1 hour, or 23 hours? There’s no “one size fits all” answer to that, so Python refuses to guess.
Then you can do arithmetic, with the semantics you want, in a natural way - and all time objects are supported.
That depends on the magnitude of the int. Python ints are unbounded, and require additional info to record how many “internal digits” each requires. As at the start, though, 24 and 28 bytes both round up to 32 bytes due to 16-byte heap alignment. 2428 bytes is the smallest an int object can be (ignoring alignment padding). Assuming a 64-bit box, ints look like so:
type pointer 8 bytes
refcount 8 bytes
sign+#digs 8 bytes
digits 4 bytes for each internal 30-bit "digit"
I tested the following text script. I did not observe an error as you described. However, I do not know what the delta time that you tested (hours, days, etc.?). I tried 3 minutes for a quick test to see if at the very fundamental level an exception would be observed as you pointed out.:
import datetime
from time import sleep
a = datetime.datetime.now()
print(a)
sleep(180)
b = datetime.datetime.now()
print(b)
print(b-a)
I wonder if the OP can use an alternative. Interesting to see if the OP would be open to this method and to know the maximum time delta values of concern in their application. Like I questioned in parenthesis, are the time deltas in hours, days, etc.? Is it in minutes, seconds?
This would help in addressing their issue.
datetime was designed mostly for efficient field extraction (year, month, …), not so much for arithmetic or minimal RAM. If you want to store more compactly in RAM, then “seconds from midnight” is a fine way to do it. Here’s a way that retains full precision (including microseconds!), at 8 bytes per datetime.
import datetime
from random import randrange
from array import array
now = datetime.datetime.now()
# chop back to midnight
base = datetime.datetime.combine(now.date(),
datetime.time())
# create a thousand random times on today
dts = []
for i in range(1000):
dts.append(base.replace(hour=randrange(24),
minute=randrange(60),
second=randrange(60),
microsecond=randrange(1_000_000)))
# store only seconds since midnight, as 8-byte floats
compressed = array('d')
for dt in dts:
compressed.append((dt - base).total_seconds())
# verify
for expected, got in zip(dts, compressed):
got = base + datetime.timedelta(seconds=got)
assert got == expected
Or arithmetic can be done directly on the floats.
If you don’t care about fractional seconds, you can cut that to 4 bytes per entry, by converting total seconds to an int (truncate or round or … whatever you want), and storing that in an array.array('I') instead.
Or even to 2 bytes (array.array('H')) if you, say, chop down to the closest even number of seconds, and store that after dividing by 2. There are 86400 seconds in a day, but an unsigned 2-byte integer can only express 65536 different values.
And one more. It’s possible to store an entire (naïve) datetime in an 8-byte signed int, by viewing it as an integer number of microseconds since 1/1/1. An 8-byte float doesn’t have enough bits of precision for this, though.
A few utility functions and you’re good to go:
[EDIT: simplified dt2us()]
from datetime import datetime, timedelta
EPOCH = datetime.min
US_PER_DAY = int(timedelta(days=1).total_seconds()) * 1_000_000
# convert datetime to integer number of microseconds since epoch
def dt2us(dt):
delta = dt - EPOCH
# The result is conceptually delta.total_seconds() * 1e6, but a
# float doesn't have enough bits of precision to avoid losing
# info to rounding. So stick to exact int arithmetic.
result = (delta.days * US_PER_DAY
+ delta.seconds * 1_000_000
+ delta.microseconds)
assert result.bit_length() <= 59
return result
# convert integer number of microseconds since epoch to datetime
def us2dt(us):
return EPOCH + timedelta(microseconds=us)
for dt in (datetime.min,
datetime.now(),
datetime.max):
us = dt2us(dt)
got = us2dt(us)
print(f"{dt} {us=:_} equal? {dt==got}")
dt2us() returns a Python int, but, as before, you can reduce the RAM needed to 8 bytes by storing it in an array.array with type code "q" or "Q". I would use "q" (the sign bit will always be clear, so it really doesn’t matter, but sticking to signed 64-bit ints can help under the PyPy implementation of Python).
Thanks for the great answers, Tim! You have really improved my understanding of Python memory usage (especially ints). Also, thanks for clueing me into the array module. It seems ideal for what I am trying to accomplish. I have also looked at numpy arrays, but that module is much less intuitive (to me at least).
Finally, I will solve your time subtraction dilemma by telling you that 1am-2am should return -1hrs.