My cProfile profiling run time has gone from 1.5 - 2.5 times normal execution speed in python3.9 to 10 times or more in python 3.11 and 3.14.
Google says it’s because of PEP 659 and cites " Function Call Frequency" as one of the possible reasons. “If your code contains short, frequently called functions or operates in “hot loops”, the cumulative stopwatch latency compounds, drastically amplifying execution time.”
I make a lot of method calls.
Profiling pypy3 shows
123918383 function calls (123902791 primitive calls) in 8.705 seconds
several methods are called over a million times a second.
I don’t understand why that static method takes so much longer when I profile it. I am happy to change the method, I just don’t know how I should change it to make it profiler friendly
It’s really only the methods that I call several million times that seem to do so poorly when I profile them.
Python needs to create a new frame object for every call made, and the cost can add up, especially when Python does not use tail call optimization and friends.
Even without the “frame object” in the strict sense, the essential metadata is still there. The changes, though great, cannot fully eliminate this, and we’re talking about millions of calls here with profiling.
Isn’t the problem the OP is pointing out different? They’re asking why it’s so much slower with profiling than without. Without profiling, the OP reports much better performance.
I think @Stefan2 may be on to something. I don’t remember exactly now but I suspect profiling might be interfering with specialization, causing us to use the slower call paths than the fast ones we introduced in 3.11.
Tachyon sounds great to me, the name very appropriate too, I like it.
I tried pyinstrument a little, but I’d prefer to use a profiler from python on python code, I think.
3.11 is where I first notice the behavior when profiling, I’m testing with 3.11 and 3.14
What I was doing is reading a video and slicing off 188 byte packets, and calling another method for each packet which in turned called a few more methods. Now, I when I slice off a packet Instead of calling another method, I processes it right there inside the loop. That eliminated about 100 million method calls. It’s not as pretty but it did drop the run time from 14 seconds to 9 seconds and the profiling time is down from 209 seconds to 48 seconds.
11533363 function calls (11533063 primitive calls) in 48.979 seconds
I’m glad you asked, I have spent a lot of time trying to figure the best way to do that for both python3 and pypy3. At one point I was checking sys.version so I could optimize for one or the other.
you’ll see calls to reader, reader returns a file handle, I use it to open files, and http(s), and Udp multicast and unicast so I can deal with everything the same way. For sockets it returns makefile()
I have split packets mostly with iter/partial this
was during my comprehension phase
from functools import partial
from threefive import reader
PKTSIZE =188
NUMPKTS=1400
CHUNKSIZE= PKTSIZE * NUMPKTS
with reader(tsdata) as rdr:
for chunk in (partial(rdr.read, CHUNKSIZE), b"")
cues = [parse(chunk[i : i + self.PACKETSIZE]) for i in range(0, len(chunk), PACKETSIZE)]
_=[func(cue) for cue in cues if cue]
then I went to a double walrus technique for a while
import io
from threefive import reader
with reader(tsdata) as rdr:
while chunk := io.BufferedReader(rdr, buffer_size=CHUNKSIZE):
while pkt := chunk.read(PKTSIZE):
cue = self._parse(pkt):
if cue:
func(cue)
I figured out where the issue seems to start, it’s when I split off the packets that 3.11 runs faster than 3.14
If I do
def test(arg,num_pkts):
pkt_size=188
chunksize = int(num_pkts) * pkt_size
with open(arg,'rb') as stuff:
while chunk := stuff.read(chunksize):
continue
python3.14 runs as fast or faster than 3.11 with
num_pkts = 1,7,70,700,7000, or 70000 (arg is an mpegts video file)
When I change the function to
def test(arg,num_pkts):
pkt_size=188
chunksize = int(num_pkts) * pkt_size
with open(arg,'rb') as stuff:
while chunk := stuff.read(chunksize):
for i in range(0, len(chunk), pkt_size):
pkt= chunk[i : i + pkt_size]
The reason I keep using multiples of seven is that mpegts often is streamed in 1316 byte datagrams, or 7 packets so it’s best to parse in groups of 7 packets.
