# Change .pyc file format to record more accurate timestamp and other information

**URL:** <https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815>\
**Category:** Ideas\
**Created:** [July 10, 2024, 12:52pm UTC](https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815 "2024-07-10T12:52:40Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zheaoli](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/zheaoli/32/20954_2.png) [@Zheaoli](https://discuss.python.org/u/Zheaoli)\
**Post date:** [July 10, 2024, 12:52pm UTC](https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815/1 "2024-07-10T12:52:40Z")

</div>

This topic is related about [pycache 32-bit timestamp seems to have second granularity and uses pycache when they're the same · Issue #121376 · python/cpython · GitHub](https://github.com/python/cpython/issues/121376)

```bash
#!/bin/sh 
 
mkdir -p src 
rm -fr src/ __pycache__ 
rm -fr __pycache__ 
 
echo 'from src.test import f' > main.py 
echo 'print(f())' >> main.py 

echo 'def f():' > src/test.py 
echo ' return "Hello!"' >> src/test.py 
# prints "Hello!"
python3 main.py 

echo 'def f():' > src/test.py 
echo ' return "Goodbye!"' >> src/test.py 
# also prints "Hello!", but note the file does not say that anymore
python3 main.py

```

In the code up, the python use the old bytecode cache because of the timestamp issue(in 32 bit timestamp, the old code and the new code are runned at the same time)

So I think maybe we need to change timestamp from 32bit to 64bit to avoid this issue. or we can provide an more stable signature for the pyc file?

---

<div class="post-metadata">

**Author:** ![Zheaoli](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/zheaoli/32/20954_2.png) [@Zheaoli](https://discuss.python.org/u/Zheaoli)\
**Post date:** [July 10, 2024, 12:53pm UTC](https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815/2 "2024-07-10T12:53:13Z")

</div>

cc @storchaka

---

<div class="post-metadata">

**Author:** ![MegaIng](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/megaing/32/16162_2.png) [@MegaIng](https://discuss.python.org/u/MegaIng)\
**Post date:** [July 10, 2024, 1:17pm UTC](https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815/3 "2024-07-10T13:17:32Z")

</div>

A better way already exists, [PEP 552](https://peps.python.org/pep-0552/), but I don’t think there is a way to always generate such files, you need to that manually with e.g. compileall.

The problem is that there is no guarantee that better than 1 second resolution of the time stamp is available across OSes.

But also, are you completely sure that your bash script does what you think it does? The pyc file should have the source file length in addition to the timestamp, and from what I can tell you are changing the length.

---

<div class="post-metadata">

**Author:** ![Zheaoli](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/zheaoli/32/20954_2.png) [@Zheaoli](https://discuss.python.org/u/Zheaoli)\
**Post date:** [July 10, 2024, 5:39pm UTC](https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815/4 "2024-07-10T17:39:26Z")

</div>

> [@MegaIng](#):
>
> A better way already exists, [PEP 552](https://peps.python.org/pep-0552/), but I don’t think there is a way to always generate such files, you need to that manually with e.g. compileall.

Actually, PEP 552 is a good choice. I think it would be better if the hash mode is default

---

<div class="post-metadata">

**Author:** ![MegaIng](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/megaing/32/16162_2.png) [@MegaIng](https://discuss.python.org/u/MegaIng)\
**Post date:** [July 10, 2024, 5:57pm UTC](https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815/5 "2024-07-10T17:57:06Z")

</div>

I am sure this has been discussed before, I suspect that this isn’t done because reading in the entire source can be very slow, defeating the point of the cache. But for detailed discussions you would have to search around a bit.

---

<div class="post-metadata">

**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\
**Post date:** [July 11, 2024, 10:03am UTC](https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815/6 "2024-07-11T10:03:08Z")

</div>

I have the following planes for future `pyc` format:

- Larger (at least 4 bytes, preferably 8 bytes) signature to detect `pyc` files. Currently starting bytes of the `pyc` file are changed for every Python version (and several times during developing time), so utilities like `file` need to support a list of signatures for different Python versions.
- Explicit Python version. Currently the `py` launcher maintains a table of Python versions to detect Python version.
- More precise timestamp of the source file. Most filesystems support more than seconds precision. This will reduce a chance of errors when the source file has been very quickly overridden with new content with the same size.
- Larger range for timestamp. This is not critical. Currently the problem can only occur after 2120th when load `pyc` files created in 1990th.

We can also consider larger changes. For example, change the `marshal` format to support lazily loadable content (docstrings, line numbers, annotations).

---

<div class="post-metadata">

**Author:** ![Zheaoli](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/zheaoli/32/20954_2.png) [@Zheaoli](https://discuss.python.org/u/Zheaoli)\
**Post date:** [July 11, 2024, 5:36pm UTC](https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815/7 "2024-07-11T17:36:09Z")

</div>

Thanks for the reply sir. I might have some issue about your plan

> [@storchaka](#):
>
> Larger (at least 4 bytes, preferably 8 bytes) signature to detect `pyc` files. Currently starting bytes of the `pyc` file are changed for every Python version (and several times during developing time), so utilities like `file` need to support a list of signatures for different Python versions.

I 'm not sure about this part. For now, we have 4 32bits for the pyc header

1. First 32bits used for version magic number
2. Second 32bits used as control flag
3. 3/4 32bits used for timestamp/filesize when not in hash mode
4. 3/4 merged as 64bit to hash data for hash mode.

I’m not sure about the `signature` means. Is this just for the magic number part?

> [@storchaka](#):
>
> More precise timestamp of the source file. Most filesystems support more than seconds precision. This will reduce a chance of errors when the source file has been very quickly overridden with new content with the same size.

Yes maybe we need to extend the timestamp from 32bit to 64bit!

> [@storchaka](#):
>
> We can also consider larger changes. For example, change the `marshal` format to support lazily loadable content (docstrings, line numbers, annotations).

Can’t agree anymore!
