Why import * only allowed at module level

Python Help
21

Thanks for your time again

if you use Windows OS, create a little c project that embed Python as simple as the one below:

#include <python.h>
int main(...)
{
	Sleep(5000);
	Py_InitializeEx(0);
	if (Py_IsInitialized())
	{
		Py_Finalize();
	}
        //Here if python would release the memory, you will see the usage going down.
	Sleep(15000);
}

Open , Task manager, then and watch how that litle project uses memory and it never release it untill the app finish.

if this little sample is not enough to you go to:
AlexSoft73/cpython

check the file obmalloc.c, I made some modifications there, read the beginning (see bellow), then run the project in a debugger and you will see how many times the python core request memory from the OS at Py_InitializeEx that never Py_Finalize return to it, specially the first time it is called Py_InitializeEx. If this is not your concept of memory leaks, then I am from Marts.

//Alexei Debugging Memory Leaks
//uncomment the following macro definition to have messages in the debugger console regarding memory allocations and releases
//#define ALEXSOFT_MEMLEAKS_PURSUING
#ifdef ALEXSOFT_MEMLEAKS_PURSUING
unsigned int AllocCalls = 0;
unsigned int FreeCalls = 0;
#endif
22

Insufficient test. Put a loop around it and see if it uses the same memory again, or if the memory usage keeps going up.

23

It does, not as much as at the beginning but it does, and I was clear the first call is the worst.

if we read in the api help for python Py_FinalizeEx(), among things it says:

Undo all initializations made by Py_Initialize() and subsequent use of Python/C API functions, and destroy all sub-interpreters (see Py_NewInterpreter() below) that were created and not yet destroyed since the last call to Py_Initialize(). Ideally, this frees all memory allocated by the Python interpreter. This is a no-op when called for a second time (without calling Py_Initialize() again first). Normally the return value is 0 . If there were errors during finalization (flushing buffered data), -1 is returned.

then, the question is: what is all that memory specially the one allocated for the first time used for?

24

Look into the way that memory allocators work. Itā€™s very VERY common for released memory to not go all the way back to the OS, simply because itā€™s impractical to do so. But that memory is still available for reallocation, meaning that it HAS been marked as released. So the only way to prove that itā€™s a leak is to do it lots of times and see what happens repeatedly.

25

Thanks for youy time Chris Angelico

There are tons of ways to justify a memory leak, yours is one of them. It is true what you said, but until you use Python again in the application where python is embedded, the memory is not accessible, becoming a memory leak, not complying with the statement in the help for function Finalize " Ideally, this frees all memory allocated by the Python interpreter.". Not to mention that if you run python code like import numpy as np things go even wost, but letā€™s leave that for future debate.

One thing I see with Python community is: The Thinking of Python language as the center of the world where that everything must be around it. And life is not like that, Python is a great scripting language with its strongness and weaknesses like every other programming language. Being that said, the big benefit of todayā€™s programming languages is to be ability of mixing them taking advantage of each language strongness.

Thanks again for your time guys

26

Define ā€œfreeā€. If I do this, have I freed all memory?

void *p1 = malloc(100);
void *p2 = malloc(100000);
free(p1);
free(p2);

I would say that, yes, I freed everything I allocated, right? But thereā€™s no guarantee from the C standard library that the memory is given all the way back to the OS - only that the memory is, once again, available for allocation. In other words, if you run this code in a loop, the processā€™s memory usage will not continue increasing.

Iā€™m not sure what youā€™re getting this from, nor what it has to do with memory leaks. Of course multiple languages exist for a reason.

27

Thanks Chris

By Free I mean, free for the application to use (not the OS), remember that if I called Py_FinalizeEx(ā€¦) I am telling python I am done with it, see help for function (Py_FinalizeEx), so, why Python keeps the memory for later use by next python embedding instance. Actually, it is healthier for next embedded instance of python to start as fresh new like the first time we called Py_InitializeEx(ā€¦).

Thanks again,
Alexei

28

Okay. In that case, you canā€™t use one initialize/finalize as a test; you have to do it multiple times and see whether it leaks more memory every iteration, or itā€™s reusing the same memory every time. Itā€™s entirely possible that there IS a leak (which would be a bug that needs to be fixed), but so far, the test is inconclusive.

29

Hi Chris

if all the time we spend arguing defending our positions regarding if it is a memory leak a bug or whatever you wish to call it. Spend it investigating the issue by now would be resolved or at least mitigated, see bellow, I wrote this in another debate thread regarding memory leaks of python (not open by me), also if you look all the thread that has been open regarding this matter, I think it deserves attention:

I wrote:
Python V3.12 has also that memory leaks when one finalizes it, among the memory leaks the biggest one (about 4Mb pages) occur by not releasing the Arenas Pool. See for
AlexSoft73/cpython

For testing I modified 2 files:

  • obmalloc.c (Trace all memory request that has no counterpart of free)
  • pylifecycle.c (includes a fix to at least release Arera pools memory used)

i spent sometime fixing python memory leaks all by myself, but, it is not easy to do all by myself, I work for a company and i donā€™t have all the time of the world for Python though I sympathies with this scripting language.

thanks,
Alexei

30

Take your existing code. Put a loop around it. See what happens.

31

I did that test with the modified python files

  • obmalloc.c (Trace all memory request that has no counterpart of free)
  • pylifecycle.c (includes a fix to at least release Areras pool memory used)

and you can see memory being free (not 100%) since like I said, I just work to release the major amounts (around 4Mb from arenas pool).

thanks,
Alexei

32

@AlexSoxft73 Have you opened a bug report on GitHub yet? Going back and forth with me and @Rosuav on a help forum serves no real purpose. One, the Python Help forum is not the correct place to report bugs. Two, while Chris and I are both long-time Python developers and Chris is a core developer, Iā€™m pretty sure you are not attracting the attention of the subset of core developers whose real interest is the deep down guts of Python memory management. Open an issue and those people will see it.

33

Thanks Skip Montanaro for your time once more,

Yes, I did:

among others and it was closed claiming this was not an issue, they said you should call only once Py_InitializeEx(ā€¦) and once Py_FinaliseEx(ā€¦) that it was not meant to be used this way and things like that. The fact it was a very ease way of dealing with the issue. That honesty, sound to me like leaving all users that use python embedded in their application abandoned!

Thanks a lot
Alexei