When writing code which loads dynamic libraries, it is often very useful to be able to query which shared libraries are already in use by the current process. There are a few well-known tricks to do this, but they all end up being platform dependent.
For example, on Linux, you can use dl_iterate_phdr, and it seems that quite a bit of Python code does. This won’t work on macOS or Windows, which provide other functions for this same functionality.
Julia provides this function in the standard library under Libdl.dllist.
A Python re-implementation of the same platform-specific code can be found at GitHub - WardBrian/dllist: List DLLs loaded by the current process. This essentially just wraps the platform specific code in an if-else based on the runtime platform
Does something like this fit in the ctypes.util module of the standard library?
It might. I’d suggest seeing how popular the dllist module is as a package on PyPI. If it gets used a lot, then that would be a good argument in support of proposing it gets added to the stdlib.
I do think a lot of people who need this functionality may already have ad-hoc implementations themselves, so are unlikely to switch to a library by some random developer (e.g., me), but would likely use a version in a standard library. We will see, I suppose!
I’m assuming that if it goes into the stdlib, you’d be creating the PR to add it - so either way, it would be the same code written by the same random developer
But I get your point. Being in the stdlib adds a level of credibility and convenience. It’s possible that if you just submit the function as a PR, one of the core devs would be happy to approve and merge it (I won’t do so myself, simply because I don’t have enough involvement with ctypes to feel comfortable merging changes to it). Being a popular module on PyPI isn’t necessary, it’s just helpful (if the proposal needs supporting evidence).
Hi @WardBrian
Today I was looking for this exact functionality you are proposing in ctypes, and happened upon this thread. Just to track any work done here, do you have an open PR to add this to the stdlib, or a plan to do so? If not, I’ll happily use your utility library for now.
I have not yet opened a PR for this being added to the standard library. I still think it would be quite useful, but my library has thus far seemed sufficient for my own purposes. I wish I could extend it to things like BSD, but lack the testing infrastructure to know if what I wrote would actually work there.
If there is interest I would be happy to open a contribution. So far my understanding is the only package that depends on dllist is another one I maintain, but besides this thread I haven’t really tried advertising it.
For what it’s worth, I was also looking for this functionality recently, didn’t find it in the standard library and had to figure it out myself. It would be helpful to have it provided as part of ctypes.util.
I didn’t know about dl_iterate_phdr and instead resorted to reading /proc/self/maps on Linux, so thanks for the tip.
@WardBrian I could probably help with some basic testing on FreeBSD. And there’s a FreeBSD builder in the CI as well. According to the dl_iterate_phdr man page, it’s been available on FreeBSD since version 7.0, so that should presumably work.
From my reading, the same low-level API I am using on Linux should also be available on FreeBSD, OpenBSD, and Solaris. I lack the ability to verify that either with my own machines or with Github Actions, but it is good to note if this would be added to the stdlib
The implementation of dllist on Windows should use ctypes.WinDLL with use_last_error=True, and get the last error via ctypes.get_last_error(). One reason for this is that a package shouldn’t modify the state of the global loaders ctypes.windll and ctypes.cdll. Those are a convenience for scripts, which was never really a good idea. Modifying function prototypes from the loaders can lead to clashes between packages that define function prototypes differently. Regarding use_last_error=True, high-level Python code really shouldn’t depend on GetLastError(). The value needs to be captured in C as soon as the foreign function call returns. That’s the reason for use_last_error, get_last_error(), and set_last_error(). The same applies to use_errno, get_errno(), and set_errno() for C errno.
At the design level, I think the implementation in “dllist/windows.py” is more complicated that it needs to be. It also shouldn’t repeatedly incur the cost of define prototypes inside of the API wrappers. Also, since the use case is just for the current process, I’d use EnumProcessModules() instead of EnumProcessModulesEx(). One last note is that slicing an HMODULE array returns a list of integers, and the typing return type should reflect that.
Here’s a modified implementation for reference:
import ctypes
import warnings
from ctypes import wintypes
from typing import List, Optional
# https://learn.microsoft.com/windows/win32/api/psapi/nf-psapi-enumprocessmodules
# https://learn.microsoft.com/windows/win32/api/libloaderapi/nf-libloaderapi-getmodulefilenamew
_kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
_psapi = ctypes.WinDLL('psapi', use_last_error=True)
_kernel32.GetCurrentProcess.restype = wintypes.HANDLE
_kernel32.GetModuleFileNameW.restype = wintypes.DWORD
_kernel32.GetModuleFileNameW.argtypes = (
wintypes.HMODULE,
wintypes.LPWSTR,
wintypes.DWORD,
)
_psapi.EnumProcessModules.restype = wintypes.BOOL
_psapi.EnumProcessModules.argtypes = (
wintypes.HANDLE,
ctypes.POINTER(wintypes.HMODULE),
wintypes.DWORD,
wintypes.LPDWORD,
)
def get_module_filename(hModule: wintypes.HMODULE) -> Optional[str]:
name = (wintypes.WCHAR * 32767)() # UNICODE_STRING_MAX_CHARS
if _kernel32.GetModuleFileNameW(hModule, name, len(name)):
return name.value
error = ctypes.get_last_error()
warnings.warn(f"Failed to get module file name for module {hModule}: "
f"GetModuleFileNameW failed with error code {error}",
stacklevel=2)
return None
def get_module_handles() -> List[int]:
hProcess = _kernel32.GetCurrentProcess()
cbNeeded = wintypes.DWORD()
n = 1024
while True:
modules = (wintypes.HMODULE * n)()
if not _psapi.EnumProcessModules(hProcess,
modules,
ctypes.sizeof(modules),
ctypes.byref(cbNeeded)):
break
n = cbNeeded.value // ctypes.sizeof(wintypes.HMODULE)
if n <= len(modules):
return modules[:n]
error = ctypes.get_last_error()
warnings.warn("Unable to list loaded libraries: EnumProcessModules "
f"failed with error code {error}",
stacklevel=2)
return []
def _platform_specific_dllist() -> List[str]:
# skip first entry, which is the executable itself
modules = get_module_handles()[1:]
libraries = [name for h in modules
if (name := get_module_filename(h)) is not None]
return libraries
You have my permission to include the sample code that I published in post 10 of this topic and make whatever changes you see fit. If you send me a link to the PR, I can review it if you want.
Thanks @eryksun - I didn’t end up making any changes, so there’s no need for an extra review
On the general topic, it’s not entirely clear to me what the contribution process is for something like adding this functionality to ctypes.utils - does it need a PEP? Is there something smaller than a PEP (… but bigger than this forum thread) that’s more suited for such library additions?
IIRC, it wasn’t fully agreed upon what should happen if the system isn’t supported. Should we raise an exception or return None? An exception isn’t particularly useful, but a None return isn’t nice to debug. It would be nice to get a core dev opinion on that before approving anything.
Yes, there was also a secondary question about what circumstances, if any, should lead to a warning. It would be great to get a core dev to give a lean one way or another
