Given mmap is at the same api level as os.open (right?) why would you use a higher level api to get the fd?
You need to read data in memory, edit it, and then write it back to the device without seeking.
Can you write to the device without using mmap?
Given mmap is at the same api level as os.open (right?) why would you use a higher level api to get the fd?
Iâve just followed the mmap â Memory-mapped file support â Python 3.10.13 documentation where the example code does the same:
with open("hello.txt", "r+b") as f:
# memory-map the file, size 0 means whole file
mm = mmap.mmap(f.fileno(), 0)
...
Iâve been searching the net for some examples mmap()'ing a device driver and could not find anything that would explicitly state on how to do it otherwise.
Yes. It is something Iâve mentioned in my opening posting.
I mean using Python, with open()?
Alright, I missed that. As far as I know, mmap in Python requires a seekable file descriptor.
1 Like
Fair enough. I do not see that mentioned on the mmap â Memory-mapped file support â Python 3.10.13 documentation. Sounds like it would be a discrepancy compared to good olâ C, therefore it might be worth mentioning.
Weâre back to open vs. os.open difference and how/where that comes about. Obviously we can open unseekable device with one and the other, but mmap seems to be more picky on what is expected to yield success.
1 Like
No, @elis.byberi Is just wrong. Clearly mmap is fine with unseekable devices (otherwise you wouldnât be able to work around it with os.open). It pretty directly maps to the C behavior (which can easily be verified by reading the source code).
The part of python that isnât find with unseekable devices is io.FileIO, i.e. the type returned by open. And I am still not really sure why. The fact that it is the one causing problem is easily visible from the error tracebacks you provided.
So no, we canât apparently canât open unseekable devices using open with mode r+b/w+b. I am not sure if this is a bug in python or something that is correct behavior and we donât know enough to figure out why.
1 Like
/dev/xdma1_user seems more like a seekable device (a block device, not a character device).
I am unable to map my mouse, though 
I can read it using any open() function (with sudo).
And yet itâs not a seekable device. Did you read this thread? Unless there are multiple conflicting definitions of âseekableâ that I am not aware of.
The file object returned from io.open and open when opening the device using mode âr+bâ is not seekable:
io.UnsupportedOperation: File or stream is not seekable.
The device in question may be exposed to user space as a character device or as a block device. However, it must be seekable for mmap to work. Since it is already working, it is capable of seeking.
In my experience, a character device is usually unseekable, hence the statement in my previous post. But it depends on the device and its associated driver whether you can map it using mmap.
That is not totally true as I can mmap it when using os.open as per previous posts.
The device in question is a) a char device and b) it is unseekable (it has no .llseek member set in its kernel drivers struct file_operations).
The culprit seems to be python performing a seek syscall between open and mmap. As soon as I disabled the buffering on the I/O I could do a mmap without errors:
f = open("/dev/xdma1_user", "r+b", buffering=0)
mm = mmap.mmap(fd.fileno(), 4096, flags=mmap.MAP_SHARED, prot=mmap.PROT_READ|mmap.PROT_WRITE)
Note the buffering=0 parameter to the open call.
I went a bit further and added a dummy .llseek handler to the kernel device driver just prove that it is a matter of missing seek support that was the initial issue. With this kernel driver modification I could do a buffered I/O open and successfully perform mmap as I was initially trying. I believe that proves the point that the seek is at the root of the problem.
FWIW, in the strace output I posted earlier, one can see a lines:
lseek(3, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
...
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = -1 EACCES (Permission denied)
Itâs not clear to me if the lseek error is benign and thats why python continues do mmap or if the execution should cease after failed seek (report a better reason for failure to the user).
1 Like
I do not understand why anyone would not use the os.open to get an fd for mmap.
Of course if its just curiosity about what python does internally that if a good reason to dig deeper.
This doesnât mean that the device isnât seekable. The driver just lacks the ability to reposition the read/write file offset.
The question is why the driver is missing the .llseek member when the .mmap member is set. Why does it allow using mmap (an inherently seekable medium), but not lseek()?
Mmap doesnât depend on lseek, as far as I know. When you open a file for updating, you must be able to seek, which the driver doesnât support, hence the I/O error.
How do you define âseekableâ if not via âthe ability to call seek on itâ?
The character device may expose a buffer, which is seekable (you can read/write a byte at any location). The lack of llseek support doesnât render it unseekable if mmap is supported.
So, in a seekable medium, you have the ability to read/write a byte at any location.
This device driver provides access to PCI cards control and status registers that get mmapâed to user process. Access to those registers is not sequential but random ; hence doing seek/read or seek/write would have have performance penalty. Doing ioctl instead would be an improvement but nothing beats mmap performance wise.
What you call âseekingâ I understand as ârandom accessâ, âseekingâ for me involves seek() syscall in this context.
Right. mmap can be made without seek. I was looking at the python code of open (not os.open) and there seek would be called (in case of buffered I/O access) and result in an error for me.
Both involve read/write at a given offset. Itâs the same thing, just in mmap, itâs faster because you avoid calling lseek() to set the offset; you just use pointers.
Would using the seek method of mmap make the device seekable?
import mmap
with open('example.txt', 'r') as file:
with mmap.mmap(
file.fileno(),
length=0,
access=mmap.ACCESS_READ) as mapped_file:
mapped_file.seek(10)
data = mapped_file.read(5)
print(data)
Also, mmap will always be seekable starting from Python 3.13.