Mmap char device driver

[hinxx (Hinko)]

I have a linux device driver that has a /dev entry that I can mmap(2). For example I can use something like this from C as a regular user:

map_base = mmap(0, map_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, target_base);

Above line is from https://github.com/billfarrow/pcimem/blob/master/pcimem.c and when used I get this:

$ ./pcimem /dev/xdma1_user 0 w
/dev/xdma1_user opened.
Target offset is 0x0, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x0)
PCI Memory mapped to address 0x7f80bfb49000.
0x0000: 0x00000101

OTOH, with mmap in python:

import mmap
fd = open('/dev/xdma1_user', 'wb')
mmap.mmap(fd.fileno(), 4096)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [Errno 13] Permission denied

What causes the ‘Permission denied’ to be returned from mmap?

I can read and write using the fd, though:

 >>> binascii.hexlify(fd.read(4))
b'00000001'
>>> binascii.hexlify(fd.read(4))
b'00000000'
>>> fd.write(int(4).to_bytes())
1
>>> fd.write(int(4).to_bytes())
1

[MegaIng (Cornelius Krupp)]

You might need to open the path itself in read-write mode, i.e. with 'w+b' (which the C code also does). Otherwise try specifying proto and flags explicitly, although they should already default to the correct values.

[hinxx (Hinko)]

>>> fd = open('/dev/xdma1_user', 'w+b')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
io.UnsupportedOperation: File or stream is not seekable.

I tried to open in rb and setting the prot=PROT_READwith flags=MAP_PRIVATE but still got the same error: Permission denied.

[barry-scott]

Use os.open() instead of open() so you have full control as you do in C?

[hinxx (Hinko)]

Use os.open() instead of open() so you have full control as you do in C?

fd = os.open('/dev/xdma1_user', os.O_RDWR)
mm = mmap.mmap(fd, 4096)

No error! Thanks!!

[barry-scott]

Mark as the solution please.

[hinxx (Hinko)]

Gladly. How do I do that?

[barry-scott]

There should be a button like on each reply to click to set the solution.

[hinxx (Hinko)]

This is what I see, sorry no such button to click on:

[Rosuav (Chris Angelico)]

@hinxx

This is what I see, sorry no such button to click on

Yeah, I’m not sure where that button would be either This isn’t Stack Overflow, you don’t have to mark a thread as having a solution. I’m glad Barry’s post helped you, and popping a heart on it helps, but mainly, your followup post with “Thanks!!” is the indication. Anyone who comes by afterwards and reads through the thread will know that that’s what worked for you.

[barry-scott]

I guess its a config option that is not turned on.
On the Fedora discuss there is a solution button, we use it there all the time.

[hinxx (Hinko)]

No worries Barry. You really made my day 'cause I was already compiling python with debug symbols and could not get that mmap to work at all.

If I think about it is kind of strange that os.open() would be so different from open and its underlying fileno() when it comes to mmap()ing it. It is a file descriptor in both cases, right … doesn’t really matter.

[MegaIng (Cornelius Krupp)]

Well, the file descriptor needs to have both read and write access, which your original call to open doesn’t do.

Why the call with 'w+b' fails, I am not completly sure. I guess python does some other bookkeeping when opening a file in such a mode which requires seek which doesn’t work for this file descriptor.

I guess 'r+b' might work, since then python doesn’t try to create/clean out the file.

[Rosuav (Chris Angelico)]

@barry-scott

On the Fedora discuss there is a solution button, we use it there all the time.

Ahh, yeah, Discourse is extremely modular so I’m not surprised there’s different features on different instances of it.

[barry-scott]

@hinxx

You really made my day 'cause I was already compiling python with debug symbols and could not get that mmap to work at all.

Your friend in situations like this is to use strace to find out what the open call python did looks like exactly. Far easier than compiling for debug.

[kknechtel (Karl Knechtel)]

@MegaIng

Why the call with 'w+b' fails, I am not completly sure. I guess python does some other bookkeeping when opening a file in such a mode which requires seek which doesn’t work for this file descriptor.

If I’ve understood everything correctly, and my guesses are right: it’s not “bookkeeping”, but a security measure having unintended consequences.

The built-in open is aliased to io.open, which makes it easy enough to find the source. And here we see that the underlying call to the platform’s open (_wopen on Windows to specify a UTF-16 path) uses mode 0666, i.e., it denies execution privileges to the file. Makes sense; the basic purpose of the built-in open is to read and write data within a Python program, not to execute anything.

But the mmap call defaults to PROT_WRITE|PROT_READ protection on Linux, and on some architectures PROT_READ may imply PROT_EXEC (i.e., reading from a memory-mapped file requires execution permission! Maybe this has something to do with protecting against self-modifying code?), and thus we get EPERM since “The prot argument asks for PROT_EXEC but the mapped area belongs to a file on a filesystem that was mounted no-exec.”

[MegaIng (Cornelius Krupp)]

That would explain why an mmap call with that fileno still fails, but if I read the error message correctly, the initial io.open call itself fails with an error “unseekable”.

[hinxx (Hinko)]

Everybody: I really appreciate the enthusiasm to get to the bottom of this mystery !

For the record, my device is unseekable.

Here come more insights using strace as suggested by Barry:

Fail case with open:

## the script

$ cat open.py 
import mmap

fd = open('/dev/xdma1_user', 'wb')
mm = mmap.mmap(fd.fileno(), 4096, flags=mmap.MAP_SHARED, prot=mmap.PROT_READ|mmap.PROT_WRITE)
print('0:4', mm[0:4])

## run the script

$ strace -e abbrev=none -o open.txt python3 open.py 
Traceback (most recent call last):
  File "/home/dev/open.py", line 4, in <module>
    mm = mmap.mmap(fd.fileno(), 4096, flags=mmap.MAP_SHARED, prot=mmap.PROT_READ|mmap.PROT_WRITE)
PermissionError: [Errno 13] Permission denied

## locating mmap for the fd open.txt 

openat(AT_FDCWD, "/dev/xdma1_user", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3
newfstatat(3, "", {st_dev=makedev(0, 0x5), st_ino=807, st_mode=S_IFCHR|0666, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_rdev=makedev(0x1ff, 0), st_atime=1714133103 /* 2024-04-26T14:05:03.330908530+0200 */, st_atime_nsec=330908530, st_mtime=1714133103 /* 2024-04-26T14:05:03.330908530+0200 */, st_mtime_nsec=330908530, st_ctime=1714133103 /* 2024-04-26T14:05:03.330908530+0200 */, st_ctime_nsec=330908530}, AT_EMPTY_PATH) = 0
ioctl(3, TCGETS, 0x7fff187eb400)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
newfstatat(3, "", {st_dev=makedev(0, 0x5), st_ino=807, st_mode=S_IFCHR|0666, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_rdev=makedev(0x1ff, 0), st_atime=1714133103 /* 2024-04-26T14:05:03.330908530+0200 */, st_atime_nsec=330908530, st_mtime=1714133103 /* 2024-04-26T14:05:03.330908530+0200 */, st_mtime_nsec=330908530, st_ctime=1714133103 /* 2024-04-26T14:05:03.330908530+0200 */, st_ctime_nsec=330908530}, AT_EMPTY_PATH) = 0
fcntl(3, F_DUPFD_CLOEXEC, 0)            = 4
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = -1 EACCES (Permission denied)
close(4)                                = 0

Success case with os.open:

## the script

$ cat osopen.py 
import mmap
import os

fd = os.open('/dev/xdma1_user', os.O_RDWR)
mm = mmap.mmap(fd, 4096, flags=mmap.MAP_SHARED, prot=mmap.PROT_READ|mmap.PROT_WRITE)
print('0:4', mm[0:4])

## run the script

$ strace -e abbrev=none -o osopen.txt python3 osopen.py 
0:4 b'\x01\x01\x00\x00'

## locating mmap for the fd in osopen.txt

openat(AT_FDCWD, "/dev/xdma1_user", O_RDWR|O_CLOEXEC) = 3
newfstatat(3, "", {st_dev=makedev(0, 0x5), st_ino=807, st_mode=S_IFCHR|0666, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_rdev=makedev(0x1ff, 0), st_atime=1714133103 /* 2024-04-26T14:05:03.330908530+0200 */, st_atime_nsec=330908530, st_mtime=1714133103 /* 2024-04-26T14:05:03.330908530+0200 */, st_mtime_nsec=330908530, st_ctime=1714133103 /* 2024-04-26T14:05:03.330908530+0200 */, st_ctime_nsec=330908530}, AT_EMPTY_PATH) = 0
fcntl(3, F_DUPFD_CLOEXEC, 0)            = 4
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7fd8a9f04000
write(1, "0:4 b'\\x01\\x01\\x00\\x00'\n", 24) = 24

The difference I can spot between the two cases is a) arguments for the openat() and b) the attempt to seek in case of open while os.open does not try to seek.

FWIW, the error PermissionError: [Errno 13] Permission denied would IMHO correspond to EACCES and not to EPERM. The Linux mmap(2) man page is quite loose in explaining what would result in EACCES, though.

And finally, this was done with Python 3.10.12.

[MegaIng (Cornelius Krupp)]

Could you try io.open with r+b?

[hinxx (Hinko)]

Sure:

$ cat ioopen.py 
import mmap
import io

fd = io.open('/dev/xdma1_user', 'r+b')
mm = mmap.mmap(fd.fileno(), 4096, flags=mmap.MAP_SHARED, prot=mmap.PROT_READ|mmap.PROT_WRITE)
print('0:4', mm[0:4])

$ strace -e abbrev=none -o ioopen.txt python3 ioopen.py 
Traceback (most recent call last):
  File "/home/dev/ioopen.py", line 4, in <module>
    fd = io.open('/dev/xdma1_user', 'r+b')
io.UnsupportedOperation: File or stream is not seekable.