Cannot attach memray to a process inside a k8s container

Python Help
1

Hello!

I have a k8s container (running with ctlptl) which runs a Gunicorn service:

root@myservice-jh6qs:/srv# top -e M -o -PID -c -b -n 1 -p `pgrep -d "," gunicorn`
top - 11:57:34 up 2 days, 21:49,  0 user,  load average: 1.27, 2.41, 2.80
Tasks:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.0 us,  1.7 sy,  0.0 ni, 92.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :  47838.8 total,   1163.6 free,  27019.1 used,  24894.8 buff/cache     
MiB Swap:  51200.0 total,  51199.0 free,      1.0 used.  20819.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     14 root      20   0  744.0m 245.4m  91.4m S   0.0   0.5   0:08.62 /opt/project-venv/bin/python /opt/project-venv/bin/gunicorn --bind 0.0.0.0:8080 --preload --timeou+
     26 root      20   0 2235.2m 178.1m  20.1m S   0.0   0.4   0:02.00 /opt/project-venv/bin/python /opt/project-venv/bin/gunicorn --bind 0.0.0.0:8080 --preload --timeou+

(gunicorn is working in sync mode, with only one “worker” so it’s easier to debug)

I’m trying to attach memray to it:

root@myservice-556f55d4db-jh6qs:/srv# memray attach --verbose 26
Debugger command line: gdb -batch -p 26 -nx -nw '-iex=set auto-solib-add off' '-ex=set $rtld_now=2' '-ex=set $libpath="/opt/project-venv/lib/python3.12/site-packages/memray/_inject.abi3.so"' '-ex=set $port=42965' -x=/opt/project-venv/lib/python3.12/site-packages/memray/commands/_attach.gdb
debugger return code: 1
debugger output:
Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Inappropriate ioctl for device.
$1 = "MEMRAY: Attached to process."
No shared libraries loaded at this time.
$2 = "MEMRAY: Checking if process is Python 3.7+."
/opt/project-venv/lib/python3.12/site-packages/memray/commands/_attach.gdb:12: Error in sourced command file:
No symbol table is loaded.  Use the "file" command.

Failed to attach a debugger to the process.
You most likely do not have permission to trace the process.

Tools are modern:

root@myservice-556f55d4db-jh6qs:/srv# memray --version
1.15.0
root@myservice-556f55d4db-jh6qs:/srv# python3 -V
Python 3.12.7

I have both gdb and lldb installed:

root@myservice-556f55d4db-jh6qs:/srv# lldb --version
lldb version 19.1.1
root@myservice-556f55d4db-jh6qs:/srv# gdb --version
GNU gdb (Ubuntu 15.1-1ubuntu2) 15.1

And memray is installed in the same environment than the running attached to the process (see the following and the path of gunicorn in the top above):

root@myservice-556f55d4db-jh6qs:/srv# which memray
/opt/project-venv/bin/memray

The file /etc/sysctl.d/10-ptrace.conf contains an 1 but I shouldn’t need that config as I’m running as root.

Any ideas on what could be going on? Thanks!!

2

This does not seem to be a python specific question?
You seem to need someone who knows about k8s containers.

3

Well… it’s a memray question (a tool for Python) with me trying to debug a Python program’s memory.

4

It is very likely that its the container security that is breaking things not an issue with the tool.

5

Barry is right. It’s related to a container security issue.
I managed to find this post useful since the issue was the debugger (gdb in this case) having an issue attaching to the PID: c - How to solve "ptrace operation not permitted" when trying to attach GDB to a process? - Stack Overflow

Add this to your container in the docker-compose file:

    cap_add: 
      - SYS_PTRACE

Or for docker run use:

docker run --cap-add=SYS_PTRACE
6

But is that capability really necessary if I’m root inside the container?

7

Yes. I am also running as root user.

8

Being root in a container does not mean you are all powerful.
The container runtime will limit what can be done in a container.
You have to tell the runtime what you want the container to be able to do.