Sys.path.append .. libdir multiple level causes error in PyCharm

Python Help
1 
Hi forum,

The following snippets of two cases runs on system console terminal.

The ver 1 runs in PyCharm. I do not know why ver 2 reports error in PyCharm.

Thanks



# ver 1. ok
# myapp.py:
import sys
sys.path.append('..')
from libdir import mylib
mylib.hello()


# ver 2. PyCharm: ModuleNotFoundError: No module named 'mylib'
# myapp.py:
import sys
sys.path.append('../libdir')
import mylib  # line 10: Error in PyCharm
mylib.hello()


# Run on System Terminal:

$ pwd
/Users/ljh/Documents/tmp3/appdir
$ 
$ python3 myapp.py
hello
$ 


# Run in PyCharm:

/Users/ljh/Documents/tmp3/venv/bin/python /Users/ljh/Documents/tmp3/appdir/myapp.py
Traceback (most recent call last):
  File "/Users/ljh/Documents/tmp3/appdir/myapp.py", line 10, in <module>
    import mylib
ModuleNotFoundError: No module named 'mylib'

Process finished with exit code 1


# PyCharm Python version:

import sys; print('Python %s on %s' % (sys.version, sys.platform))
Python 3.8.9 (default, Oct 26 2021, 07:25:54) 
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
2

Btw. there is no __init__.py under ./libdir with mylib.py. But when I run it on system console, it does not report error.

$ pwd
/Users/ljh/Documents/tmp
$
$ find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g'
.
|____appdir
| |____myapp.py
|____libdir
| |____mylib.py
$
3

Relative paths are calculated based on the cwd to run the script. So basically, avoid relative paths in your script, because you can’t predict where the script is to be run and the behavior will become highly unstable.

PyCharm doesn’t work because it isn’t in directory /Users/ljh/Documents/tmp3/appdir. To find out where it is you can print os.getcwd() at the start of your script.

1 Like
4

Thanks Frost,

Can I use chdir to change the cwd.

Does the import statement command happens before top level code. Is there a way that I can change the cwd for PyCharm?

Can I set the working directory to path of the top level script file in PyCharm?

Thanks

5

os.chdir() works, but as I said, I do not recommend doing so. Just change the relative path to absolute path:

import os
# Add the parent directory to the sys.path
sys.path.append(os.path.dirname(os.path.dirname(__file__)))
1 Like
6

Thanks Frost,

I tried running the script on command line with or without path part:

$  python3  myapp.py
$  python3  ~/../ljh/./Documents/helloPy/appdir/myapp.py

os.path.dirname(os.path.dirname(__file__)) is empty if run it without path.

os.path.dirname(os.path.realpath(sys.path[0])) works on both situation.

Thanks

7

Now I changed my little log snippet like this without using relative paths with your help. It works in PyCharm and on system console terminal.

Thanks

$ python3 appdir/myapp.py
$
$ python3 ~/…/ljh/./Documents/helloPy/src/appdir/myapp.py
$
$ python3 myapp.py # under current dir

$ pwd
/Users/ljh/Documents/helloPy/src
$ ls
appdir	libdir
$ find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g'
.
|____appdir
| |____myapp.log
| |____myapp.py
|____libdir
| |____mylib.py
$
$


# myapp.py:

import os
import sys
import logging
from logging.handlers import RotatingFileHandler

sys.path.append(os.path.dirname(os.path.realpath(sys.path[0])))
from libdir import mylib  # noqa

def log_config():
    filename = os.path.realpath(sys.path[0]) + '/myapp.log'
    size = 10 * 1024 * 1024
    count = 10
    fmt = "%(asctime)s %(levelname)s %(pathname)s:%(lineno)d:%(funcName)s: %(message)s"
    logging.basicConfig(handlers=[RotatingFileHandler(filename, maxBytes=size, backupCount=count)],
                        level=logging.DEBUG, format=fmt)
    logging.info('%s', __name__)

def hello():
    logging.info('%s', __name__)

def main():
    log_config()
    hello()
    mylib.hello()

if __name__ == '__main__':
    main()


# mylib.py:

import logging

def hello():
    logging.info('%s', __name__)


# test:

$ python3 appdir/myapp.py
$
$ python3 ~/../ljh/./Documents/helloPy/src/appdir/myapp.py
$ 
$ python3 myapp.py  # under current dir
$


# log:

2022-03-05 15:38:02,163 INFO /Users/ljh/../ljh/./Documents/helloPy/src/appdir/myapp.py:17:log_config: __main__
2022-03-05 15:38:02,163 INFO /Users/ljh/../ljh/./Documents/helloPy/src/appdir/myapp.py:21:hello: __main__
2022-03-05 15:38:02,163 INFO /Users/ljh/Documents/helloPy/src/libdir/mylib.py:5:hello: libdir.mylib

2022-03-05 15:38:19,717 INFO myapp.py:17:log_config: __main__
2022-03-05 15:38:19,717 INFO myapp.py:21:hello: __main__
2022-03-05 15:38:19,717 INFO /Users/ljh/Documents/helloPy/src/libdir/mylib.py:5:hello: libdir.mylib
8

I would suggest not to rely on sys.path[0] nor sys.argv[0], both may vary depending on how you pack your project.

Just stick with os.path.dirname(os.path.abspath(__file__)) for the directory the script resides.

9

Thanks Frost,

I wanted to write log file and store log in the directory related to the entry __main__.py file. My log function is not in the main.py but in mylog.py. So I want to in mylog.py, get the path of main.py:

../appdir/__main__.py
../libdir/mylog.py
../log/myapp.log.1

What do you think about I do it like the following.

Thanks

# -*- coding: utf-8 -*-
# mylog.py

"""
Log to stdout and file with rotating.

    import logging
    import mylog

    mylog.config()
    logging.info('hello')
"""

import os
import sys
import logging
from logging.handlers import RotatingFileHandler


def config():
    """ Call config() once in top-level to log. """

    filename = '../log/myapp.log'
    size = 10 * 1024 * 1024  # 10M
    count = 10
    level = logging.DEBUG
    fmt = "%(asctime)s %(levelname)s %(pathname)s:%(lineno)d:%(funcName)s: %(message)s"
    path = os.path.realpath(sys.path[0])
    filename = os.path.join(path, filename)
    file_handler = RotatingFileHandler(filename, maxBytes=size, backupCount=count)
    console_handler = logging.StreamHandler(sys.stdout)
    handlers = [file_handler, console_handler]
    logging.basicConfig(handlers=handlers, level=level, format=fmt)

    logging.info('%s', __name__)