FileNotFoundError: [Errno 2] No such file or directory although the file exists

[Arnibandyo (Arnab Banerjee)]

I am defining a logger and periodically exporting the content to a blob. I’ve been using this for some time with DBR 10.4/python 3.8 without any issue. However we’ve moved to DBR 15.2/python 3.11, and now the logger function is working as expected. But when I’m trying to read and export it it’s throwing the following error:

FileNotFoundError: [Errno 2] No such file or directory

I additionally tried to specify a log folder and use the logpath + filename to create the log and read it. Although the logs are created but iIt’s still throwing the same error. After getting the error if I run os.listdir(logpath) I don’t see the missing logfile in the list. But if I run %sh ls -als logpath I can see the ‘missing’ log file listed. If I run os.listdir(logpath) again, the missing logfiles appear in the list.

I’ve also tried the following in both environment:
(print(os.popen("id").read())) (print(os.popen(r"/Workspace/Users/arnab.banerjee@nielseniq.com/log").read()))

For both env, I’m getting uid=0(root) gid=0(root) groups=0(root) for the first command. But for the second command I’m getting blank in 3.8 env &
/bin/sh: 1: /logpath: Permission denied in 3.11 respectively.

I haven’t posted any of the codes as that is discouraged in this forum, but if required I can post the codes as well in a follow up comment. A detailed post on the same topic can be found here: [logging - log files are vanishing from the working directory when trying to read and export them in Python 3.11 - Stack Overflow]

Any help will be welcome on this issue. Thanks in advance!

[onePythonUser (Paul)]

Hi,

so you moved from Python v3.8 to v3.11. Note that when you move between different Python versions, all packages that you had (installed) in a previous version do not automatically migrate to the new version. They are two independent versions with their own unique PATHs installed on your system. If you want to continue using previously installed packages, you can do either of the following:

1. Re-install the packages to your new PATH via pip.
2. Append the PATH directory of where the required packages are currently located to the new Python v3.11 PATH directory.

import sys
sys.path.append(r'enter\package\path\here')

Here is a related (I think) post:

Module Imports Not Displaying

Hello! First time using the forum… I am a beginner with python and I recently started having an issue with modules not showing on Visual Studios. I keep getting the “ModuleNotFoundError: No module named ‘requests’”. As far as I am aware, I have the most up to date version of python and VS. I have tried so many video guides attempting to fix the problem but nothing has worked. When I check the command prompt it shows the modules are downloaded. Is there a specific file location for something that…

[Arnibandyo (Arnab Banerjee)]

Hi,

Since I’m running these in databricks cluster, all required packages are needed to be installed at every instance. So I’m re-installing every package with updated compatible version in this v3.11 run and not using any of the old packages. And I’m not missing any package from the v3.8 run. At least not the basic required ones.

Thanks!