File (csv) written by Pandas is mysteriously invisible in its directory

I have posted a complete description with copies of code and screenshots here: https://stackoverflow.com/questions/60426506/file-is-not-listed-in-directory-but-can-be-written-and-loaded-from-there-with-p (The responses there show that the respondents don’t know Python.)

The csv file is being written with this command:

DataPA.to_csv(“c:\\YelpData\\Data_PA.csv”)

I can read this file from the same Python program, and print (correclty) the first row with:

import pandas as pd
city_data = pd.read_csv(“c:\\YelpData\\Data_PA.csv”, usecols=range(1,11), encoding=‘latin-1’)
print(city_data.iloc[0])

The file is not present in c:\YelpData (from Windows). It is not a hidden file.

In doing a lot of testing scenarios I have determined the following:

  1. I can’t write the csv file to a non-existent directory. (Expected.)
  2. I can write the csv file to an existing directory, then delete that directory, and then the Python program (a new instance) continues to be able to read the file from the deleted directory. (!!!) (Not expected.)
  3. I can add a file to the directory, with the same name, with 0 size (Data_PA.csv), and Python continues to read the file it previously wrote (to somewhere) (!!!) (Not expected.)

I uninstalled Python 3.6.2 and installed Anaconda. I get the same results.
Windows 10.

I have never set up any virtual environments on this laptop, and I have never worked on Jupyter notebooks.

Thanks for any help.

2 Likes

Strange. Things that come to my mind:
a) Have you tried using single forward slashes instead?
b) Do you have any other process running that could have this file open all this time (office/browser/another python script)?
c) I’m not sure - but aren’t you by chance creating and then reading this file in the same script?

1 Like

I started with singles slashes. I changed it based on a response from stackoverflow. Same results.

There shouldn’t be another process that has the file open.

I have a different script reading the file. I manually start this script after the first one exits (the write). It sees the invisible file and prints the first part of it. The steps I described above, where I delete directories and add 0-byte files applies.

1 Like

The culprit was Comodo Firewall. At some point Auto-Containment had gotten turned on. My shadowed files were in a mirrored directory structure under c:/VTROOT/… Along with a lot of other stuff I don’t want in there. I have disabled Auto-Containment. Google shows many people have ran into this.

I know when I installed Comodo over 1.5 years ago that I didn’t set that, as I didn’t want it. Well, now I know how it works.

2 Likes