Python Variables Persisting in a Python Script

Python Help
1

I have the following script and I am trying to understand the passing of values to a variable. number_rows in my script below.

Thanks

import csv
import json
import os
import time

print(‘LINE COUNT GET’)
number_rows = os.system(‘wc -w < IBM.csv’) ### This shows count

print(‘The number of rows…’)
print(number_rows) ### THIS SHOWS nothing

number_rows = os.system(‘wc -w < IBM.csv’)
exit()

2

You’re finding this confusing because you think your variable passing is
wrong, but it isn’t. It is just that os.system() returns None. Its
output goes to the output of your programme, so probably displayed on
your terminal.

What you want to do is use the subprocess.run function, much detail
here: subprocess — Subprocess management — Python 3.10.2 documentation

Untested example:

from subprocess import run

with open('IBM.csv') as f:
    cp = run(['wc', '-w'], capture_output=True, stdin=f)

number_rows = int(cp.stdout.strip())

Notes:

  • we open IBM.csv ourselves, directly, as the variable f
  • we specify the command as a list of strings instead of as a single
    shell command; run() has a shell=True mode but you should avoid
    it almost all the time - it has the same injections risks and quoting
    difficulties as hand writing SQL queries

So the above:

  • opens the CSV file
  • calls run() in capture mode with the CSV file as stdin
  • extracts the word count from the captured output

BTW, doing a “word count” on a CSV ile is not a very meaning idea
usually. Note that you can’t run wc -w against a CSV file, just that
the number is not normally useful.

Cheers,
Cameron Simpson cs@cskk.id.au

3

Thanks Cameron, that worked perfectly. So if I want to run a terminal command like the following which I want to use to strip the parens from an output file with a perl expression, is it a similar exercise of opening the file stripping the parens and closing?

$ perl -p -i -e s/[’()]//g /Users/sean/python/data/emps.dat <---- cmd line

input
(1621, ‘Mami’, ‘Nishi’)
(1625, ‘Yoshimi’, ‘Kato’)
(1702, ‘Martin’, ‘Gerard’)

Output
1621, ‘Mami’, ‘Nishi’
1625, ‘Yoshimi’, ‘Kato’
1702, ‘Martin’, ‘Gerard’

4

Thanks Cameron, that worked perfectly. So if I want to run a terminal
command like the following which I want to use to strip the parens from
an output file with a perl expression, is it a similar exercise of
opening the file stripping the parens and closing?

Yes and no.

First up, you’re better off doing this directly in Python anyway.

$ perl -p -i -e s/['()]//g /Users/sean/python/data/emps.dat <---- cmd line

Second. This uses the -i option, which does in-place editing of the
file.

1: Do you really need to do that?

2: If you’re doing in-place editing, is there any output? And the stdin=
thing will not work for in-place editing. (a) it opens the file read
only and (b) the perl command won’t be given the filename, so it
cannotrewrite the file.

Finally, the command above is where subprocess.run shines. If you were
doing this as a shell command (which i think your line above is, you
need to quote the shell-special characters. By using the argument list
approach this is simpler:

subprocess.run([ 'perl', '-p', '-i', '-e', "s/['()]g" ], ......)

Notice using double quotes to avoid any need to escape the embedded
single quote.

input
(1621, ‘Mami’, ‘Nishi’)

Shouldn’t the perl above strip the single quotes as well? Also, it would
strip brackets inside the string, which may not be what you want.

You’re better off writing something more targetted, though the
targetting depends on how much you’re sure of in the line format.

with open('/Users/sean/python/data/emps.dat') as f:
    for line in f:
        line = line.strip().lstrip('(').rstrip(')')
        print(line)

You can be more precise (and more verbose) if necessary. Notice there’s
no regexp gunk in the code above, too.

Cheers,
Cameron Simpson cs@cskk.id.au

5

Thanks Cameron, that worked perfectly and was able to use the following to remove the single quotes. Thanks for taking the time to kelp me out.

line = line.translate({ord("’"): None})

6

I suspect:

line = line.replace("'", "")

is faster. It certainly seems more readable to me.

If you’re parsing CSV data, removing the quotes is almost certainly a
bad thing to do. What if the names contain single quotes?

Have a look at the “csv” module - it is the standard way to read a CSV
file in Python.

Cheers,
Cameron Simpson cs@cskk.id.au

7

Hello Again Cameron. and thanks for taking the time to assist me.