The 2 code review is from a spaghetti bash script that I use a lot, but I can’t show it to you because there are some sensitive commands in there.
Before I start to convert it to Python WITHOUT the help of an AI, do you have any suggestions?
1 Like
Write some kind of test to make sure the python script does the same thing as the bash script. Usually good practice that quickens the development process.
2 Likes
I convert large bash to python all the time.
With python you can uses classes to help organise the scripts logic.
I would not convert line for line. Rather think about the design of the script
and use the power of python to create a simpler, easier to maintain script.
Unless the script is small I tend to start with:
#!/usr/bin/env python
import sys
class App:
def __init__(self):
pass
def main(self, argv):
return 0
if __name__ == '__main__':
sys.exit( App().main( sys.argv ) )
I find using self to store state easier then lots of globals.
shutils, pathlib, subprocess.run and logging module are often what I end up using.
I see people use subprocess command lines thae pipe output into grep/awk etc.
This is no needed with python, just grab the command output and process the text your python code.
1 Like
Not really comfortable using classes, maybe I should start now.
Unit Test?
Suggestion Design in your debug logging so that it makes fixing issues easy.
I tend to have a debug(msg) function in my App class that only logs if i run the script with —debug.
For complex code i have multiple debug functions so that in can focus on sections of code that is having problems.
I’m interested, can you explain more?
Aren’t classes only useful when you manipulate a lot of variables?
Classes are useful if you have to maintain state (remember stuff). I’d suggest starting with functions. It’s easier to write, easier to test and often more clear.
Examples
# greet.bash
function greet(){
echo "Hello world!"
}
greet
Approx. conversion to Python:
# greet.py
def greet():
return "Hello world!"
print(greet)
After writing some simple functions, you could write unit tests - tests to verify your functions do what you want.
A very basic unit test:
# test_greet.py
def test_greet_basic():
actual = greet()
expected = "Hello world!"
assert actual == expected
Then simply call the test functions, e.g. test_greet_basic(), and eventually work up to using pytest to run them (recommended). If the function passes, convert the next bash feature to a python function and repeat.
There are many tutorials online on testing with python. Tests aren’t required, but will likely save you headache in the future.
I left out details in the latter examples so as not to confuse you (procedures vs. functions, side effects vs returns, etc.), but hopefully it’s enough to get a broad idea of how to proceed.
1 Like
Should I separate the unit test from my script?
There’s no harm in putting your unit tests in the same script – for larger projects, you want unit tests separate, but you probably want more than one file anyway.
Since you are starting out, I"d keep it simple. as @pylang suggests:
one by one, write functions that do individual logic. for each function write one (or more) unit tests, in teh same file.
if you call your test functions test_something then pytest will find them.
However: if you are converting a bash sciript, then may not be a whole lot of self contained logic that fits well into functions. That’s OK. (if you defined function in bash, then do the same in Python).
It’s OK to have a top-to-bottom script – jsut start at teh top and make sure it works at eash step as you move on.
Other notes:
many (most?) bash scripts call out to system utilities to do “the real work” – that’s a bit more awkward in Python (you can do it with subprocess) – so do it if the system utility is doing something complex, but if not, just write that utility in Python. You really don’t want to use a subprocess for things like printing to the screen, jsut use print().
If you want to write to stderr:
import sys
sys.stderr.write("something")
1 Like
Can you explain this in more detail and provide an example code block?
I was just using echo as an example, but my intention was to change it to an actual command, sorry for the confusion.
I’d suggest you use a framework to give you help with argument parsing and help texts. There are several to choose from: I prefer typer because I use type annotations everywhere but click and fire are other options.
With any of these frameworks you end up exposing one or more functions, or methods of a class that can be invoked by running the script (or as an entry point but you might want to stick with a simple script for now) and the command line arguments and options get validated and converted into function arguments. It saves on a whole lot of boilerplate code that you’d get using something like ArgParse.
1 Like
This is really hard to define in a concise manner-- it’s pretty key to programing when to use functions and when not to.
The really obvious one is if you need the same more-than-one-tiny-line of code in more than one place. But it’s often a good idea even if the function will only be called once.
What is your (short term) goal here?
If it’s to learn to program in general, then these kinds of questions are the right once to be thinking about.
If it’s to get a bash script translated to Python, (and presumably, earn some Python while you are at it) – then I’d do a more or less one-for one translation (depends a bit on how well designed the bash script is…) So: if there’s.function in the bash script, use a function in Python, etc.
You’ve got plenty of time to learn better code structure 
The thing is, the bash script structure is really terrible. Is it okay to do just functional programming like top to bottom is just functions?
Optional:
Should I use Cheatsheet or Documentation? I like cheat sheet because it is straight to the point.
Thank you for your time and insight!
[quote=“Christopher H.Barker, PhD, post:14, topic:26372,
username:PythonCHB”]
If it’s to get a bash script translated to Python, (and presumably, earn some Python while you are at it) – then I’d do a more or less one-for one translation (depends a bit on how well designed the bash script is…) So: if there’s.function in the bash script, use a function in Python, etc.You’ve got plenty of time to learn better code structure
[/quote]The thing is, the bash script structure is really terrible.
This is sometimes the case. A Bourne shell script (of which bash is a
variety) doesn’t lend itself to nicely structured stuff as easily as
Python, but is far more direct for invoking and connection other
programmes.
You can do functions and local variables etc in shell, but there’s a
degree of difficulty; what language we use depends on the task at hand.
I write plenty of shell scripts, but at some point an increase in
complexity pushes me to rewrite in Python.
Also, the structure of a script is very dependent on th author, and
whether they were doing something quick and dirty, or something of some
complexity with warrants better structure.
Is it okay to do just functional programming like top to bottom is just
functions?
Aside: “functional programming” isn’t this. It’s a variety of coding
where there’s almost no overt state i.e. variables. You do define the
code in terms of functions though.
Both the shell and Python are almost alwas used in “procedural
programming” (here a “procedure” tends to map to a function).
Anyway, that is terminology.
If the structure of the script is poor, it is often worth rewriting it
when you move to another language eg Python. Take the opportunity to
make a clean implementation, since you’re rewriting it all anyway.
Optional:
Should I use Cheatsheet or Documentation? I like cheat sheet because it is straight to the point.
Cheat sheets are ok as idioms for small things: how to sum some numbers,
etc - problems of that size. They’re not great for larger problems.
Though we tend to solve large problems by breaking them into smaller
problems.
Documentation is essential: when you’re calling something from, for
example, the Python standard library the only way to call it correctly
is to consult the documentation to find out how to use it.
You may recall that I mentioned that the first subprocess/echo call in
your v0.0 code was nonsense? That kind of thing requires consulting the
documentation in order to find out how to call it correctly.
Try to define to yourself the various tasks the shell script performs.
Write Python functions for each such task. Connect them together by
calling the functions as appropriate.
Cheers,
Cameron Simpson cs@cskk.id.au
1 Like
Yeah, that’s what we’re saying. Try to jam as much bash code converted to python functions as you can, top down. Bash scripts are notoriously all over the place, but you get a chance to clean some (if not all) of that up with simple functions. You’ll get more benefits later from this approach, so start there.
Should I use Cheatsheet or Documentation? I like cheat sheet because it is straight to the point.
What is “Cheatsheet” and what do you mean when you say “Documentation”?
1 Like
I think I know where this is going so I will try not to be lazy and read the documentation because it is newbie friendly and I will try to read more in general.
I’m really sorry if I don’t sound very nice. But yes, I’m convinced I should try documentation.
Thank you for your time and insight!
No that’s not what I mean. I genuinely don’t know what “Cheatsheet” you are referring to. And “Documentation” can also refer to documenting your own code such as docstrings, which can be helpful and self-documenting. Example: this a standard practice in the Scipy world. I was seeking clarification.
1 Like