Ability To Modify Code During Runtime (Feature from Microsoft VBA Language)
Proposal:
Hi,
Microsoft VBA programming language has a fantastic feature which is that you can pause the code during runtime, make an edit, and resume the running with the new modifications without having to restart the program from all over again. This saves hours!
Imagine this situation: Your code has been running for an hour then you discover an upcoming line that you’d like to modify to avoid an error. You pause the code, fix it, and resume the program. Currently you have to rerun the whole code from all over again for the tinest modification, which is very annoying.
This is somehow similar to Jupyter notebook concept but without having to live with the drawbacks of using a Jupyter notebook. Can python script somehow treat each line behind the scenes as if it is a seperate jupyter notebook cell?
Also tools, like pycharm have debugging support included.
Sometimes, it is helpful to call your script with the -i option, like python -i myscript.py
In this case, the interactive mode of python (pyrepl) is started after the script ended (either by exception or in the usual way).
Thanks appreciate the help but this is slightly different than what I was talking about. I’m not talking about debugging. I’m talking about pausing the program, changing or even moving a line of upcoming code and resuming the run with the new code changes.
Perhaps the pictures will make it clealer. This is a Excel VBA program that is paused. See the Yellow Arrow on the left? This shows you the next line that will be processed. The cool thing is that is you can move it up or down to show it which line to resume from. So if I wanted to take Debug.Print (“Done”) out of the loop I can pause the program, move it out of the loop and resume without having to restart.
In python if you move that line you must restart the program from all over again so if takes an hour to reach that point you have to waste another hour to reach it again. You can’t move parts of the code during runtime and you can’t tell it which line to process next whether it is a previous line or a next line. You don’t have that ‘Yellow Arrow’ to move up or down.
I think you were understood, but that debugging support is the closest you will ever be able to get to this in Python.
The language is very different from VBA, and there are two issues which make this more or less a non-starter:
there is no universal way to suspend or pause a Python program
code during execution is no longer plain text, but is converted to in-memory objects
It sounds to me like the type of application you’re working on needs the ability to checkpoint data as you run your analyses, so that you can fix and resume as necessary. In the case of Python, I think that’s the most appropriate way to address this kind of usage (and I’ve had great success with this strategy in the past).
Yes Jupyter Notebook can do something similar but working with Jupyter Notebook is a pain. You lose most of the functionality of VS code and you spend enormous effort and time managing cells. On top of that you don’t want non technical users to run notebooks.
I don’t see why doing this like the VBA could not be done. If in the backend everyline is treated like a Jupyter Notebook cell and this can be incorporated in IDEs (like VSCode does with the debugger) it should work nicely.
Currenly I’m waisting endless hours testing a complex code 100 times for the smallest changes. If I’m unable to fix the code while debugging then the debugging process takes 10 times more time.
This may be technically doable by making this special mode effectively an REPL mode that reads from a script instead of the console so that every statement is read, compiled and executed separately in the same namespace.
This is the kind of proposal where one of the biggest hurdles is going to be technical feasibility. To have any chance of this proposal happening someone needs to at least create a Proof of Concept. And @OP don’t expect anyone else to do this for you.
“is converted to in-memory objects”…If that’s the case then how come Jupyter Notebook can do something somewhat similar to the VBA functionality that I mentioned? The main difference I see is that VBA is treating every line like a Juputer notebook cell without using the notebook ‘cell’ interface.
Setting a breakpoint involves editing source, so it’s not really comparable to pausing an execution which hasn’t been specially prepared to pause.
Are you proposing that every line of source be treated as though preceded by a breakpoint? I think that would be pretty bad behavior.
Jupyter execution is a rather specific environment to target. As far as Python is concerned, a “cell” is just some user input which will be fed to an interactive interpreter. Nothing happens until you run the cell, at which point your code is finally sent to Python to execute. Asking “Python to be like Jupyter” may be perfectly reasonable, but asking “why can’t it just be like Jupyter” is stacking up abstractions the wrong way around. There would be nothing trivial or simple about pushing this mode of behavior down into the lower, more fundamental layer of the software stack.
You may be better served by asking for advice on how to solve your problems in the help forum. Even if you were to present a more complete proposal, it would likely be years before something usable would be available in a CPython release. By the sound of it, you have some legitimate needs, but there are probably some enhancements which you can apply today if you are willing and interested in making changes to your workflow.
Thanks I appreciate your response. I’m not sure about how the backend works but this feature in VBA saved me thousands of hours during my career. I’m doing complex webscraping which is super unreliable so I need to tweak the and test the code like 100 times to get it to work and every time takes a lot of time for the slightest change. If wish this VBA feature was in Python but if it can’t be done then it is what it is. Thanks.
What do you mean you lose most of the functionality of VS Code? VS Code supports Jupyter notebooks with intellisense, autocomplete, formatting, etc. as it does normal python files. Regardless, if you don’t like to work with Jupyter because you like to share a simple script with others, you can use interactive code cells within normal python files. This way you can interactively execute code cell by cell when prototyping and other users can run the file as a normal python script.
This exists in VSCode via the Jump to cursor command. (And it works; I tested it just now. But it might have some limitations.) You can access it in the context menu or command palette window, or set up a custom keyboard shortcut.
If you find specific limitations you might have more luck reporting them directly to the debugpy project. PEP 768 might be the relevant topic to follow if you’re interested in the core Python part of this.
