@petercordia yes but don’t want the script code put into a function. One very good reason is that all the HTML would then get that extra indent added to it and there is no way to take that out again without causing more problems..
Only if you put it there.
That is a very beefy machine, I was under the assumption that you were trying to run this on something like an old raspberry pi…
My suggested textwrap.indent() is the thing that would add extra indentation into all tripple-quoted multiline string literals.
It could be made to work if authors of the script files are disciplined enough to carefully wrap all of such literals in textwrap.dedent(), but it’s a serious flaw in my suggestion, and probably makes the whole idea unworkable.
@Stefan2 It seems that returning a value using try… except doesn’t actually have any speed overhead that I can see:
Constants.py
class ReturnException(Exception):
pass
Script1.py
from Constants import *
retVal = "Test"
Script2.py
from Constants import *
retVal = "Test"
raise ReturnException(retVal)
Test.py
from Constants import *
import time
f1 = open("Script1.py")
f1.close()
f2 = open("Script1.py")
f2.close()
start = time.time()
import Script1
print("Duration import: ", (time.time()-start)*1000) # 8 usec
start = time.time()
reload(Script1)
print("Duration reload: ", (time.time() - start) * 1000) # 8 usec
start = time.time()
local_dict = {"retVal": None}
exec(open("Script1.py").read(), {}, local_dict)
retVal = local_dict["retVal"]
print("Duration Script1: ", (time.time()-start)*1000) # 2 usec
print( f"{retVal=}")
start = time.time()
try:
exec(open("Script2.py").read())
except ReturnException as exp:
retVal = exp.args[0]
print("Duration Script2: ", (time.time()-start)*1000) # 2 usec
print( f"{retVal=}")
If speed is a concern, you definitely shouldn’t be using exec. But you seem determined.
@nedbat As you can see from the above code (sorry, now edited to include speeds) import takes much longer than exec as does reload.
If you do things the normal Python way, you will import each module only once. The module will define a function, you will call the function many times. Calling that function will be much much faster than exec’ing the module every time you need it.
There really are established patterns for how to write Python programs, and those ways work. Please back up and learn how to do it, you will be much happier.
We’ve been at this for two days, and you still don’t have an answer to your original question, because we’re going further and further down the wrong paths.
page.py:
from textwrap import dedent
import Authentication
def render(parent_object):
html = dedent("""\
<!DOCTYPE html>
<html lang="en">
""")
oAuth = Authentication()
authname = oAuth.Authenticate()
html += dedent(f"""\
<head>
<meta charset="UTF-8">
<title>title = Index.htm</title>
</head>
<body>
my test in Index.htm
parent object: {parent_object}
authname: {authname}
</body>
</html>
""")
return html
caller.py:
import page # This will be instantaneous after the first time.
parent_obj = Something() # Not sure what you want here.
html = page.render(parent_obj)
# Do something with html
We do not have a full picture of the code you are using. If you refactor your code as suggested, does it work quicker with imports?
Every time I ask for help, I have to refactor my code to take advantage of the time PPLs are taking to help me.
@nedbat Ok so here is my attempt to do it the Pythonic way, albeit not fully working as fails at the import line:
import time, pathlib, textwrap, importlib
script = "Page.py"
modified_script = "Modified" + script
orig_source = pathlib.Path(script).read_text()
fn_source = 'def render(parent_object):\n' + textwrap.indent(orig_source, " ")
f = open(modified_script, 'w')
f.write(fn_source)
f.close()
# import (modified_script) <- doesn't work, corrected below
modified_script = modified_script.replace(".py", "")
importlib.import_module(modified_script)
start = time.time()
parent_object = None # Something() # Not sure what you want here.
html = (modified_script).render(parent_object)
html = textwrap.dedent(html)
print("Duration page: ", (time.time()-start)*1000, "usec") # 500usec
print(f"{html=}")
Edit:
import line corrected
Thanks for trying it out. I’m suggesting that you manually write your files as I showed, not that you try to massage them programmatically.
To be frank, your problem is that you have decided that you must have script files that run code when executed or imported. This pattern you’ve decided on is fundamentally broken, and you’re not going to be able to get much help if you continue to insist on it. Any potential solutions are going to be far, far more complex than if you actually just use a standardized structure.
You keep saying that you’re trying to keep your scripts simple, but in reality you’re forcing an incredible amount of unnecessary complexity on your project. Open your mind a little and you’ll find that there are a multitude of better ways to do what you’re trying to do.
@nedbat I can see why you suggest importing and running each page as a function but the script files need to be kept clean and so the process of adding the function and indentation has to happen programmatically. There are many more modifications which happen between the original script file and the modified (compiled) version than is shown in this basic example.
I can check if the page is already loaded by:
if modified_script.replace(".py", "") not in sys.modules:
However, what I am having trouble visualising at the moment is how I can save the modified file as a Page.pyc so that import automatically imports the compiled and modified version of the page. At the moment I am using an importlib.util.spec_from_file_location PatchedLoader so I need to find an alternative way of doing this. I might be wrong but I don’t think the PatchedLoader writes a .pyc file until it actually runs spec.loader.exec_module(module).
Edit:
dynamic import now solved and previous post edited to show the fix. Next problem is how to refer to the method in the dynamically imported module, code below doesn’t work:
html = (modified_script).render(parent_object)
You really need to explain this requirement in more detail. This single requirement is the source of your problems here. Why is it completely out of the question for you to define a function in the script file?
I can guarantee that whatever sort of metaprogramming you’re trying to do, there’s a better way to handle it than by manipulating the source of script files at runtime. (Most likely, the better way is to use a dedicated web framework that has already solved most of these engineering problems for you.)
Because we want the basic script files to be editable in an HTML editor.
Which is precisely what we don’t want to do; hence my previous comments around trying not to get distracted by this type of discussion.
Either the files are pure HTML, in which case put them in .html files and read the text at runtime, or they are a bit more than HTML (as you have shown, and as you asked about in the very first question), in which case they won’t be editable in an HTML editor.
The more we ask, honestly, the less sense it makes.
You can use an AST transformation instead:
import ast
script = """
return '''
abc
'''
"""
result = {}
exec(
compile(
ast.fix_missing_locations(
ast.Module(
body=[
ast.FunctionDef(
name='_render',
args=ast.arguments(
posonlyargs=[],
args=[],
vararg=None,
kwonlyargs=[],
kw_defaults=[],
kwarg=None,
defaults=[],
),
body=ast.parse(script).body,
decorator_list=[],
)
],
type_ignores=[],
)
),
'<script name>',
'exec',
),
result,
)
print(result['_render']())
abc
importlib.import_module returns the module:
mod = importlib.import_module(…)
mod.render()
However, if you’re insisting on preprocessing the files, there’s no reason to write them to disk just to import them.
Which is why in the beginning I tried to dissuade questions about why and just focus on the possible solutions.
Now it turns out that further to your your suggestion of using indent/import/dedent; rather than being a complete change to the way I am currently doing things that could actually be a good enhancement to the existing version, potentially speeding-up subsequent page requests.
Just need to find a way of getting Python to import from a dynamically named script and run the method within that dynamically named module.
@BenjyWiener Very interesting, many thanks for that.
The pre-compile processing is an important part of what we are trying to do and potentially rather slow so the complied modified files do need to be written to .pyc files and run from there.
However, having said that, there may also be a way of putting that modified script code within a function so then importing once and running _render function on each hit. Just need to wrap my head around how/where I can use that, extremely interesting, AST modification prior to the .pyc getting saved.