For loop when importing from module (is it possible)

I’m importing some variables from another module. There’s a heap so thought rather than:

import params
if params.var1:
    var1 = params.var1
if params.var2:
    var2 = params.var2
etc

was thinking I’d do something like

import params
for p in ["var1","var2",...]:
    if params.p
        p = params.p

Obviously that isn’t going to work. But I’m not sure if it’s possible and if it is how to code it.

Note that this is more in the nature of “is this possible” and if so how. It’s not how I’m intending to do this in the final version.

In normal situations, the natural way to do this is just from params import var1, var2, var3 etc.

The more important question is why on Earth you want the conditional logic.

If you really are trying to check whether the values are “truthy”, then it should only make sense to assign all the variables regardless. You can check their values later, and the logic will be much simpler since you won’t have to worry about whether variables in your own code exist or not.

If the names might not actually exist in the other module, then the question is still why that happens. All you did was run the top-level code, and you didn’t give it any input; presumably that should have the same result every time you run the program, right? You know which module you’re importing, so presumably you should know whether it has those variables. It would already be weird for the code in that module to make decisions about whether or not to define something at all; and especially weird if that choice were not deterministic.

Even if you had to check for that, instead of trying to use separate names for the possibly-there values, it would make more sense to store them in a dictionary. That way, you can use the normal functionality of a dictionary to see if the key is there or not, instead of having to worry about inconsistent variable definitions.


But yes, it is possible. Instead of params.p, the magic you are looking for is getattr(params, p). Instead of p = ..., you would similarly need to use globals()[p] = ....

(I’m not going to refer you to the standard Stack Overflow Q&A about this, because that Q&A is a huge mess.)

1 Like

The above code will not work because p is being used as the for-loop element as well as being assigned a value within the loop.

import params as p will enable use of p.var1, p.var2, etc, without further coding.

Thanks, I’ll have a play with that.

The reason I’m using the conditional logic is that all of the variables are not required. In fact only two are required. But of course could just assign them a null value.

The other thing is that this is really a training exercise. I’m (re-)learning python and the way I find best is to try stuff. And in this case as you’ve pointed out it’s definitely not the best way but still worth finding out for myself if you get my meaning.

Thank you. I realised the code wouldn’t work but was just leaving it there as a pointer to what I was after. Thanks for the correction, I’ll test this.

Then why not just use from params import var1, var2 (with whatever are the two that actually are required)? Or else what exactly do you mean by “required”? It would really help to have proper context for the problem.

1 Like

Sorry, I’ll provide more info (was trying to keep it simple lol).

This is the help output. Basically it’s a program I use for testing our mail servers. The only requirement is a sender and recipient. But it’s possible to include other options.

It works perfectly using argparse but my next step was to have the ability to import the options from a file. It’s not really needed, this is more about me learning techniques and having a bit of fun doing it. The program does get used a lot but further mods aren’t really necessary.

usage: davos_emailer-v2.py [-h] [-F] [-r] [-f] [-fh] [-p] [-s] [-j] [-t] [-au] [-ap] [-l] [-T] [-E] [-v | -q]

Davo's emailer v2

options:
  -h, --help            show this help message and exit
  -F, --file_parameters
                        if set, will import parameters from file davos_emailer_param.py, parameters set in cmd line will take priority
  -r , --recipient      recipient address - REQUIRED if not imported from file
  -f , --from_envelope
                        from address - envelope - REQUIRED if not imported from file
  -fh , --from_header   from address - header (i.e. if not specified will be same as from-envelope
  -p , --port           SMTP port to use (default 25
  -s , --server         destination SMTP server (default sg1.zen.net.au)
  -j , --subject        message subject
  -t , --message_text   message text - single line
  -au , --auth_username
                        authentication username - optional, if not used from_envelope address will be used
  -ap , --auth_password
                        authentication password - optional, note if this is not specified no authentication will be used even if -au is set
  -l , --loop           continuously loop - variable is time between sending in seconds
  -T, --message_text_input
                        Input message text - multi-line - flag
  -E, --error_handling  Error handling - not normally recommended but will ensure it will keep looping even on an error - flag
  -v, --verbose         verbose diagnostics - flag - NB: -v or -q
  -q, --quiet           quiet - no output - flag - NB: -v or -q

A Python source code file is not a good way to store config information. Instead, consider a format such as JSON or TOML (if you need tree-structured data, or data that has types more interesting than ordinary numbers and strings) or INI (for simpler cases).

That was going to be my next version. I was just experimenting on what was possible in this version. But good idea and I’ll change the data type tomorrow when I’m working on it again.

What I like to do in cases like this is to define a dataclass or similar that represents all the options. All other configuration sources (environment variables, configparser files, and/or argparse) get converted to that.

I haven’t tried this yet myself, but (in the interest of satisfying any more curiosity about using imports) maybe have a look at the dir() and vars() builtins? I think you may be able to get a modules contents in the form a dictionary, which would make it easy to get your required and optional values.

And fwiw, neat looking script, based on the usage anyway. Sounds useful. :slightly_smiling_face:

1 Like

The HuggingFace transformers and datasets code also deals with similar design problems - I learned a lot from studying their code. Seems you have the general problem of how to merge your CLI with your code pretty well under control. But you might still get some more ideas, either for future work or for improving your current code, by looking at for instance https://github.com/huggingface/transformers/blob/main/src/transformers/commands/transformers_cli.py (and then looking at the subclasses).

In the past I’ve also combined this kind of code with dataclasses; those are really great for this kind of usage where you need to organize lots of cli options plus defaults And it’s then also pretty straightforward to de/serialize those from/to config files.

Really appreciating the dialog and advice here. I’m able to use the method I was working on here (using a module to store variables), but as pointed out by many it’s not a good approach.

I’m now re-working the program to store the options as a dictionary and will use a json file to store these. This will be my v2 with argparse not used.

I’ve worked out my code for importing from json. The logic is for priority of parameters (using smtp sending port as an example):

  1. defined as a command line parameter
  2. defined in parameters json file
  3. default value of 25

Thoughts on the code and if I could make it more efficient?

try:
    with open('emailer_params.json', 'r') as openfile:
        params = json.load(openfile)
except FileNotFoundError:
    print("parameters file does not exist")
    quit()

recipient = params.get("recipient")
from_header = params.get("from_header")
from_envelope = params.get("from_envelope")
port = params.get("port")
sending_server = params.get("sending_server")

#print(f"port is {port}, type ",type(port))

# define smtp sending port
if args.port:
    port = args.port
elif args.file_import and port:
    print(f"port is {port}")
else:
    port = 25
  • better logic I think:
if args.port:
    port = args.port
elif not (args.file_import and port):
    port = 25

also, the file import bit is simply from argsparse:

parser.add_argument("-F", "--file_import", action="store_true",
                    help="if set, will import parameters from json format file")

argparse allows you to set defaults, so you can set the default port in the parser: parser.add_argument("--port", type=int, default=25). Although this makes your current precedence rules difficult to implement, because you won’t know if they passed the value explicitly.

For your config, I’d suggest having the file as an argument (again with a default value), rather than a flag. That way you allow for multiple config files. This can be useful for testing, or just because you have different uses for the script.

In this situation I might eschew the default value entirely and require a config file–either explicitly or using the default file. Then the workflow is

  1. parse any options that were explicitly passed
  2. load the config file (either one that was passed, or the default). Error if not found
  3. for any option that wasn’t specified explicitly, use the value from the config file
  4. (optional) use the value from the original, default config if something is still missing (so you always load this one)

This maintains a lot of flexibility: you can multiple config files for different uses, and you can tweak any given config with a command-line option.

1 Like

Also keep in mind that the dict.get method can specify a custom default: port = params.get("port", 25).

I was thinking about that, but don’t think using the built-in defaults (via either the dict.get or via argparse (as suggested by @jamestwebber ) will give me the logic I’m after:

  1. argparse
  2. json file
  3. if neither of the above built in default

TBH there’s only 2 variables that this applies to (port and sending server) so probably not a big deal to keep it as it is. But thanks for the suggestions - always good to keep these alternatives in mind.

Excellent idea re allowing user specified file. And I really like your idea about having a default file with those variables in it that are required (port and sending server). I’ll make changes to go this way. Much easier to make changes down the track as well to the default options (if for example I change the default sending server).

For the future, another useful trick here is updating and merging dictionaries with the built-in functionality. There is the update method:

>>> x = {1: 'one', 2: 'wait, what comes after two?'}
>>> x.update({2: 'ah, right, two', 3: 'and three'})
>>> x
{1: 'one', 2: 'ah, right, two', 3: 'and three'}

As well as unpacking syntax to merge dictionaries into a new result:

>>> a = {1: 'one', 2: 'wait, what comes after two?'}
>>> b = {2: 'ah, right, two', 3: 'and three'}
>>> {**a, **b} # a new object, without modifying the other two
{1: 'one', 2: 'ah, right, two', 3: 'and three'}

As described in more detail on Stack Overflow:

I was trying to work out if I could use this to merge args and params to give me my combined data. Is this possible?

Any other suggestions to tidy my code? I’m happy with the result but always worth looking at better ways for future projects.

from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
import smtplib
import ssl
import argparse
from time import sleep
from datetime import datetime
import json

def send_mail():
    SSLcontext = ssl.create_default_context()
    if verbose: print(f"server: {sending_server}  port: {port}")
    with smtplib.SMTP(sending_server, port) as server:
        if verbose: server.set_debuglevel(1) # set diagnostics to verbose
        server.starttls(context=SSLcontext)
        if not auth_password:
            if verbose: print("no authentication")
        else:
            server.login(auth_username, auth_password)

        server.sendmail(from_envelope, recipient, text)
        if not quiet: print(f"{datetime.now():%Y-%m-%d %H:%M} - e-mail sent from {from_envelope} to {recipient} with subject: {subject}")

parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter, description="""Davo's emailer v2

This program is aimed at helping the user test mailservers. 
I've tried to include all scenarios but happy to add more options if requested (and it's possible).

Note that the parameters file is required to list sending_server and port if these are not defined
via the command line arguments (-s and -p).

The default parameters file is emailer_params.json and needs at a minimum the following:

{"port": 25,
"sending_server": "<mailserver to send to>"}""", epilog="""
This program is not for commercial modification or resale without permission from me.
Permission is granted for commercial or personal use.""")
parser.add_argument("-F", "--file_import", type=str, metavar="", default="emailer_params.json",
                    help="user defined parameters file otherwise emailer_params.json will be used, parameters set via cmd line will take priority")
parser.add_argument("-r", "--recipient", type=str, metavar="",
                    help="recipient address - REQUIRED if not imported from file")
parser.add_argument("-f", "--from_envelope", type=str, metavar="",
                    help="from address - envelope - REQUIRED if not imported from file")
parser.add_argument("-fh", "--from_header", type=str, metavar="",
                    help="from address - header (i.e. if not specified will be same as from-envelope")
parser.add_argument("-p", "--port", type=int, metavar="",
                    help="SMTP port to use (default set in params json)")
parser.add_argument("-s", "--sending_server", type=str, metavar="",
                    help="destination SMTP server (default set in params json)")
parser.add_argument("-j", "--subject", type=str, metavar="", help="message subject")
parser.add_argument("-t", "--message_text", type=str, metavar="", help="message text - single line")
parser.add_argument("-au", "--auth_username", type=str, metavar="",
                    help="authentication username - optional, if not used from_envelope address will be used")
parser.add_argument("-ap", "--auth_password", type=str, metavar="",
                    help="authentication password - optional, note if this is not specified no authentication will be used even if -au is set")
parser.add_argument("-l", "--loop", type=int, metavar="",
                    help="continuously loop - variable is time between sending in seconds")
parser.add_argument("-T", "--message_text_input", action="store_true",
                    help="Input message text - multi-line - flag")
parser.add_argument("-E", "--error_handling", action="store_true",
                    help="Error handling - not normally recommended but will ensure it will keep looping even on an error - flag")
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true",
                    help="verbose diagnostics - flag - NB: -v or -q")
group.add_argument("-q", "--quiet", action="store_true",
                    help="quiet - no output - flag - NB: -v or -q")
args = parser.parse_args()

file_import = args.file_import
print(f"parameters file used: {file_import}")

#import_file - either default or defined
try:
    with open(file_import, 'r') as openfile:
        params = json.load(openfile)
except FileNotFoundError:
    print("parameters file does not exist")
    quit()
except Exception as err:
    print(f"error in import - probably incorrectly defined variable. error code: {err}")
    quit()

# import data from either argpass (command line) or parameters file
# priority is given to argpass so if that is defined the value will not be imported from the parameters file

if args.recipient: recipient = args.recipient
else: recipient = params.get("recipient")

if args.from_header: from_header = args.from_header
else: from_header = params.get("from_header")

if args.from_envelope: from_envelope = args.from_envelope
else: from_envelope = params.get("from_envelope")

if args.port: port = args.port
else: port = params.get("port")

if args.sending_server: sending_server = args.sending_server
else: sending_server = params.get("sending_server")

if args.subject: subject = args.subject
else: subject = params.get("subject")

if args.message_text: message_text = args.message_text
else: message_text = params.get("message_text")

if args.auth_username: auth_username = args.auth_username
else: auth_username = params.get("auth_username")

if args.auth_password: auth_password = args.auth_password
else: auth_password = params.get("auth_password")

if args.loop: loop = args.loop
else: loop = params.get("loop")

if args.message_text_input: message_text_input = args.message_text_input
else: message_text_input = params.get("message_text_input")

if args.error_handling: error_handling = args.error_handling
else: error_handling = params.get("error_handling")

if args.verbose: verbose = args.verbose
else: verbose = params.get("verbose")

if args.quiet: quiet = args.quiet
else: quiet = params.get("quiet")

if not quiet: print(f"quiet: {quiet}, verbose: {verbose}")

# Logic checking of variables and rules in place
# quiet and verbose are mutually exclusive
if (quiet and verbose): raise ValueError("quiet and verbose cannot be set at the same time. check parameters file")

# recipient - must be defined by argument or parameters file
if not recipient: raise ValueError("recipient must be defined")

# note the from envelope is what is in the initial connection (i.e. MAIL FROM: from_envelope)
# note the from header is what shows in the e-mail client. This is often faked.
# if the from_header isn't specified we assume it's the same as the from_envelope

# define from_envelope - must be defined by argument or parameters file
if not from_envelope: raise ValueError("from_envelope must be defined")

# define from_header. if not defined by params or argument use from_envelope
if not from_header: from_header = from_envelope

# subject and message text need to be defined or there will be an error when sending
if not subject: subject = ""
if message_text:
    message = message_text
else:
    message = ""

# if auth_password specified authentication will be attempted
# in this case the auth_username will be used. If not specified the from_envelope address will be used as auth_username
if auth_password and not auth_username: auth_username = from_envelope

if verbose: print(f"recipient: {recipient}, from envelope: {from_envelope}, from header: {from_header}")
if verbose: print(f"sending server: {sending_server}, port: {port}")
if verbose: print(f"subject: {subject}")
if verbose: print(f"auth_username: {auth_username}, auth_password: {auth_password}")
if verbose: print(f"loop: {loop}")

# quit()

if message_text_input:
    print("Enter message, end with a blank lines (i.e. enter twice)")
    message = ""
    counter = 0
    while True:
        line = input()
        if line.strip() == '':
            counter += 1
            if counter == 2:
                break
        # \n is new line, %s is string formatting
        # str.strip() will remove any whitespace that is at start or end
        message += "%s\n" % line.strip()

if verbose: print(f"message: {message}, length: {len(message)}")

msg = MIMEMultipart()
msg['From']= from_header
msg['To']= recipient
msg['Subject']= subject

# removed - not implementing bcc cc due to issues and it's not really what I need
# if args.bcc: msg["Bcc"] = args.bcc
#if args.cc: msg["Cc"] = args.cc

if verbose: print("This is our message format: ", msg)
if verbose: print("This is the message envelope: ", "From: ", from_envelope, "To: ", recipient)

msg.attach(MIMEText(message, "plain"))
text = msg.as_string()

if verbose: print("This is the text field")
if verbose: print(text)

# if -l loop parameter set the send_mail function will run continuously
if loop:
    if error_handling:         #this will keep the script working if an error occurs
        if not quiet: print("Error handling active")
        while True:
            try:
                send_mail()
                if verbose: print(f"Waiting for {loop} seconds")
                sleep(loop)
            except Exception as err:
                print(f"""{datetime.now():%Y-%m-%d %H:%M} - An error occurred
    - for more detailed info on the error run with the -v (verbose) switch
    - the error message was
    {err}""")
                if verbose: print(f"Waiting for {loop} seconds")
                sleep(loop)
    else:
        while True:
            send_mail()
            if verbose: print(f"Waiting for {loop} seconds")
            sleep(loop)
else:
    send_mail()

quit()