Running Python Scripts at Startup and in Background Launchd macOS

Python Help
1

Hello there,

So I have a situation (trying to work through more general/hypothetical with applications to the ‘real-world version’. The situation let’s say is I have 3 different python scripts written by third parties I would like to employ on my Mac, having them ‘turn on’ at startup and sit in the background and wait for additional commands. To clarify this, I’m trying to say “launch script A” at startup, and do nothing until further commands are received by/given to script A.
In this hypothetical I’m referring to the 3 python scripts Bazarr, DAPS, and Kometa. To me, all 3 should perform this way of launching at start and waiting for further at the very least.

I’ll now try and illustrate how these 3 are launched/invoked/used.

Bazarr - this app has a (semi-usual) wiki on how to get it started with macOS. Specifically, here is the wiki page bazaar wiki detailing how to ‘Autostart on macOS’ according to this group/person. In essence, I create a launchd item with this plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>com.github.morpheus65535.bazarr</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/bin/python3.8</string>
      <string>/Applications/bazarr/bazarr.py</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardErrorPath</key>
    <string>/usr/local/var/log/bazarr.log</string>
    <key>StandardOutPath</key>
    <string>/usr/local/var/log/bazarr.log</string>
  </dict>
</plist>

which to my understanding runs the bazarr python script by calling this script against the local python exec the instructions have users install (3.8), which is set to RunAtLoad (to me meaning at startup) and to KeepAlive (to me meaning be ready in background and await further commands. This seems ok generally to me but if you see something odd please illustrate! My question for this one is in the part of the instructions that instruct to create an Automator app to run shell script (loading the plist into launchd) to have something to put in login items…? This doesn’t make sense to me. Isn’t a launchd item set to RunAtLoad essentially a login item? Why do I need something to ‘launch’ a launchd property on start if this is how things are ‘launched’ at start usually?

DAPS - this is basically one large (main) python script which calls a half dozen lesser python scripts with certain details/configuration settings (so they can distribute a suite of scripts that work by putting all personal configuration details into one script to edit). So in theory an either/or to me is: I have to create a launchd plist to call/run each of the lesser scripts in order to run them at scheduled times or I can create one launchd item to call/run the large (main) and keep it alive so that it can run individual scripts on their own individual schedule. I’ve been going with option 2 here i.e. one launchd item to get the main running at start and in background but I’m having trouble theoretically reconciling how this one is any different to the last one (Bazarr). Just trying to figure out/understand generally why 3 python scripts require 3 different launch methods (or so it seems)?
So for this one I have some scheduling info set in the main script and I launch this with a launchd plist of

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>com.nathan.dapsmain</string>
    <key>ProgramArguments</key>
    <array>
      <string>/Users/nathan/daps_other/myscripts/other/dapsmain.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardErrorPath</key>
    <string>/tmp/stdout.log</string>
    <key>StandardOutPath</key>
    <string>/tmp/stderr.log</string>
  </dict>
</plist>

Which calls my script ‘dapsmain’ which is

#!/bin/bash

cd /Users/Nathan/daps/
source .venv/bin/activate
python3 /Users/nathan/daps/main.py
deactivate

This gets into the whole virtual environment for python script thing that I don’t really understand at all. I kinda get the purpose of using a virtual env for a script (to protect the main os/file/folders from accidents). I kinda get that when manually activating/invoking/deactivating in a Terminal window is when a ‘deactivation’ is required. I’m guessing that’s only somewhat ‘required’ though especially when activating a virtual env for the script’s use in the background - right or wrong? Basically, I’m thinking/wondering if I could just remove the ‘deactivate’ from these sh scripts? Going back to the Bazarr example above, I’m wondering again why the Automator run shell login item?

Kometa - rather than further elongating this now I’d just like to say here Kometa is similar to DAPS in invocation but without the internal scheduling detailed above. So, when I want to run Kometa on schedule I have a launchd plist similar to the DAPS one above but with a CalendarInterval for each call to each function e.g.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>com.nathan.kometa</string>
    <key>ProgramArguments</key>
    <array>
      <string>/Users/nathan/daps_other/myscripts/other/kr.sh</string>
    </array>
    <key>RunAtLoad</key>
    <false/>
    <key>StartCalendarInterval</key>
    <array>
      <dict>
        <key>Hour</key>
        <integer>17</integer>
        <key>Minute</key>
        <integer>0</integer>
      </dict>
    </array>
    <key>StandardErrorPath</key>
    <string>/tmp/stdout.log</string>
    <key>StandardOutPath</key>
    <string>/tmp/stderr.log</string>
  </dict>
</plist>

But with Kometa I want to run say 3 different functions at 6 different times so I’ve found that I need 3 different launchd plists with their respective times? And the kr.sh script is very similar

#!/bin/bash

cd /Users/Nathan/Kometa/
source kometa-venv/bin/activate
python /Users/nathan/Kometa/kometa.py -r
deactivate

I know this is extra long but I’ve been reading/experimenting for a while now and it just feels like I need to bounce these specifics off of someone who can help me focus learn to be able to apply things more generally later. Thanks for reading!

<?xml version="1.0" encoding="UTF-8"?> Label com.nathan.dapsmain ProgramArguments /Users/nathan/daps_other/myscripts/other/dapsmain.sh RunAtLoad KeepAlive StandardErrorPath /tmp/stdout.log StandardOutPath /tmp/stderr.log Which calls my script 'dapsmain' which is #!/bin/bash

cd /Users/nathan/daps/
source .venv/bin/activate
python3 /Users/nathan/daps/main.py
deactivate
This gets into the whole virtual environment for python script thing previously discussed. Just to make sure/try here I believe you said it doesn’t need to be manually ‘deactivated’ in these cases, it’s only needed when invoking manually in Terminal correct? If so, I suppose I should remove this deactivate from my sh script? Going back to the Bazarr example above, I’m wondering again why the Automator run shell login item?

Kometa - rather than further elongating this now I’d just like to say here Kometa is similar to DAPS in invocation but without the internal scheduling detailed above. So, when I want to run Kometa on schedule I have a launchd plist similar to the DAPS one above but with a CalendarInterval for each call to each function e.g.

<?xml version="1.0" encoding="UTF-8"?> Label com.nathan.kometa ProgramArguments /Users/nathan/daps_other/myscripts/other/kr.sh RunAtLoad StartCalendarInterval Hour 17 Minute 0 StandardErrorPath /tmp/stdout.log StandardOutPath /tmp/stderr.log

But with Kometa I want to run say 3 different functions at 6 different times so I’ve found that I need 3 different launchd plists with their respective times? And the kr.sh script is very similar
#!/bin/bash

cd /Users/nathan/Kometa/
source kometa-venv/bin/activate
python /Users/nathan/Kometa/kometa.py -r
deactivate

I know this is extra long but I’ve been reading/experimenting for a while now and it just feels like I need to bounce these specifics off of someone who can help me focus learn to be able to apply things more generally later. Thanks for reading!

2

The activate is not required you can do this.

cd /Users/Nathan/daps/
.venv/bin/python3 ./main.py
3

There are system and user launchd items.
I think that the system ones run when you boot macos
and the users one can be set to run when you login.

I likely missed some of your questions, it is very long post.

4

Interesting. So with your simplified version here for the script it looks like it replaces the need to ‘activate’ and ‘deactivate’ both, is that the correct idea? Seems definitely easier in that respect!

5

User vs System. I guess that sounds logical but… For example, the Bazarr Automator action to launch the script via user kaunchd…? Am I interpreting that correctly? Why would one need a system launchd (agent vs daemon) as opposed to a user level?

I guess my outstanding questions for this round are 1. What’s the main advantage in picking a system level vs user level? 2. What’s the main difference between an Agent and a Daemon? I’m guessing these are basic Mac questions that I need to get a handle on to really grasp all these other tasks. I appreciate the help in this!

6

I was just doing some reading on this and it seems like for my purposes at least user vs system doesn’t matter to me. For simplicity sake I’d like to keep this to running the three scripts at startup on a user level. So all launchagents are stored in/loaded from Users/nathan/Library/LaunchAgents/

7

I’m going to try boiling my mess here down into a TL;DR. First question group was concerning the activation/deactivation of Python virtual environments - if I understand correctly (thank you!) your simplified version is more of an implicit activation call as opposed to my original more explicit?
Second question concerns this launchd stuff. System vs user, daemon vs agent, aside - what I need to understand is when/how a launchd xml that calls a sh script gets started and loaded. My understanding is that as long as the xml includes a RunAtLoad or a KeepAlive a user agent will start and load when user logs in…? If that is the case, why is there the Automator specific task for loading the xml part of the ‘Bazarr’ tutorial in op? Seems like the xml is being loaded twice…?
Thanks! Feel free to ignore the op otherwise!

8

I, personally, never use activate with venv, it’s only provided as a way to setup the PATH for easier interactive use. Just use the path to the python program inside the venv and Python will figure out it is in a venv and do the right thing.

9

See man launchd that I quote from

In the launchd lexicon, a daemon is, by definition, a system-wide service
of which there is one instance for all clients. An agent is a service that
runs on a per-user basis. Daemons should not attempt to display UI or
interact directly with a user’s login session. Any and all work that
involves interacting with a user should be done through agents.

daemon == system
agent == user

Also see man launchd.plist for info on what you must and optional can configure.

Your shell scripts are not doing much now that activate is gone.
You could set the current directory and from the python with an arg of the script directly. However I often use a shell wrapper so that I have a place to add debug statements.

No idea. I always use the launchctl command from a terminal.
See man launchctl.

10

This is excellent info, thank you!

11

I think I understand your meaning here. Do a WorkingDirectory in the xml and call the venv path exactly like in the wrapper now, that read right? Could you point me/teach me how to play with the “add debug statements” to the wrapper? I’m real curious what I could be doing there. Cool, thanks so much for your replies!

12

I have a script that using python and imap to auto-file my emails.
I was worried it was not being run by launchd so I added a log to the script.

#!/bin/bash
LOG=~/tmpdir/run-nightly.log
echo "Info: run-nightly $(date +%Y-%m-%d\ %H:%M:%S)" >>${LOG}
cd ~/Projects/ComputerConfiguration/Utilities/ImapTools
~/bin/tool-python3 imap_tool.py imaps://barry-mail@fender script nightly.imap-script 1>>${LOG} 2>&1

And this is ~/bin/tools-python3 that using by tools venv.

#!/bin/bash
# run inside of the tools venv
exec ~/.local/tools.venv/bin/python3 "$@"
13

Nice, appreciate that!
I wonder if you could partially explain the ‘instructions’ in one of these other scripts I’m looking at. I’m extra hesitant these days to just ‘run commands’ that some rando internet thing says to run lol. The script is not all that technical really, I’m just starting to wrap my head around scripting generally though. The script is

#!/usr/bin/env bash

# Original script by: Chris.DC
# Usage 1: Replace the paths to your preferences!
# Usage 2: alias logsearch="/volume1/scripts/log_search.sh"
# Usage 3: `logsearch "title"`
# Usage 4: if it does not work, chmod +x log_search.sh

## ADAPT THE FOLLOWING! (if required)
## How many lines to show before the match of your search term
B=${B:-3}
## How many lines to show after the match of your search term
A=${A:-3}
## Default search term is set to "test"
search_term=${1:-"test"}
## Default filepath is set to "/volume2/docker/daps/logs/poster_renamerr/poster_renamerr.log"
file_path=${2:-"/Users/nathan/daps/logs/poster_renamerr/poster_renamerr.log"}

# Some checks to see if everything is fine
if [ -z "$B" ] || [ -z "$A" ] || [ -z "$search_term" ] || [ -z "$file_path" ]; then
    echo "One or more variables are empty"
    exit 1
fi

if [ ! -f "$file_path" ]; then
    echo "File does not exist: $file_path"
    exit 1
fi

# Print some output
echo "Search Term: $search_term"
echo "File Path: $file_path"

# Add wildcards before and after the search term
search_term=".*${search_term}.*"

# Run the command!
grep -B "$B" -A "$A" -i -E "$search_term" "$file_path" --color=always

Mostly I need a primer on the meanings behind ‘usage 2’ & ‘usage 3’

14

It is just greping the log file for string, defaults to searching for “test” in the log file.

15

I get that, but I’m not familiar with that ‘alias logsearch’? Is this creating a shortcut/pointer of some kind to make running it easier or something? Or, should it work by regular invocation (cd ./log_search.sh) now?

16

No idea the scripts docs are odd and do not seem useful.
Maybe I’m missing something?

17

Well I know if you’re missing something there’s no telling what I’m missing lol. This is the repo where I got these.

18

I would be concerned about any instructions telling you to use an old of support python e.g. 3.8. Is it poor docs or is bazarr not maintained?

19

Yep. It’s a bit of both really. It’s semi-maintained in the sense that I see an ‘update’ once in a while. Why it only works with 3.8 I have no idea. Most of these things I’ve found claim you can run them locally (on a Mac) but they always recommend/primarily write for docker installs. I don’t like docker on Mac at least so I keep trying to get these things to run natively.