Add a threading.run convenience function

Ideas
21

Personally, my preferred way of dealing with this is to use lambdas:

t = threading.Thread(target=lambda: func(3, 1.0))

Which I find more readable than the proposed run method:

threading.run(func, 3, 1.0)

Another advantage of the lambda-based idiom is that type checkers already work well with it, whereas the usual Thread(target=func, args=(3, 1.0)) or the proposed threading.run(func, 3, 1.0) don’t work well with static type checkers.

It goes without saying, but an additional obvious advantage is that it works with all current Python versions without requiring any changes to CPython.

One disadvantage is that there are some surprises around lazily-bound names (notice the i=i required to make this work):

for i in range(10):
    Thread(target=lambda i=i: spam(i))

I had a refactoring of the aforementioned ThreadSet to lean into the lambda style, but left it hanging. Will probably pick it up again at some point.

22

That’s vulnerable to variable capture problems. Try doing that in a loop.

23

See the third code block.

24

This would work:

threading.run(lambda: func(3, 1.0))
25

Should I make a pull request to progress the subject?

26

It would be nice to see the proposed reference implementation.

I’m looking at the code for subprocess.call and subprocess.run and I notice that they’re geared around:

  • context managers to avoid resource leaks
  • timeouts
  • subprocess.run communicating with stdio to avoid EWOULDBLOCK
  • wait until the subprocess has finished, handling exceptions properly

threading.run wouldn’t have any of these right? I’m thinking of it as more comparable to a raw os.fork() which returns immediately, and it’s up to the user to handle resource collection later (threading.Thread.join and os.waitpid).

subprocess.call is short but looks worth it for exception handling. Seeing the reference implementation for threading.run would show whether it’s:

  1. short but with side effects and repeated so often it’s worthy of its own function
  2. just a convenient wrapper to save one or two lines, for the most common case of immediately starting a thread once its created (which doesn’t necessarily rule out the admission of threading.run; people may still really want this shortcut :sweat_smile: )

As a side note for resource cleanup, it makes me think of socketserver, and how the ForkingMixIn cleans up its children when __exit__ing a server context. Is there an equivalent ‘threadingserver’ module for creating threads and having them cleaned up when exiting a ‘ThreadServer’? *Googling*. Is that what concurrent.futures.ThreadPoolExecutor is for? I only occasionally use threads. I prefer forking, giving each subprocess memory autonomy. That’s why I pushed for ForkingUnix{Stream,Datagram}Server to be added. Those were a oneliner each

# socketserver.py
class ForkingUnixStreamServer(ForkingMixIn, UnixStreamServer): pass
class ForkingUnixDatagramServer(ForkingMixIn, UnixDatagramServer): pass

but were a healthy addition, and the server I actually wanted :smiley:

So tldr: yeah let’s see it!

27

Yeah the initial idea would be a simple wrapper.

If you want more functionality, I’m thinking of something:

# (remake of threading documentation example)

links = [
    "https://python.org",
    "https://docs.python.org",
    "https://peps.python.org",
]

pages = threading.join(threading.run(fetch, link) for link in links)

# or plainly

reqs = [threading.run(fetch, link) for link in links]
pages = [req.join() for req in reqs]

threading.run would not return a thread but another thing capturing the function return value. The implementation would be less straighforward though.

28

At that point you should probably be farming jobs off to a ThreadPoolExecutor.

4 Likes
29

Right, your main concern was removing the start() call then, rather than the parameter passing? Personally I prefer the explicit version, but I’m in favour of having an additional choice.

What I found myself wanting more is to call start and join over a set of threads, which is why I implemented ThreadSet. You can find several usages in that package’s test suite. In general, I like this pattern:

readers = ThreadSet(...)
writers = ThreadSet(...)

(readers | writers).start()
writers.join()
writers_done.set()
readers.join()

It’s the for-loops that I find unnecessarily verbose.

Note that join() always returns None, so your example here may need some clarification:

30

Yeah I meant req could be a thread-like object that return target result on join().

31

I like the convenience of the proposed decorator form, but also agree that there should be a way to optionally configure a thread. We can do this by making threading.run an optional decorator factory based on the presence of a positional argument to support both of the following decorator usages:

@threading.run
def task():
    ...

@threading.run(name="foo")
def task():
    ...

while also supporting non-decorator usage both with and without thread configuration (args and kwargs are supported in this form):

t = threading.run(task, 1, option=True)

t = threading.run(name="foo")(task, 1, option=True)

The decorator form doesn’t need to support passing arguments because such usage is meant for one-time-use target functions that typically don’t need to be parameterized.

So something like:

def run(*args, **kwargs):
    def configured_run(target, /, *args, **kwargs):
        thread = Thread(target=target, args=args, kwargs=kwargs, **options)
        thread.start()
        return thread

    options = {} if args else kwargs
    return configured_run(*args, **kwargs) if args else configured_run
32

Regarding names this is possible:

@threading.run(name="foo")
def task():
    ...
# ->
@threading.run
def foo():
    ...


t = threading.run(name="foo")(task, 1, option=True)
# ->
t = threading.run(foo, 1, option=True)
1 Like
33

I think we’re overcomplicating this. This should be fine and cover most use cases:

def run(func, *args, **kwargs):
    thread = Thread(target=func, args=args, kwargs=kwargs)
    thread.start()
    return thread

def run_daemon(func, *args, **kwargs):
    thread = Thread(target=func, args=args, kwargs=kwargs, daemon=True)
    thread.start()
    return thread

People who want to change name or context can continue to use the Thread constructor.

4 Likes
34

Agreed. Let’s not let generality be the enemy of utility. The functions you’ve suggested will serve nicely in multiple contexts and be convenient.

1 Like
35

Okay that signature / syntax is quite nice.

example use 
import threading

LOCK = threading.Lock()

def print_stuff(n: int, name: str | None = None):
    s = f"{n}" if name is None else f"{name} is {n}"
    with LOCK:
        print(s)

threads = []

for idx in range(8):
    t = threading.run(print_stuff, idx, name="xyz")
    threads.append(t)

for thread in threads:
    thread.join()

Okay that’s fine, and so my only major remaining concern is:

In the past in small codebases I have created threads via

threads = [threading.Thread(target = download, args = (idx, i, j, k)) for idx in range(number_of_threads)]
for t in threads:
    t.start()
for t in threads:
    t.join()

in which the separation of ‘create’ and ‘start’ was just an artefact of using a list comprehension. In bigger codebases maybe people want this as intended behavior to produce a separation of powers in (create-create-create)-(start-start-start)-(join-join-join); one block of code doing the creating, another doing the starting when ready, another joining to finish up.

With create-start-create-start-create-start-join-join-join each ‘create’ stage might have delays leading to really staggered 'start’s. I don’t think it’s too much of an issue, and code shouldn’t be relying on side effects of thread start times, but it does make me think of how bind(2) and listen(2) are separate partially to allow lots of binding, then any final cleanup, then fast listen calls, ie (bind-bind)-cleanup-(listen-listen)-poll, as otherwise with bind-cleanup-listen-bind-cleanup-listen-poll connections to the first sockets would be accepted by kernelspace early on but not accept(3)ed by userspace for a while as the userspace code takes time with cleanup.

Overall I’m +0 now, swayed by the simple func, *args, **kwargs syntax.

36

It’s also notable that bind can require elevated permissions, which can be relevant.

TBH if I could steal the time machine for a bit, I’d make it that threading.Thread(func) created and started a thread (meaning that the target would also be the first argument), and if you DON’T want it to immediately start, threading.Thread(func, start=False). It’s far more common to want it to start.

1 Like
38

I disagree that my proposed API is overcomplicating this, when it supports the common usage of run(func, 1, option=True) just as well with about the same number of lines as your sample code above. It does incur one more call in an attempt to keep the code DRY but I wouldn’t say the extra call makes it overcomplicated, and the extra call can be avoided by providing the common unconfigured form a direct path:

def run(*args, **options):
    if args:
        thread = Thread(target=args[0], args=args[1:], kwargs=options)
        thread.start()
        return thread

    def configured_run(target, /, *args, **kwargs):
        thread = Thread(target=target, args=args, kwargs=kwargs, **options)
        thread.start()
        return thread

    return configured_run

I personally find naming threads important in larger projects. Popular projects routinely name threads after their operational role, owner, or instance, not merely the callable. If threading.run is released without the ability to name a thread, it may actually lead to people giving up on a more readable debugging output for the luring convenience of the more ergonomic API.

As a reminder (I’m sure most of you are well aware though), here are a few quick examples from popular projects that illustrate how meaningfully named threads can be helpful:

Uniquely identify a worker:

Distinguish multiple threads running the same target for different resources:

Replace a nearly meaningless target name with the thread’s actual purpose:

Identify which component owns an otherwise generic event loop:

39

If a project really wants to name their thread, they can continue to use the Thread constructor, or set the name attribute on the returned Thread object. In most cases, the callable name will be fine anyway; do you really find asyncio-waitpid that much better than _do_waitpid?

The proposed solution makes this API’s behavior much harder to describe, and will likely lead to some interesting gotchas. I’m inclined to quote the Zen here: “If the implementation is hard to explain, it’s a bad idea.”

40

A threaded decorator would allow to make threaded functions

threaded_fn = threading.threaded(func, name=None, group=None, daemon=False)

threaded_fn(*args, **kwargs)  # start

This would allow OP case to be done with

threaded(func)(3, 1.0)
threaded(func, daemon=True)(3, 1.0)
41

All of these kinds of things are still possible. If they make sense for your project, you are very welcome to create them as local utility functions.

Let’s get a simple convenience function. It doesn’t have to do everything, it just has to do what’s actually common and useful. For everything else, the underlying Thread constructor is still available.

3 Likes