Add a threading.run convenience function

Ideas
42

You’ve got to be rather deliberate to cut out the whole PID counter part from f"asyncio-waitpid-{next(self._pid_counter)}". I even captioned that specific example with “Uniquely identify a worker:”, which is the whole point of a thread name like asyncio-waitpid-1, not because of the asyncio-waitpid part but because of the -1.

The proposed solution is no different from the numerous existing optionally configurable decorators out there including the stdlib, such as functools.lru_cache, dataclasses.dataclass, unittest.mock.patch, etc. just to name a few. It isn’t hard to explain.

43

Sorry, I wasn’t trying to cherry-pick. PIDs are non-deterministic; seeing no PID vs some random number is not particularly helpful when looking at a holistic thread view.

These functions aren’t designed to be used without a decorator, so I don’t think that’s a reasonable comparison.

To be blunt: If we do these weird function-returning-function semantics, I am -1 on the API. I think the behavior is too weird for a stdlib utility.

44

As the name implies, it’s a PID counter, not a PID, and it’s initialized with self._pid_counter = itertools.count(0) so it is a process-local deterministic counter. But even if it were an OS-assigned PID, it would be unique on the system, and would be particularly helpful in distinguishing message origins when threads output to the same logging channel.

If having the run function perform double duty as a factory feels too unconventional, we can separate it to two functions, not unlike your two-function proposal:

threading.run(task, 1, option=True)

configured_runner = threading.runner(name="task-1")
thread = configured_runner(task, 1, option=True)
# or threading.runner(name="task-1")(task, 1, option=True)
45

I agree with Peter it would make the function too weird.

You can set the name after the fact:

for kind, name in targets:
	t = threading.run_daemon(self._run_kuberay_resource_events_watch, kind, name)
	t.name = f"kubernetes_events_watch_{kind}_{name}"
	self._threads.append(t)

or set it directly in the thread:

def _run_kuberay_resource_events_watch(self, kind, name):
	threading.current_thread().name = f"kubernetes_events_watch_{kind}_{name}"
	...
1 Like
46

Pretty sure that would run into a race condition since run/run_daemon starts the thread already when it returns–if the first thing that the thread does is to output its name it may very well output the default name rather than the name you set separately.

47

I think it would be better as a class, something like this:

runner = threading.Runner(name="task-1")
thread = runner.start_thread(something)

But anyway, I’m not convinced that this is a common enough pattern to warrant a stdlib helper. I think we should just add threading.run for now.

@rqndom, please open an issue for this on our issue tracker. Feel free to CC me and any other interested core devs (@encukou? @vstinner?) on it.

48

If that’s what you’re doing, then put the naming of the thread into the thread itself - as it starts up, it selects its own name. In the given example, the name can be entirely derived from other parameters to the thread, so it makes no difference where it’s set.

49

That works only if you own the target callable, which isn’t the case in the latter example I gave above:

50

Yes, you can ALWAYS come up with some example that won’t work with the convenience API. Congratulations. You can simply not use the convenience API then. Are you really arguing with me on the basis that the easy option isn’t applicable to a sufficiently-convoluted example? Really?

Can we stop bogging down the thread with “hey, look what won’t work here”?

51

My point is that my proposed solution, especially the revised one, isn’t complicated whatsoever at all, so I struggle to understand your perceived downside to offering the flexibility to configure the thread properly for legitimate use cases when it costs pretty much nothing over the run/run_daemon alternative.

3 Likes
52

But it is. It’s not nearly as simple as “just call this and now you have a thread”. Maybe it works nicely if you only ever define a brand new function to use as each thread, but it won’t work for anything where you want to create more than one thread from the same code.

You are, of course, welcome to create a function like this in your own code. Nobody is stopping you from making whatever helpers you need for the situation that demands them. That’s fine, same as it has always been. But for this, we’re looking for something simple that can go into the standard library, and which doesn’t need to be overburdened with edge cases.

53

I have open an issue on github.

54

Um, what’s the proposal is at this point?
I wouldn’t support the variants without name, as I’d like to keep it easy to add a name to existing thread start code. Any convenience function discourages using the advanced features that it doesn’t cover.

55

Unless we make the name argument required (which I think would go against convenience), I’m not particularly convinced that supporting it will result in more people actually using it.

56

Not at first, sure.
You add it when you want to debug things, which is a bad time to expand the run call to the equivalent 2 lines.

57

Maybe a design like this:

t1 = threading.Context().run(func, *args, *kwargs)
t1 = threading.Context(name='Example').run(func, *args, *kwargs)
t1 = threading.Context(name='Example 2', daemon=True).run(func, *args, *kwargs)

I think there is a an example like this earier that use the name Runner in place of Context. Not sure what a good name would be off the top of my head.

1 Like
58

This reminds me of the database library API with cursors and connections, where connection.execute(...) is shorthand for connection.cursor().execute(...).
It works very well as far as I’m concerned. For complicated tasks one can use the cursor explicitly, and most of the time one can just use connection.execute.

So people could modify their code as

t1 = threading.run(func, *args, *kwargs)

to

t1 = threading.Runner(name='Example').run(func, *args, *kwargs)
t1 = threading.Runner(name='Example 2', daemon=True).run(func, *args, *kwargs)

when the need arises.

1 Like
59

(reposting the message from github here)

What about: if the first argument is a dict then configure the thread with it?

threading.run({'name': 'http-worker'}, fecth, url, timeout=3)
threading.run({'daemon': True}, fecth, url, timeout=3)

with a signature threading.run([config, ]func, /, *args, **kwargs)

def run(func, /, *args, **kwargs):
	if type(func) is dict:
		config = func
		func, *args = args
	
	thread = Thread(target=func, args=args, kwargs=kwargs, **config)
	thread.start()
	return thread

(I would still want to keep a run_deamon shortcut)

For run vs. start naming I’m neutral. You can vote on your prefered name:

  • run()
  • start()
  • other
  • no preference
0 voters
60
  • The rationale for using start: Thread.run() is a separate method than Thread.start().
  • The rationale for using run: it doesn’t make sense to start a function, even if the return value of .run() is a thread (and not some function’s result)

For the mental model, having start() would make more sense. But it’s more convenient to also say “run this function in a thread, and give me back the thread”.

Alternatively, threading.eval(), threading.exec(), threading.schedule(), threading.spawn(), threading.threaded(), would all be reasonable alternatives. I like threading.threaded() because you create a thread from a function but I can see why threading.run() or threading.start() are more natural.

1 Like
61

That means that type checkers will not be able to tell you of mistakes in the names compared to threading.Runner(name=‘http-worker’).