Would you mind giving an example?
Can’t we explain it in Python?
Semaphores are a pretty basic and fundamental concept. If someone can’t use them appropriately, they are in for a hard time.
That’s exactly what I am talking about. They are fundamental concept and they should stay somewhere deep in fundaments. To use overused parallel, they are as fundamental as GOTO, and thus they should be as hidden as much as it is (even though we all know that it is all GOTOs down there). When you start using GOTO and semaphores, you suddenly are in the world of threadlocks, deadlocks and all that funny stuff and you are in the world of bholley - Must be This Tall to Write Multi-Threaded Code . What I was talking about by mentioning concurrent.futures was that we need more abstractions hiding this stuff from us who are not that tall.
Also, Notes on structured concurrency, or: Go statement considered harmful — njs blog
1 Like
And if you read the rest of the post you quoted, you’d find that the right tool for your example is setting the limit in the http library of choice. Putting this in an executor doesnt work for limiting it, because the libraries you call into might not have the “simple” behavior you think they have. Libraries doing more for you means you need to also work with the libraries when it comes to limiting their behavior.
Every time we “hide” the complexity, it just shifts it or leads to misconceptions. Sure, we can make it easier to manage in some cases, but the example of loading random urls and wanting to limit it at an executor shows that people need to understand what a limit actually applies to.
1 Like
If it’s exactly the same as asyncio, then it’s vulnerable to the exact same problems that asyncio is, including getting “stuck” on gethostbyname and stalling out the entire virtual thread subsystem (effectively blocking all other virtual threads as well). So it’s like threading in that you need explicit semaphores to protect critical sections (which programmers are notoriously bad at doing), and it’s like asyncio in that you can’t context switch inside system calls (which programmers are notoriously bad at remembering to account for).
1 Like
I tried looking for information on Java, virtual threads, and gethostbyname, and came up blank. Perhaps someone who’s advocating this can provide more information?
First of all, the example I posted was using the standard library which doesn’t have any internal pools of connections as far as I know.
Second, more importantly, I was not using that example to talk about HTTP Connections in the first place, but about concurrency, and how to make it more useful even for the rest of us.
And yes, some of the methods how to hide concurrency is to burry it in a good API for the given library (which both httpx and requests do). Which is why I have been always (completely unsuccessfully) lobbying for improving standard library urllib.request to be more useful for the production use. For reasons which I don’t dare to suggest here, we got instead that stupid ad for requests on the documentation page. Oh well.
Well, one hiding of the complexity is inside of the API as httpx does, which is IMHO perfectly OK solution for that problem, but it is irrelevant to the overall discussion about accessibility of concurrency.
The longer response back and forth was specifically because you asked:
Which is a limit your original code didnt have. Your original code also didnt define load_url, anyone could guess if you meant the standard library or some other library, but the only limit in your original code was the number of threads the threadpool executor was responsible for.
If you are relying on this to limit anything else, it’s incorrect. The standard library could change other (internal, implementation detail, not guaranteed) behaviors, and your “implicit” limit would no longer function as your intended limit.
1 Like
Not sure where is this fascination with gethostbyname coming from (as opposed to any other blocking syscall), but in my view the runtime would provide a version of it that blocks the virtual thread but doesn’t block the OS thread. How the runtime implements it is left to the runtime (how does asyncio do it?).
I think Go uses a different approach where it spawns more threads than GOMAXPROCS and uses some of them for waiting on syscalls, but not running “user” code. It’s quite interesting, but also feels like an implementation detail at this phase.
1 Like
I wouldn’t have known your example was using urllib. It uses a function urllib doesn’t define, and you didn’t define it yourself or link the origin of the example before now.
Even so, that example is dangerous if people assume it means it limits concurrent connections and urllib ever gets any performance improvements based on internal connection reuse, and can clearly mislead people about what is being limited.
Nothing other that it’s a very common one, and thus a useful example. Everything I’ve said about ghbn would be equally true of any other blocking syscall.
It doesn’t. There is an implementation of getaddrinfo() which is similar, but there simply is no asyncio version of gethostbyname. And in that particular instance, you can MOSTLY ignore ghbn and use gai (which has an asynchronous version in the underlying C library), but this won’t be true of all syscalls. There are a lot that will block, and usually we don’t notice because they’re fast - but when they AREN’T fast, there’s no solution. And sometimes this means that a process gets stuck in disk sleep, whether it’s for half a second or forever. If you have some sort of slow device, or a network-mounted file system, or anything else, perfectly innocent-looking code could start taking much longer than you think it does.
Fundamentally, this CANNOT be solved as easily as you’re implying. If it were, why hasn’t it already been? Why wouldn’t OS threads be as lightweight as this? If you could make threads more efficient without cost, everyone would have jumped on it.
This is a common approach, but it just means that you have a different limiting factor. For example, you could have a thread pool dedicated to hostname lookups, but then you still hit the limit in that, which might be fine right up until the file system containing /etc/hosts starts throwing errors and your entire system slows to a crawl. Or something crashes unexpectedly when it’s not even looking like it’s doing networking. Don’t ask me whether this can happen or how I know.
1 Like
Yes, it has all of the same problems as asyncio. That is the nature of cooperative scheduling). That is why these tools are useful specifically for problems that require a lot of IO. They are terrible for problems that require a lot of CPU processing. This is another reason why I don’t like asyncio. It forces async/await syntax in libraries which bleeds into other code, even when cooperative scheduling is not an appropriate solution for the problem you are trying to solve.
I’m actually surprised asyncio doesn’t have a cooperative version of gethostbyname. Here is an example using gevent which does not block because gevent.socket is cooperative.
>>> import gevent
>>> from gevent import socket
>>> urls = ['www.google.com', 'www.example.com', 'www.python.org']
>>> jobs = [gevent.spawn(socket.gethostbyname, url) for url in urls]
>>> _ = gevent.joinall(jobs, timeout=2)
>>> [job.value for job in jobs]
>>> ['74.125.79.106', '208.77.188.166', '82.94.164.162']
Virtual threads would presumably work similar to this, but likely with a different API. I think Java’s own docs on Virtual Threads is quite clear, if you want a more in-depth description.
I don’t know how hard it is, but it’s obviously not impossible since virtual thread-like technology today can perform dns requests, so it can’t be impossible. Maybe it introduces some additional constraints, but from what you say, that’s no different from asyncio today.
I’m not sure what argument you’re making here. I don’t know why OS threads are an order of magnitude heavier than tasks/goroutines/virtual threads/whatever, but they are. It doesn’t really matter why. A layer on top of them turns out to be very effective (as asyncio itself proves); now we’re talking about a new layer on top of them, with some different properties.
3 Likes
DNS requests aren’t the problem. System calls are. Notably, gethostbyname does not just do a DNS request. There are alternatives, but as I mentioned above, this is about blocking syscalls, of which there are many, and most of them are fast most of the time - which makes them hard to test, but a very real problem when they aren’t fast.
My point is that, if it were simple to achieve concurrency WITHOUT the overhead that OS threads have, why would OS threads have all the overhead? Are the Linux kernel devs just so stupid that they can’t take advantage of these obvious improvements? Or, perhaps, are OS threads actually doing something that simply cannot be done with the lighter-weight ones?
Whenever you move away from real OS threads, you ARE compromising in some way. Sometimes that’s okay, but trying to deny it is not doing you any favours. Being dismissive of it, as you have been, gives the impression that you have no idea what the issues at hand are.
1 Like
No one is making the argument that virtual threads will be just like OS threads, only better. They (if they come to exist) will be their own thing, a new abstraction built on top of OS threads, but similar enough to remove the need for function coloring. I’m being dismissive of your concern about performing blocking syscalls since I consider it a solved problem, one way or another.
I don’t have a lot of expertise in threading, but I thought some research into Java’s virtual threads could be useful to the discussion. It looks like Java’s virtual threads will temporarily expand the OS thread pool when blocking syscalls without non-blocking equivalents are called. What is blocking in Loom? (specifically the Files & DNS section)
4 Likes
It’s worth noting that newer languages aren’t trying to remove async or design without async in favor of other concurrency options, nor are they considering function coloring a problem beyond what it causes for code duplication to support sync and async simultaneously. They are looking at options for tackling that duplication.
Rust has the in-progress keyword generics initiative and zig is going a different route of passing that IO interface to functions that use it: Zig's New Async I/O | Loris Cro's Blog with future planned syntactic sugar.
The primary complaint about async/await[1] that doesn’t directly map to a similar problem that would by necessity also exist in virtual threads people keep bringing up is function color. So, is there something we can do to assist at the language and standard tooling levels for function color beyond telling people to write everything sansio? This is an approach that seems more productive than adding another way for the ecosystem to be split on concurrency, but it could be viewed orthogonally to it if even with this there would be a benefit to virtual threads (I have not seen one presented that actually makes a fair comparison in favor of virtual threads if function color and duplicated code were suddenly not a concern, whether it is orthogonal or not really depends on if this alone would be sufficient)
I’m saying async/await and not asyncio intentionally here, there’s certain APIs in asyncio that I believe could have been better, but are far too late to change, even with the benefit of hindsight. ↩︎
4 Likes
Saying independent tasks is like saying a perfect sphere in a vacuum. Tasks share resources all the time, so they are very rarely independent. The very simplest CRUD app is going to share a database connection pool between the handlers.
Work-stealing is a huge deal. You mention Rust, Go and Java are other examples of widely used work-stealing concurrency.
But it’s not just about work-stealing. At a high-level, it’s about sharing resources to use them more efficiently and avoiding congestion/starvation; work-stealing is about sharing OS thread capacity, but any non-trivial network service will have plenty of resource (usually connection) pools too.
Speed is an imprecise term so I tried avoiding it. I don’t care about speed as much as consistency, so I mentioned p99 instead of, say, p50. Avoiding starvation. In asyncio, neither tasks nor open sockets can jump threads. I don’t think tasks ever will be able to since it would violate the asyncio invariant. The idea virtual threads bring to the table is exactly that a virtual thread may continue running on a different physical thread, and a connection pool may be shared between many virtual threads running on several physical threads.
This sounds really complicated, only to still be exposed to starvation in individual event loops. Just out of curiosity, do you know of an open source framework that does this? I’m asking in good faith here so I can investigate it.
1 Like
Agreed to an extent, but if people work toward this model, only the resources shared across internal event loops need any level of extra protection, and it becomes more obvious what those resources are than what typically happens in most virtual threaded environments (java being one exception here, as it comes with directives to have the JVM provide the neccessary synchronization via synchronized, rather than forcing callers to know when locking is needed, or internally lock pessimistically, rust being another with lifetimes and the borowchecker.)
outside of something like uvicorn, which is a step removed from this? No, nothing that directly maps to this yet. It’s something I’ve started writing my design thoughts for, but prior to free-threading, wasn’t a priority for me. It made more sense to me to have multiple processes with eventloops as needed than threads so long as the GIL was in play.
You’ve got a few different people independently working on some various building blocks that will be necessary for something like this to come to fruition as well, and it won’t be a drop-in replacement for asyncio, even if it uses async syntax. Importantly, rather than asyncio’s current guarantee about state and context switches, you get a slightly weaker, but still incredibly useful “tasks on the same event loop will not overlap except at explicit yield points”[1], so long as the scheduling can be controlled by users, this can be enough to get the safety they need without explicit locking, or at least as much explicit locking (if an app does 4 different kinds of things, doing each of those kinds on their own thread might be safe without additional locking)
Technically, this is already the only guarantee, but multiple event loops is a rare case currently. ↩︎
One might point to the number of thread-related bugs we’ve seen during that time as evidence that we should have been introducing these concepts up-front and making people think about them, or that we should have been searching for a better way to do things that doesn’t hand programmers quite so many projectile weapons pre-loaded and pointed in the direction of their feet.
My own personal point is just that async/await makes the yield points syntactically explicit, which means it’s easier to reason about what might happen/when/where/etc. and thus it’s more obvious where protective mechanisms are needed. And thus I strongly prefer it over threading in any form.
The proposal here for virtual threads, even if limited to “virtual threads can only yield when doing I/O”, still requires the programmer to constantly be aware of every line which might perform I/O, including those which might only do it deep inside a stack of calls to other functions/methods. The async/await approach does not hide this; in fact, the “function color problem” literally becomes a signaling mechanism telling you which function/method calls might be doing I/O because they have a different calling convention and propagate that calling convention all the way through your call stack.
1 Like
we should have been introducing these concepts up-front and making people think about them
But we don’t because it would have no effect. I was programming for years before I finally understood what a pointer is. I bloodied my keyboard for months trying to understand concurrency. It was over a year before I finally felt comfortable with it. We delay the introduction of these topics because it would be impossible to understand them without first understanding the fundamentals. How are you going to teach concurrency or threading to someone who is still trying to get a handle on recursion, or even the relatively basic concept of an atomic operation? Frankly, (and no offense intended, we are all learning) I think it’s evident in this thread that quite a few people are using asyncio with little understanding of concurrency.
async/await will not teach you that. Knowing where IO happens is not a sufficient condition to understanding concurrency. And if you don’t understand concurrency, you are going to implement concurrency bugs whether you can see the IO or not.
There have been an awful lot of opinionated claims in this thread, and very little demonstration. It is very easy to say that async/await prevents bugs, but it seems quite difficult to produce any distinguishing examples. At least, when I search for examples all I find is posts about confusion with or rejection of async/await.
I did find this chapter in “Operating Systems: Three Easy Pieces” (Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau, 2023). It distinguishes two major forms of concurrency bugs: deadlock and non-deadlock. Deadlock bugs clearly require locks, which you have to deal with whether your switches are explicit or not.
That leaves the non-deadlock bugs, where they state that a “large fraction (97%) of non-deadlock bugs studied by Lu et al. are either atomicity or order violations.” Atomicity is the same with both explicit and implicit switching. It’s a big reason why concurrency is safer than threading. We are left with order violations. An example of that is one coroutine accessing shared memory before it’s initialized, or after it’s deleted. Seeing every switch point explicitly does not protect you from having to control access to shared memory. You still need a lock.
Of course we should always try to make it safer to not hurt ourselves while programming. But async/await is overkill. It’s like putting the type of every variable on every line where you access it (plus the color thing).
That makes me think… why don’t we just get rid of await? async def is fine as a hint your editor can use to fill in the awaits if you want them. Why can’t we approach this problem the way we did with typing?
1 Like