My use case for this would a bit of an odd one. I want to implement a library of decorators that allow users to use this really slick syntax to interact with an LLM in a multistep way.
@use_lm("gpt-4o")
def multi_step_promopt(problem : str = "1+1"):
response_from_lm = yield "hey llm how is it going " #" good and you"
second_response_from_lm = yield f"could you calculate {problem} and format your answer as Answer: "
# Here the messages are automatically accumulated behind the scenes (so "hey ...", "good and you", "could you" are in the context of the llm on the second yield)
return second_response_from_lm.split("Answer: ")[-1]
print(mult_step_prompt()) # 2
The reason I want it to all live in a function like this and use the yield syntax and so on is because of the version control of individual fucntions we’ve developed as a part of https://docs.ell.so/ and especially Versioning & Tracing | ell documentation which involves versioning a users calls to an LLM as they do prompt engineering by computing the source code of the lexical closure of a user’s prompt function and storing that.
With the current implementation of generators I can implement @use_lm for the sync case, but I cannot implement it for the async use case.
This use of generators might look horrid to you but its a type of sugar that drastically reduces the amount of pain in the AI/prompt engineering community while very much so increasing the readability of multi-step interactions with LLMs. (We’ve seen a ton of interest at least in our standard decorator library with 4k stars in the past two weeks and I’m considering introducing this generator style prompting method but this arbitrary difference between async and non async generators would require me to implement a different solution for async users. I’m happy to do hack it out, but again I just see a clear path here to make async and sync generators have parity.