A look into workflow tools - package management in the Python extension for VS Code

[courtneywebster]

I am one of the Program Managers working on the Python extension in VS Code! As part of my work, I have been taking a look at package management tools. The Python extension is considering ways in which we can improve the package management experience in the extension with a standard Python install with only pip and venv. I compared popular package manager tools and published a blog post documenting the investigation and proposal. I would love any feedback on how we are approaching this problem as well as the current proposal. We want to ensure we are not leading users astray while offering them enough support to be successful and efficient. Thanks!

[brettcannon]

@courtneywebster

I would love any feedback on how we are approaching this problem as well as the current proposal. We want to ensure we are not leading users astray while offering them enough support to be successful and efficient.

To be a bit blunt: this is our plan that will get pushed out to pretty much every Python developer using VS Code which isn’t small. We are also seeing more usage in education. Those two combined mean we don’t want to mess up and start directing people towards anything people might consider bad practices, so if you see something you’re scared your new hires might come in doing because we taught them to do that, speak up now.

I will also say this is why I would love a way to install just dependencies from pyproject.toml, so we don’t start off supporting requirements.txt and instead start with a standardized file (if need be we can change what we do by default later on).

[pradyunsg (Pradyun Gedam)]

Haven’t looked at the blog post in detail yet but --only-deps in pip is on my radar as something I’m interested in doing (I started looking at it on Monday) and I’m hoping to get that done by the next pip release (i.e. not the one this month).

[pf_moore (Paul Moore)]

This sounds like a very interesting idea. The blog post says

Initially, these features primarily focus on assisting beginners who work on scripts, applications, and projects that don’t require building a wheel file.

That sounds awesome, and is a use case that’s not well supported at the moment. I’d love to see more focus on this from the whole packaging ecosystem, and if this proposal encourages that, then great.

But it does mean that I’m concerned about typing this to the [project.dependencies] key in pyproject.toml, because PEP 621 was created purely on the basis of projects which do target building a wheel^[1]. I don’t think it is necessarily the case at all that the existing project metadata specs (PEP 621, and the packaging metadata spec) meet the needs of adhoc scripts, for example.

Maybe we need to cover that here (probably in a separate thread) and get consensus before implementing something in pip that would effectively become the de-facto approach?

---

1. Unless there was an implicit expectation - but I certainly only approved the PEP based on that use case. ↩︎

[pradyunsg (Pradyun Gedam)]

@pf_moore

Maybe we need to cover that here (probably in a separate thread) and get consensus before implementing something in pip that would effectively become the de-facto approach?

Let’s do that here (to avoid hijacking this thread):

Projects that aren't meant to generate a wheel and `pyproject.toml`

And… I think using project.dependencies is OK even in projects that don’t intend to generate a wheel. Even for projects that aren’t intended to be wheels, specifying this information in this form seems like a good idea to me. That said, PEP 621 states that name and version are required. Further, given the lack of pip install --only-deps at the moment, there isn’t really any way for users to only-install-dependencies when specified in this manner. Assuming that feature happens[1], do folks hav…

[groodt (Greg Roodt)]

Great initiative @courtneywebster

This looks like a great quality of life improvement for new and experienced python developers. It’s also not locking anyone into VS Code and seems to play well with the built-in and standards-centric tooling.

[pf_moore (Paul Moore)]

One comment I’d make is that, while writing an initial pyproject.toml when creating a new project is a reasonable thing to do, modifying an existing pyproject.toml is much harder. Because it’s human-edited you need to preserve formatting and comments, which while possible is tricky.

What would definitely be a step backwards is to create a new generation of programmers who expect to have a tool to manage pyproject.toml, rather than being comfortable editing it by hand when needed. Or projects with unreadable pyproject.toml files because they were written by a tool (whether VS Code or something that follows its example but doesn’t make the effort to preserve formatting).

[pf_moore (Paul Moore)]

@brettcannon

I will also say this is why I would love a way to install just dependencies from pyproject.toml , so we don’t start off supporting requirements.txt and instead start with a standardized file (if need be we can change what we do by default later on).

If you start with pyproject.toml, you will definitely need to change later, because pyproject.toml can’t support all the use cases for requirements.txt^[1] - so whatever is ultimately the requirements.txt replacement, it won’t be pyproject.toml…

So I’m not sure what it buys you pushing for a stopgap change to get a “part of requirements.txt” solution right now, rather than just using what we have until it actually does get properly standardised.

---

1. Specifically, pyproject.toml can’t support sharing the same requirements file between two projects, or having a requirements file for a project that doesn’t live in its own dedicated directory… ↩︎

[brettcannon]

@groodt

his looks like a great quality of life improvement for new and experienced python developers. It’s also not locking anyone into VS Code and seems to play well with the built-in and standards-centric tooling.

Thanks! We never want to lock anyone into VS Code, hence why we are limiting ourselves to things people could do on their own in a shell or via some other manual step(s).

@pf_moore

If you start with pyproject.toml, you will definitely need to change later, because pyproject.toml can’t support all the use cases for requirements.txt

It honestly doesn’t need to. We only care about beginners and providing an opinionated flow for the common case. We are going into this knowing full well we won’t be able to come up with something for everyone or something that even intermediate/advanced users will necessarily want.

@pf_moore

What would definitely be a step backwards is to create a new generation of programmers who expect to have a tool to manage pyproject.toml

Are you saying that’s okay for requirements.txt? Or are you suggesting we don’t write down what a user wants to install or has installed (if PEP 710 - Recording the provenance of installed packages is accepted)?

@pf_moore

doesn’t make the effort to preserve formatting

I wouldn’t let that ship/happen.

[pf_moore (Paul Moore)]

@brettcannon

It honestly doesn’t need to.

You don’t need it to. But a standard that doesn’t replace all the use cases of requirements files means we’ll still have people using them, and we’ll have people confused over which to use. I’m trying to look at the wider context here.

Which is why I think sticking with requirements files for now might make more sense. Yes it means you will need to change when (if) we get round to standardising them, but if we do a partial solution now, I can see a later standardisation effort replacing both with something unified, so you will have gained nothing.

@brettcannon

Are you saying that’s okay for requirements.txt?

No, just that I think that manual creation and editing of files should remain a primary use case. My perspective is obviously different from yours, but I worked a lot with Java, which has reached the point where a lot of the infrastructure cannot be managed without an IDE, and even the experts concede that’s bad. I don’t want python (and python packaging in particular) to go down that route. Editors and IDEs are great tools, but bad masters.

@brettcannon

I wouldn’t let that ship/happen.

Thanks - I assumed as much from you, but I worry about other tools not being so careful. Having the data in its own file limits the problem to just the requirements data. A careless tool can’t wreck a carefully commented ruff config, for example.

[brettcannon]

@pf_moore

You don’t need it to.

Correct.

@pf_moore

But a standard that doesn’t replace all the use cases of requirements files means we’ll still have people using them, and we’ll have people confused over which to use.

Also correct. I’m just trying to figure out the scope of this so that I/we know what we are potentially thinking of doing to solve this issue since this isn’t a new problem, but we have also punted on solving it since it is a big one. Obviously a small fix solves my need, but I realize it doesn’t necessarily make it a bigger solution for most people.

@pf_moore

No, just that I think that manual creation and editing of files should remain a primary use case.

Definitely. Speaking as the dev manager for VS Code, we never want vendor lock-in for our extension. Everything we do is meant to make your life easier, so that if you were to leave VS Code for some reason it shouldn’t break your workflow to the point that you have to re-implement something (just maybe a pang of regret for leaving ). In this instance that means continuing to push and support things people can edit by hand when it’s meant for human modification.

My PhD involved aspect-oriented programming, I know what the IDE-required coding life is like and I don’t want that for anyone.