Ok, so my $.02 about some high-level conceptualizations. (Amongst other things, I make PursuedPyBear, and we’re looking at briefcase to enable the creation of end-user distributables. I also have made plenty of services and created many deployment pipelines.)
I’m sorry this is so long. I’m not sure there’s a good tl;dr.
I’m going to be summarizing/restating a lot of stuff, partly to set up my conceptualization, partly so that if my assumptions are wrong I can be refuted at that level, and partly so that we’re all on the same page.
There’s a lot of tools and a lot of uses for the dependency-locked environment workflow/concept/etc. The goal of this thread/effort is to standardize the format of the lockfile (and intrinsically some of the concepts) to improve interoperability–“pipenv creates a lockfile that heroku uses to deploy” kind of stuff. It would also be good if a set of libraries to work with this were created so that new tools and workflows can be created and meaningful innovation can happen. (It’s much easier to make new kinds of bikes if you don’t have to re-engineer the bolt every time.)
I’m going to define some terms, and they’re going to wildly conflict with already-overloaded terms. Hard problems and all.
Packages are things that can be installed into an actualized environment. They have versions, dependencies (both hard and optional), artifacts used in the actual installation, restrictions about when those artifacts can be used, etc. Note that different artifacts can have different lists dependencies. (This maps to how sdists and wheels work. “installation restrictions” just means stuff like platform, Python version, Python ABI, etc.) Note that for this discussion I only care about runtime requirements, not build requirements.
Conceptual environments are ideas like “production” or “debugging” or “building” or “formatting”, or briefcase’s “mac”, “windows”, “android”. They have a specification and lock data.
A specification is a list of “top-level” packages that a human specifies, with version requirements. This is where a person says “Give me flask, sqlalchemy with postgresql support, and flask-redis of version 4.3.*”.
Lock data is computed given a specification and package data. It describes the total set of packages that need to be installed to satisfy a specification: All the packages specified are installed, including dependencies recursively, and nothing has unmet dependencies.
(Note that lock data is only valid for a given artifact restriction context: because artifacts have installation restrictions, and different artifacts of the same package/version can have different dependency lists, the lock solution is only valid for a given restriction context.)
Conceptual environments can be actualized with the assistance of a tools. To be extra clear, actualization is turning the lock data into something that actually exists on the system and is usable. I’m deliberately being vague about it because pipenv and briefcase have pretty different ideas, but it generally means something to the effect of “download, unpack, and possibly build all packages in the lock data”.
A few times I refer to a project. This is the set of source code, metadata, etc that does something. Some projects produce packages. Some produce end-user deliverables. Some are deployed to a server as a network service. I think I generally say things like “package produced by the project” to mean “this project, but interpreted as a package”. I don’t mean to imply that you must build this project into artifacts to do anything useful with it.
So, how do I see these concepts mapping to existing technologies?
Actualized environments would be things like venv, conda, containers, or who knows what else. Most of the existing stuff is about development and deployment environments, but these can also be actualized into portable(ish) environments (buildpacks, briefcase, or my own poetry2container).
Specifications are pretty varied–things like requirements-formatted data, setuptools specifications (setup.py or setup.cfg), poetry, pipenv, briefcase, tox I think, and probably more. There exist PEPs to specify some common syntaxes, but there’s still variation. Also note that a specification can be composited from multiple sources–something poetry-like might look at both the installation requirements of the package produced by a project as well as some variant of “development” requirements.
I mentioned package data about producing the lock data. This would be the total set of available packages and their metadata. Usually from a package registry. For Python+PyPI (I don’t know Conda), some the necessary data is available from the pypi index api, but some of it can only be found inside artifacts. (Good news, both sdist and wheel provide all the necessary metadata without having to actually build/install anything, so you could produce lock data for py3.8+windows from py3.6+linux. Bad news, you’ll have to download all the artifacts of interest even if you never use them.)
The big thing this thread is trying to nail down is the lock data schema and the format in which it is serialized to disk.
A conceptual environment doesn’t really exist–it’s an idea, created from and providing context for specifications and lock data. Briefcase might define one for each distribution target. Poetry might have prod and dev environments for the current system.
Note that it’s the restriction context (platform) that some current tools fall over on; for example, pipenv has (had) bugs to the effect of “projects locked on linux fail to install on mac”. Lock data only makes sense given that context, because you need that context to select dependencies. And because Python is cross-platform and lock data is shared among all developers in a project (even if they’re on different platforms), any solution needs to account for this.
Note that in this, I’m not defining that actualized environments are mutable or immutable, or that lock data describes the totality of what is installed in an environment. I think these are tool-specific decisions best left to the tool. (Although maybe providing utilities to make this easier would be appreciated.)
Universal, however, is the flow of specification -> lock data -> actualization.
Ok, so how would a tool use this workflow?
Ok, let’s take xot. It’s basically tox, but I get to define how it works so I don’t conflict with how the real tox actually works. xot is a testing (task) framework that lets you run validation things in a variety of configurations.
We’ll map those configurations to environments. Some have unique specifications (like a lint task) and some have shared specifications (like a testsuite matrix), but each one has a specification, which implicitly includes the dependencies of the package this project produces. Note that conceptual environments include the artifact restriction context (platform), so we’ll have to account for that. Let’s say that xot does workful thing and produces lockdata for the major platforms (linux/mac/windows) for each of its configurations. So each configuration would map to 3x conceptual environments (only some of which are actualized on any given system).
So the workflow is such: The developer writes a file describing the configurations and the commands to run under each configuration. The developer then run something like xot lock which resolves the specifications defined in the xot file into lock data for the platforms they care about. (xot may opt to also choose to actualize the environments pro-actively, so they’re ready to use.) They can then use xot go to actually run the validation stuffz. It’ll use the environments (actualizing from the lock data if needed) to run the tasks from the xot file and report the results to the user.
Both the xot file and the xot lock data would be added to VCS and shared amongst developers.
I think this was a bit more than $.02, but that was a lot of ground to cover. Hopefully, I actually contributed something.
)
)