I’m new to the python packaging realm, but wanted to propose something that I’ve been wanting for a while.
I want to standardize hatchling targets.
From what I can tell hatchling and setuptools (which treats them as custom “commands”) are the only popular backends that support custom targets.
Build frontends have no reason to support custom backend targets, so if you want to use custom targets you have to use hatch.
What do I mean by target?
I wasn’t actually sure the correct terminology here, but I went with target since hatchling uses it.
A target can be any artifact that is created by applying some transformation to the source.
Obviously, wheels and sdists are targets.
I will refer to them as “standard targets” going forward.
Motivation
I personally believe that providing extensibility to the interface is a good thing.
The current system forces people who want to create a custom target to either use hatch specifically or create their own build frontend and backend.
Zipapps
The primary purpose custom targets will serve me is the ability to create targets like shiv which vendor their dependencies.
Other Examples
These examples are linked in the hatch documentation
I also found this
Other ideas
-
If you had a python program that accepts python plugins, an alternative packaging scheme can be desirable for the plugins.
-
Other forms of packaged python meant as a standalone application.
What This is Not
This is not intended to provide a means to provide different flavors of targets.
What I mean by flavors is performing some sort of transformation prior to creating a standard target.
I.e. if you wanted to perform some form of obfuscation or minification prior to building a wheel, this is not a separate target.
This is intended to be done through the config_settings passed to the backend.
Interface
Here’s my idea for what an interface like this would look like.
This is a rough outline and I welcome any ideas on the best way to do this.
To keep in line with PEP 517 every target will have at least one required hook for building and one optional hook for getting build dependencies.
Build Hook
build_XXX(output_directory, config_settings=None):
...
Like the PEP 517 hooks, the first argument is the output directory of the build process. The same requirements hold for build_wheel and build_sdist as in PEP 517, but for targets outside of wheels and sdists, the hook must place the artifact(s) in the output_directory. If the hook creates a single file or directory, it must return the basename of that file or directory. Otherwise, it must return a list of the basenames of each file or directory it created.
Other than the special case of build_wheel, no additional arguments are allowed.
Every build configuration option should be set through config_settings.
Requirements Hook
get_requires_for_build_XXX(config_settings=None):
...
This follows the same specifications as in PEP 517. The returned dependencies will be installed when calling the associated build_XXX hook.
Target Naming
It might be convenient for the frontend or other tools to know what targets can be built.
Additionally, certain targets might not want to use the build_XXX or get_requires_for_XXX naming scheme.
My initial solution to this is putting something in the pyproject.toml.
Targets are listed in the build-system table.
The targets key is a table containing the target name associated with another table containing an optional build_function and an optional requires_function.
By default, the build_function is build_TARGETNAME and the requires_function is get_requires_for_BUILD_FUNCTION.
Assume that the wheel target and sdist target are included by default, but they can be overritten by the targets key.
[build-system]
# Defined by PEP 518:
requires = ["my_custom_build_library"]
# Defined by PEP 517:
build-backend = "local_backend"
backend-path = ["backend"]
# Defined here
targets = {
shiv = {},
custom_target = {
build_function = "build_my_target",
requires_function = "get_my_requires"
},
other_target = {
build_function = "do_other"
}
}
Alternatively, they can be defined like:
[build-system.targets.shiv]
#build_function = "build_shiv"
#requires_function = "get_requires_for_build_shiv"
[build-system.targets.custom_target]
build_function = "build_my_target"
requires_function = "get_my_requires"
[build-system.targets.other_target]
build_function = "do_other"
#requires_function = "get_requires_for_do_other"
In this example, the backend may provide the build_wheel, build_sdist, build_shiv, build_my_target, and do_other functions.
If a build function is not provided by the backend it should raise an AttributeError when the frontend calls it.
The backend may optionally provide the get_requires_for_build_wheel, get_requires_for_build_sdist, get_requires_for_build_shiv, get_my_requires, and get_requires_for_do_other functions.
If a requires function is not provided by the backend, the frontend should assume an implementation equivalent to return [] as described in PEP 517.
Notes
I am looking for feedback here. I feel like this overcomplicates the pyproject.toml
The primary intention with including the targets key is to provide static information to the build frontend and other tools that might use the pyproject.toml.
I don’t know if this is a desirable feature.
I’m not sure whether or not the build frontend should support building targets that are not listed in the targets table.
I’m not sure if the pyproject.toml should include this information, or if this is something that should be handled solely by the build frontend.
For projects with many targets, it seems like using this might bloat the pyproject.toml.
I don’t want build backends to have to support multiple targets outside of wheels and sdist.
Does including the targets as a key confuse users that lack a thorough understanding of their backend?
Should custom function names be allowed or should the format be restricted to build_TARGETNAME?
This would allow the targets key to just be a list of names instead of a table.
The requires_function is optional.
If the pyproject.toml specifies a specific requires_function and it doesn’t exist, should this be silently ignored?
Should the frontend warn the user?
It might also make sense to support using different backends for different targets, but thought that would complicate this even further.
Something along the lines of this could make it easier to distribute custom backends meant for different targets.
[build-system]
requires = ["hatchling", "shiv_builder"]
build-backend = "hatchling.build"
targets = {
shiv = {
backend = "shiv_builder.build",
build_function = "build_shiv"
}
}
Minimal Alternative
To avoid the complications described above, it might be easier to just set up a basic interface that involves the build frontend dynamically looking up functions from the backend.
This might make implementation a lot simpler and get something like this pushed through the PEP pipeline much faster.
Simply, the build frontend exposes the ability to the user to set a target name, and then attempts to call build_TARGETNAME/get_requires_for_build_TARGETNAME from the backend.
This solution would be backwards compatible and would not require pyproject.toml updates.
A backend is free to use any implementation they like, but I imagine a module level __getattr__ might make a lot of sense here:
# backend/build.py
def build_sdist(...):
...
def build_wheel(...):
...
...
def __getattr__(name):
return do_dynamic_lookup(name)
Another Alternative
One other possible option to putting targets in the pyproject.toml would be to have a function like get_targets which the backend should define which lists the available targets (and possibly their custom function names). If undefined, the frontend should only assume the functions from PEP 517 are defined.
Conclusion
To recap, I just wanted to get the ball rolling on something that I’ve wanted out of python packaging. I’m not super familiar with how this process should go.
I’d be happy to flush this out further since clearly there’s a lot of specifics that still have to be determined, but I wanted to get the general reception of this idea before spending more time working on this.