Custom build targets (for cleanup) post PEP 518

I maintain several internal packages and 1 public package, and while I understand why direct invocation of setup.py is deprecated, I’m also used to Java builds where “cleanup” just works. I have a custom “purge” command I copy into setup.py in each package I maintain.

I have two questions:

  • Does the build package or another have a way to clean my working directory/repository clone?
  • What is the future direction for custom commands like this?

Hi @danizen, I think currently the best way of doing that is via tox or nox, or any task runner you might already be used to.

So, you think of tox/nox as a task runner rather than a purely testing tool?

Definitely.

This is a nice article about that: Managing a Project’s Virtualenvs with tox | seanh.cc

It is super common to see tox/nox usages for tox -e docs, tox -e lint, tox -e typecheck, etc…
I personally also use it for cleaning things.

1 Like

The way I understood it, the “direct invocation of setup.py is deprecated” bit is mostly about the install (including develop but it is a somewhat different story), sdist and bdist_wheel commands. Nowadays, those are solved better with PEP 517 and co. (python -m pip install ., etc.).

But from my point of view the actions you are mentioning are not covered by this.


This is up to the development tools, and most likely the tool that brings the build back-end, since I guess it is the artifacts created by the build back-end that we want to clean here.


I do not know of any modern packaging tool that has a “clean” command. All tools have a different set of features because there is no standard here. But some packaging tools can be extended.

For PDM:

For setuptools:

For Poetry:

Personally I tend to use Makefile as some kind of task runner to do these kinds of things (cleaning build artifacts, etc.), I find that using tox for this is uncomfortable. And then there are some actual Python task runners:

Thanks all. There are issues here that have to do with the my organization, but my questions are answered.

Personally, I prefer to use conservative backends that will only include in the sdist the correct source (eg. whatever is committed into git) and then use python -m build, which will build an sdist first and then use it to build your wheel, thus preventing any unexpected files to leak or mess up the build process, making cleanup unnecessary.

2 Likes

Like from .gitconfig:

[alias]
    distclean = "clean -dxffi"

FWIW, the next release of PDM will support pre- and post-scripts like npm. This is similar to what taskipy does and will make PDM’s task runner more flexible and powerful. Stay tuned.

1 Like

Hatch v1 supports this as a core feature: Environments - Hatch

The Hatch CLI rewrite is still in beta so you’ll need to do pip install -U --pre hatch.

How similar are all of these approaches? I would be interested in the community adoption of these features because if there’s similarity between them then it starts to suggest a PEP might make sense to standardize how to write the tasks down.

It’s great to have experimentation, but I agree that where there is commonality, it may make sense to standardize.

I was thinking about something similar in a different context. A lot of tools (e.g. flake8, isort, and black) support maximum line lengths. Each tool sets these in a different way and if you’re using all three tools, you can get into trouble if those values don’t match. I’m not sure there are enough other similar settings across the tool landscape, but it sure would be nice to only have to set line length once.

1 Like

It looks like conda’s meta.yml has support for defining tests. Basically you can specify what to have installed, what to import, and then what shell commands to run.

Just to clarify for others, the imports section is just for a simple smoke-test of package import, independent of the other items; the commands section is where the test command(s) requiring the test dependencies in test.requires are run (alongside simple stuff like pip check). Though unfortunately, many if not most conda recipes don’t run the package’s actual test suite, either due to them being based on the PyPI sdist instead of the git tarball and the sdist not shipping the tests, or simply the maintainers not adding them.

Barry, this is sort of the reason I raised it here. The current plurality of tools is OK for me, where I’ve used make, cmake, ant, maven, npm, yarn, Rakefile, tox and what is one more. However, I am a bit of a leader where I am, I cannot tell my manager that everyone should “check out poetry” about tools like this. I am looking for a mature solution I can treat as a best practice.

In the end, I did create a tox environment, but it feels rather like a square peg to me. The reason is that for removal of files I don’t really need a virtual environment at all - allowlist_externals and rm are all I need.

I would like to see the Python packaging community do a bit better to define in a uniform way the tasks associated with building a package - and clean-up is important even when storage is cheap. Organization of storage is still not cheap :slight_smile:

1 Like

Are you interested in making a proposal to add a hook definition for pyproject.toml that build back-ends can be called with to invoke cleaning up after themselves?

1 Like

I totally get this @danizen - at my $work we are looking at moving toward much more native build environments, i.e. so that language experts from different communities (e.g. Python, JS/TS, Go, Rust, etc.) can come in and use tools that feel very similar to those developed in open source. So while I have my preferences there, we will I’m sure need hooks to talk to our internal systems at various phases of the build process. At least with PEP 518 and related tools, I think we have a principled way forward rather than the hack-upon-hack we’re used to.

FWIW, if you’re using a task runner like tox, you might be interested in looking at nox as well; which uses a Python program for handling task running.

Just to clarify for @danizen others, both Tox and Nox are Python packages written in Python. What @pradyunsg is referring to is the fact that Tox is configured declaratively in a tox.ini config file, while Nox is driven by a noxfile.py script.

Feel like I must mention that tox v4 released later this year will also support configuration via Python under toxfile.py :blush:

2 Likes