Github tarballs are also not signed, while git tags and commits can be PGP signed. In fact, all my projects use signed tags for release.
Git does support naming a tag or branch like this, but GitHub will always resolve that URL to the commit (you need to add refs/heads or refs/tags to access the branch/tag), so it’s safe to reference a full Git commit hash. Of course there are still multiple other problems with this approach, as already mentioned.
One interesting thing of note is the spec for declaring extras explicitly reserves test and doc for listing the dependencies necessary to run your test suite and build your docs, respectively. So if we can toss in a way to specify how to do either we could get sdists which include PKG-INFO using a new-enough metadata version to be relatively self-contained when it comes to this stuff.
I also wonder if build tools should consider stripping out test and doc from wheels which don’t contain the test and doc directories necessary to actually work?
1 Like
I really wish they would also reserve tests and docs, and more generally, anything that presumes or specifies the singular should also accept the plural. It’s just too hard to remember which ones are acceptable/default and which ones aren’t.
1 Like
I believe you are familiar with the PEP process if you wanted to see this change. 
(You might also just propose it and get enough consensus here as a separate topic to skip the whole PEP thing.)
Yeah, I noticed that when reviewing the current spec in the context of PEP 685. Perhaps we could, for example, specify in the sdist spec that any directory named (test|doc)s? in the project root SHOULD automatically be included in the sdist unless explicitly excluded; that would be fairly straightforward. Tests in the (import) package source tree are already included; the one problem is people using top-level tests? packages rather than either having tests? in the project root, or at least one level underneath the top-level import package, but that is still a problem either way.
A much less trivial issue is, I imagine, getting consensus on standardized hooks for invoking tests and docs builds, given the many times this has come up previously without a clear resolution (yet) and a diversity of implementations in existing tools.
If I understand what you’re suggesting correctly, this presupposes a certain naming and layout of tests and docs in the source tree, which is far from universal (particularly for tests)—other than deprecated, and removed/soon-to-be-removed old distutils commands, is there anything that formally specifies this?
Indeed, though by the same token, it might make sense to formalize the also-very-common dev and (perhaps) lint extras, as defined to mean any dependencies needed to run the project’s linting suite (with perhaps an eye toward future standardization on that front), and all optional dependencies needed for project development (docs, tests, linting, and any other such functionalities), which if not explicitly defined could be specified to be interpreted by tools as tests? | docs? | lint—though perhaps that’s a stretch too far.
@pf_moore , any thoughts on this?
Personally, I wouldn’t use any of tests/docs/dev/lint extras in my projects. In my view, extras should be for additional runtime functionality only, targeted at end users, not for developers of the project. As I’ve already said, I view development activities as something that should be done from a git checkout of the project, using the tools the project maintainers prefer (so, for example, tests should be run using something like tox, with the test dependencies specified in the tox config file). I don’t think we should be trying to mandate development processes in the way you’re suggesting.
But please understand this is only my personal view. I wouldn’t try to impose it on others - if some developers want to use extras for this purpose, then that’s their choice. (But conversely, don’t expect me to follow conventions like this in my own projects )
6 Likes
There’s definitely plenty of people who prefer using requirements-dev.txt, lockfiles or other tools to manage groups of development dependencies, as opposed to using extras for that purpose, and reasons for each. Given the diversity of opinions, we would certainly not mandate that projects take the latter approach.
To note, all of this would still be done from a git checkout, of course; at its most basic level, you’d just specify pip install -e .[dev] instead of pip install -e .. But more abstractly, one could thing of this as a standardized means of specifying the dependencies necessary to run tests, build docs, or execute a linting suite instead of in bespoke tool-specific config (requirements-XXX.txt, tox.ini, lockfiles, tool.*, etc) or defining a whole new mechanism for this. Tools can then either call an external installer on the extra, or consume and handle it themselves, without every tool having to have their own unique format and conventions for doing so.
It wouldn’t be required that devs adopt this, but it would enable interoperability between various tools without a coming up with a whole new format. But maybe there are compelling use cases that would make the latter worthwhile.
EDIT: @brettcannon maybe it would be worth splitting the discussion from your post on into a separate thread, to make it more manageable and visible?
I think that’s dictating a project’s layout a bit too much (e.g. should you be using src/?).
It might make sense, though, to just say people SHOULD include the docs and test code in their sdists.
I was actually referring to the extras, not the code. I already assume most build tools leave docs and tests out when putting together a wheel.
The issue with that view is folks taking your sdist and trying to repackage it (e.g. Linux distros) have a harder time if this information is left out. It also potentially hurts verification of wheel builds.
I personally don’t think so as this all ties into whether you should include your docs and tests with your sdists. If people want a discussion that tweaks/expands then they can do a separate discussion. My key point in all of this is the specs have support for folks shipping tests and docs in a way that they can be utilized from an sdist in a way that manual intervention is minimized.
1 Like
I guess so, though that would be aimed more at tool developers rather than project authors, in that tools should ensure that, by default, docs and tests are included if present for the great majority of existing projects that have them. But perhaps that’s overly prescriptive for a specification and better handled by choices made at the individual tool level, and its better to just state that “Documentation and tests SHOULD be included in sdists” without specifying particular mechanism for doing so or explictly stating what is the responsibility of the tool versus the user.
And maybe, if test invocation and docs building is standardized, as part of that test and/or docs directories could be specified as well, which would automatically get included in sdists and wheels (without some automatic mechanism, it seems likely that substantial numbers of sdists will get shipped that cannot actually invoke docs or tests via the specified method), but perhaps that’s overcomplicating it.
Ah, I see. For tests and docs outside the package I’d assume that to be generally true, though not necessarily for tests inside the package (at least for setuptools).
That only applies if you’re shipping the tests in your sdist. In which case, you should also be shipping your tox config (which contains the dependencies anyway), surely? Otherwise consumers still don’t have enough information to run the tests without checking the source repo.
I should say that to an extent, the only reason I’m still contributing here is that I was drawn in by the comment
As someone who resists adding tests to my sdists, I still feel like no-one is explaining what exactly people want when they say “please include tests in the sdist” (it’s now expanding to cover including test requirements, for example) or why it’s important to them. Honestly, I’m not even clear in some cases why I should even want to help people to repackage my code in the first place!
maybe the discussion is now more about, assuming sdists should include docs and tests, then what does that imply? In which case I don’t really have anything useful to add.
3 Likes
There are downstream repackagers which consume your package, such as Linux distros, but folks like corporate consumers fall into this category too, as do other use cases. It’s extremely helpful to include tests in your sdists because they can sometimes do wider dependency checks than you might do when creating your wheels. Back when I was Debuntuing, we’d do distro-wide rebuilds, and library test failures or dependency test failures often were an indication of bugs in new releases of upstream libraries. I guess it all depends on whether you care about that or not.
4 Likes
Another big use case is Conda packages, which is by a large margin the most popular distribution mechanism in the scientific Python ecosystem, one of the largest and fastest-growing communities of Python users today. While it is possible to source conda packages from GitHub rather than PyPI sdists and manually specify the test dependencies in order to run the test suite as part of the package build process, on the coda-built and installed package, as others illustrated above there are significant advantages to doing the latter.
Particularly as someone who cares a lot about standards for overall ecosystem interoperability rather than tying things to however one particular tool does things, I find it interesting that you say
when that requires one particular tool, when there are other modern, popular options (e.g. Nox) and the Tox isolated environment workflow, while useful for testing the project code itself, is not what a distro, conda or other downstreams would use to test the built package integrated into their whole stack/ecosystem. While standardizing a method of specifying test and doc dependencies may not get us all the way there without a standardized hook to invoke the projects testing/docs tool of choice, it seems to be a step in enabling such, and complement such well (as is also being discussed to some extent here).
1 Like
Thanks for the explanation. It expands on my limited understanding. But part of why I’m unsure is that one case I’m specifically aware of is the editables package. That is only intended for use in two cases - for build backends (where it will be installed along with the build backend into a transient build environment) and as a runtime dependency for editable installs. In neither of those two cases do I see a real use case for having a distribution-packaged version of the library, as it’s not something that should get installed in your system Python. So why should I follow this advice in such a case?
That’s really the crux of the issue for me. I understand that there may be reasons people want to repackage, but rather than blindly doing stuff that makes my development workflow more complex[1] I’d like to understand and judge whether the requested changes are worth it to me and my users. And not knowing anything about why people repackage makes that hard (especially when people start taking it as a matter of faith that “tests in your sdist” are unequivocally a good thing).
Same question again - why do people want to conda install editables? The PEP 660 discussions never touched on conda, so my assumption is that they don’t use that mechanism at all.
Again, my point here isn’t specifically about editables, but generally about the fact that this should be a per-project decision. Maybe we should have a section in the PUG covering “laying out your project to assist redistributors”, which could go into this sort of thing in more detail? Or maybe the community consensus is that we should include tests in the sdist, and we just document that and I’m just an outlier. (I did a quick check of some PyPA projects, and it appears that pip is the only one that doesn’t include tests in the sdist - so that might explain my confusion here, because pip is the project whose release process I’m most familiar with).
I use tox as my example precisely because that’s how I prefer to run my tests for this one project. There’s a lot more in my tox config than just test dependencies - coverage reporting, for example. Switching test runners involves a lot more than just “use nox rather than tox” and I want anyone running the tests for my package to be getting the same results as I do - so they need to use the same tools and setup.
(FWIW, I had the same reservations about PEP 621 - metadata in pyproject.toml - which gives similar promises of allowing you to switch build backends easily, and yet how you specify your metadata is only a small part of the story).
I do think that if any consensus comes from this discussion, it would be good if tools like flit init, the PyPA sample project, and defaults for build backends like setuptools, made that consensus dirt simple to follow. At the moment, I find that I waste a lot of time at the start of any new project setting stuff like this up - or I just copy something I’ve done before, therefore perpetuating out-of-date approaches.
At the moment, adding tests to my sdist is extra work for me. I need to look up how to do it in setuptools, test that it’s working (and how do I test that I got it right, given that I don’t personally run my tests from a sdist?) and maintain it when I change things. For projects as small as many of mine are, that’s a significant proportion of the total maintenance work…
Although the added complexity is not huge. I don’t want to make this seem like a bigger issue than it is. ↩︎
4 Likes
Personally, I’d think the non-trivial time, cognitive complexity and risk of having to think through all the possible downstream use cases for your project (not to mention having to explain that analysis to those who request such, and having this whole discussion) to determine if it is a niche case that can get away with not shipping the tests, etc. in the sdist without significant impact to downstream redistributors would greatly outweigh the time it would take to add a couple lines to your MANIFEST.in, or approve a PR that did the same. This would be particularly true with most modern build backends that include everything by default or at least make it much simpler to do so than in the past, where it would take actual effort and complexity to exclude them.
As a general principle, I tend find it to be both more effective at ensuring reliable outcomes and a more efficient use of developer bandwidth for humans and tools to just default to doing “the right thing” that “just works” for any likely case, even if it is not strictly necessary for all of them, rather than spending the cognitive and/or code complexity to analyze and decide whether it is possible to get away with another approach, and the risk of judging incorrectly, unless there are substantial practical downsides to not doing so.
If its needed “as a runtime dependency for editable installs” using pip, then conda (at least CF) would want to package it, as otherwise it would mean injecting non-conda-managed deps into the environment every time pip install -e is used.
Right, but its important to recognize that, at least in terms of the rationale presented, editables is really quite a special case. For the great majority of packages, especially for important ecosystems like conda, these reasons don’t apply, and in general, the number of packages where they actually do is far outweighed by those where there is a benefit to doing so, but they are not included, whether due to the maintainer judging incorrectly or not making a judgement at all (if they aren’t using modern tools that include everything checked into the VCS by default, or either tools or a directory structure that automatically includes tests, etc).
After being convinced myself by the reasons presented by @brettcannon and many others on this thread, I it makes sense to simply adjust the default recommendations to include tests, etc. in the sdist (or use a tool, e.g. flit, setuptools_scm, etc that includes all VCS-tracked files), rather than requiring uses (especially beginners to packaging) to worry about whether their package falls into one of the fairly narrow special-case exceptions.
Yes, but I’m a little unsure how this affects the points I was making, sorry. The primary purpose of downstream testing is not to test the upstream’s code, but rather the downstream’s packaging and integration of it. As such, running tests on the project’s source tree or using an isolation mechanism like tox, nox, etc., rather than testing the package as built and installed Into the downstream’s environment would rather defeat the purpose of the endevour.
Also, I didn’t mean to claim that the configuration of tox, nox or other automation systems should be interoperable (ala PEP 621 for metadata) but that there is no standardized hook to to invoke whatever backend the project specifies to run its tests (or build its docs) and retrieve the dependencies thereof (analogous to PEP 517 for building the package itself).
Fully agreed there. As far as I’m aware (and I checked the docs for each of them), most of the other modern build backends (Flit, Poetry, PBR, not 100% clear on PDM, as well as Setuptools with the increasingly widely used setuptools_scm) include all VCS-tracked files in the sdist by default, so it “just works” in those cases.
Furthermore, besides that, pretty much all of those tools allow including and excluding files by just specifying globs in include and exclude keys in their tool section of pyproject.toml; with no need to deal with the confusing cacophony of MANIFEST.in, include_package_data, package_data, data_files, etc.
And IIRC there is an intention on the Setuptools side to simplify that as well as implement smarter default behaviors that should cover most common cases with no extra config, connected with some of @abravalheri 's work implementing PEP 621 support.
1 Like
Can you provide some detail about why “tests in your sdist” is a bad thing? Or is it “just” that it changes your workflow or config files, etc. in order to add them? I don’t particularly buy the space saving argument these days. Disk space is cheap and networks are fast, so – outliers aside – including tests (and docs) in the sdist seems like a win for your consumers to me. It’s easier to ignore them if you don’t care than to have to figure out how to get them if you do.
For sure, and it probably always will. I think we’re talking about best practices, defaults, and conventions.
That’s pretty much a common problem. Like for example, we have templates at work that create new projects. They work pretty well in practice for creating the project, but then what happens when the best practices embodied in those templates change? Well, now you have to go through an ecosystem-wide change management procedure to update every repository.
1 Like
(Responding to this point alone) I thought flit had stopped doing this? If it hasn’t, that means that the sdist will contain irrelevant stuff like .gitignore or .github.
I was seriously considering switching some of my projects to flit precisely because it no longer insisted that “everything in VCS” matched what should go into the sdist 
But we’re getting even more off topic now. I’ll shut up at this point.
Just the changes needed (and any maintenance of the config involved, which I fully expect to be minimal). Part of those changes, of course, involve testing whether they have worked correctly - which means working out myself how to run the tests from the sdist.
I’m not pretending it’s a big chunk of work, just something I don’t particularly want to do.
Agreed the space is not really a big deal, although I will note that pip’s test suite is 50% bigger than pip itself 
Ignoring the fact that you’ve already injected a non-conda-managed dep by usingpip install -e, and the fact that the pip install -e will install editables using pip if it’s not already been installed manually, this suggests that maybe I should package the runtime component of editables separately.
1 Like
Given the non-trivial scale of daily PyPI downloads and unpacks (particularly on CIs), I would actually be somewhat sympathetic to that argument, except that we’re only talking about sdists here, which only comprise a tiny proportion of such so long as projects offer wheels, either pure-Python or for common platforms (and if they don’t, they probably have bigger things to worry about than a modest space savings).
At least according to the docs, this is true for the filt_core backend as invoked by other frontends, but not flit’s own flit build and flit pulbish frontends.
Getting myself back on topic, similar @barry 's question, is there really such a harm in including perhaps a few hundred to a few thousand extra bytes in a ≈few hundred + kilobyte sdist that justifies spending non-trivial time manually configuring includes and excludes, much less choosing a whole different build system? Maybe there’s something I’m missing here, but this sure seems like purity over practicality 
especially if you’re concerned about
for manually including tests/docs, eh?
That’s true, but its much less of a problem (for conda, at least) when its the package in question being actively developed, and at the very tip of the stack, as opposed to a common, low-level runtime dependency, for reasons that wander somewhat out of scope of this thread, and it at least gives conda distributions and users the option of (pre) installing with conda.