Should packaging understand platforms?

Packaging
1

I had a package, and I ran it on Linux, and it required the xattr.py package. I then tried to pip install it on windows. the xattr package tried to build and failed to install with a build failure.
Was using setup.py at the time. the xattr package was half-installed, and would not remove.
looked in xattr’s setup.py and noticed:

    ...
    platforms=['MacOS X', 'Linux', 'FreeBSD', 'Solaris'],
    ...

So my question is, why didn´t the packaging stuff notice that this package doesn´t work on windows and refuse to try?

Originally submitted as a about setuptools, but they don’t think it is relevant:

From the other bug report, it looks like a few others ran into the problem, so perhaps worth discussing.

Thanks.

1 Like
2

The platforms field doesn’t have a standardized list of values, so that wouldn’t be possible. Even if values were standardized now, it wouldn’t affect existing distributions that don’t use them. Additionally, it would be very difficult to fill in the field accurately, since it’s unlikely that you have tested on or even know of all potential platforms. The only way to really know is to install it, building from sdist if a platform wheel isn’t available.

In this case, the package you were installing should have used a platform expression in its requirements to say that xattr is only needed when the platform is not Windows.

The detail that xattr only partially installed and couldn’t be uninstalled is separate from this and should be reported as a bug to whatever installer you were using. Note that setuptools should not be used as an installer, only as a build backend, so if you report that error to setuptools it’s unlikely it will be fixed.

2 Likes
3

I think this mostly arises with packages that have C in them (xattr being a good example) in cases like that, they just won’t work on platforms that the developer hasn’t tested on. and in the case of xattr, the developers are clearly saying, by omission, that it doesn’t work on windows.

My project is completely portable and so does not list any platforms. When the developer takes the trouble to indicate which platforms it works on, the packaging system should at least notice that the current platform.system() (the actual field listed in this case.) isn’t in the supported set, and maybe install anyways if there is a flag like --ignore-system-platform

The fact that the data is not standardized is a by-product of it not being used. If the packaging honoured it, then over time, it could converge.

4

The real problem seems to reside here (“half-installed, and would not remove”), not in the fact that it tried to install at all.

1 Like
5

The developer says it’s supported on mac & linux through the platform metadata, and is saying it is known not to work on windows. So what the installer does on windows is inherently undefined. Should not the installer understand that and refuse to perform undefined behaviour?

6

As others have already said, Platform metadata is not used as part of the resolution process for determining what to install on a given system. If you feel that it should be, feel free to write up a proposal for how it would work - but please be aware that it’s likely to be a lot harder than you think it is.

2 Likes
7

There is nothing “inherently undefined” about not working due to the system not providing the required functionality: it seems quite well-defined to me. The main problem, again, seems to be that the installer leaves the Python environment in a half-broken state instead of cleaning up.

So IMHO you should actually report that bug upstream, where it has a decent chance of getting fixed; while the problem of not proactively refusing to install has, as others pointed out, a much smaller chance of getting fixed in the middle term.

2 Likes
8

When installing a package, pip looks for dependencies and satisfies them, or refuses to install if it cannot. It requies a switch ( --no-dependencies ) to install without them, and then it is understood that the package behaviour when installed without it’s dependencies is undefined… it could partially work, work most of the time, or not at all. It totally depends on how the dependencies are used in the package.

The platform is another kind dependency, if pip is installing a package that lists what platforms it works on, installing on some other platform is similar.

9

in terms of “leaving the python environment in a broken state” … if the developer has stated that this package does not install on the platform where it is being run, he needs to test on the platform he doesn’t support anyways to figure out how the installation will fail and cleanup properly? Really?

It doesn’t sound reasonable to make the package maintainer responsible when pip triggers installation of a dependency in an environment that is documented as unsupported for that dependency.

(EDIT: trying to fix clarity complaint in last paragraph.)

10

As I said, please feel free to write a full proposal (and better still, some working code in the form of a PR) to define precisely what you mean.

What you’re saying here is barely understandable to me, because you’re using terms either incorrectly or far too generally. For example, “platform” in the context of package metadata is an arbitrary string. What precisely would you expect pip to check, in order to decide whether it’s OK to install a package that says platform = "Windows 10+ 64-bit"?

Packages don’t run installers, users run installers. The user is absolutely responsible for not running an installer on a package that doesn’t work on the platform they are installing on.

If the user is installing a package that claims to work on a given platform, the author of that package is responsible for making sure that the claim of support is valid, by ensuring that they only depend on packages that work on that platform. For cases where a dependency is only needed on some supported platforms, markers are available to encode that in a machine-readable form in the dependency. If the package author isn’t using the correct markers, they are responsible for the fact that their claim to work on a given platform isn’t true (just the same as they would be responsible for ensuring their code works on all platforms they support).

6 Likes
11

A user is trying to install package A, which depends on B, which depends on C.
C has a platform restriction. So you are saying the user should look at the entire dependency tree of package A, and B, to figure out that C won’t run before trying to install A?
Does not sound like a reasonable expectation.

On writing up a proposal… what format of proposal would make sense?

I downloaded the pypicache and looked at the platform entries:

  • 86336 total packages… the platform field is present in all (so I guess it’s mandatory?)

  • 73053 have None

  • 5341 have ‘’

  • 3280 have ‘any’

  • 2153 have ‘UNKNOWN’

So those values should be ignored, or not a constraint, for at least 83827 or 97% of the
packages in pypi.

For the remaining 3%, it is indeed quite variable… we have still more ignore type settings:

  • plat=‘Posix; MacOS X; Windows’ 243
  • plat=‘OS Independent’ 121
  • plat=‘Platform Independent’ 33
  • plat=‘ALL’ 38
  • plat=‘Any’ 178
  • plat=‘Operating System :: OS Independent’ 6
  • plat=‘All platforms’ 5

So for another 624 packages it should still be ignored.

The rest is all a huge mess of around 200 different entries, which lots of them occurring once.
I understand the reticence. The starting point isn’t good.

There are wacky ones like:
plat=‘Probably any platform that uses floating point that kind of resembles IEEE 754.\r\n Tested on Linux.’ 1
plat=‘Tested on Ubuntu 22.04’ 1

So I get that the source information is free form and unspecified, but is that not something that can be improved?

I guess one would have to start with saying what should be in the platform package metadata field. Either structuring it a bit more, or asking
for additional fields. perhaps print a warning for the first few years "freeform platform entries deprecated: please follow some specification
After a few years could then start enforcing/using the platform info.

Is this the right place to ask about the definition of platform, and what should go there?

I looked here: Core metadata specifications - Python Packaging User Guide
But I’m not even sure if that is the same information as what is in pypi, and it seems
like one can have many platform entries which the pypi cache entries do not have.

12

There isn’t any kind of automatic rollback mechanism, if that’s what you mean. Pip wouldn’t be able to tell whether an installation “failed” in general; it could only possibly know about e.g. an error in a subprocess that was invoked to compile something (and then only if it plays by the expected conventions about process exit codes).

I’ve never tried this, but theoretically it should be possible to set up your site-packages folder as a Git repository and commit changes after a verified “successful” installation. I suppose it could lead to some bloat if you are installing a lot of the kinds of things that do require local compilation, because Git is optimized for handling text.

It would be hard for the A author not to know about this, because the information would back-propagate through documentation. The B author would know about the restriction due to the research needed to choose C as a dependency in the first place; assuming this is acceptable, the B author would become aware that this limits the environments where B can be installed, and document it. Then the same process would play out for the A author.

1 Like
13

It would be hard for the A author not to know about this, because the information would back-propagate through documentation. The B author would know about the restriction due to the research needed to choose C as a dependency in the first place; assuming this is acceptable, the B author would become aware that this limits the environments where B can be installed, and document it. Then the same process would play out for the A author.

if only there were some kind of metadata that would track this information so that this process could be automated…

14

Reading this thread it sounds like this is probably a bug in xattrs that somehow isn’t specifying the right platform tags/markers. I can’t say how they would fix that though, because unsurprisingly the pypa/setuptools docs are opaque and inscrutable. This issue seems maybe relevant?

I see that you commented on an issue on the xattr tracker and the maintainer said “xattr does not claim any Windows support in its distutils metadata or anywhere else” and suggested it was a distutils issue. If you got your installation errors by just running pip install xattr, it sounds to me like what the maintainer said is wrong or at least misleading, since “not claiming” any particular platform support in the metadata presumably is equivalent to claiming it works on all platforms. Probably there is a way for them to fix that in metadata that is actually considered by pip, but I’m not sure what that is.

15

No. In this example, B would use structured markers to declare that C is an optional / platform specific dependency. When the user installs A, the installer resolves dependencies and evaluates markers as appropriate to install the relevant dependencies.

Do you have an example where this isn’t working?

16

No, I’m saying that if B claims to support the user’s platform, then B’s author is responsible for not depending on C when installed on that platform. Similarly for A and B. The user is only responsible for confirming that A is supported on their platform (and trusting the authors of A, B, and C, obviously).

Precisely. You’d need to start by standardising a list of valid platforms. And that means deciding what a “platform” is. Is “Windows 10” a distinct platform, or is it just “Windows”? What if a library only works on Windows 10, because it uses an API only available on Windows 10?

There is no current definition of “platform”. It’s free-form, as you found out. A proposal would need to fix that. And would need to address backward compatibility - what platforms should we assume a package with plat=‘Probably any platform that uses floating point that kind of resembles IEEE 754.\r\n Tested on Linux.’ would work on? You can’t just say “it’s not valid” - it exists, and it’s not going to disappear just because it’s inconvenient.

The reason packaging doesn’t already do this is because it’s really, really hard to do. We didn’t forget. It’s not that we don’t care. You’re not pointing out an issue that no-one has noticed. You’re welcome to have a go at solving it, but most of the insights you’ll get from the community will be ways in which what you propose isn’t going to work. If that’s going to be demoralising for you, you might want to rethink whether it’s something you want to tackle. And be prepared to invest a lot of time into this - it’s not something you’ll be able to solve with just a week or two of work.

4 Likes
17

The maintainer is wrong in this. If xattr doesn’t support Windows, the error is yours for trying to install it. Just like you’d be responsible if you tried to install the fish shell (which also doesn’t claim to support Windows) on your Windows PC.

If you didn’t install xattr, but installed something else which (a) claimed to support Windows, but (b) relied on xattr when installed on Windows, then that package’s claim to support Windows is clearly wrong.

5 Likes
18

Just to expand on this, if B is cross-platform, then they are presumably testing across platforms and will have adapted their code to tolerate the absence of C. They can then include an environment marker in their dependency list:

dependencies = [
  ...
  "C; platform_system != 'Windows'",
]

Now anybody who includes B as a dependency does not need to worry about pulling in C on Windows.

If B is not cross-platform, then the author of A should do the same dance, but with B:

dependencies = [
  ...
  "B; platform_system != 'Windows'",
]

Just adding this, since this discussion has been more about what we might wish the platform metadata could do, but there are no explicit examples of how this is actually solved.

3 Likes
19

I think the maintainer did the best he good given the metadata available… he has a platform field in xattrs’s metadata that says: platforms=[‘MacOS X’, ‘Linux’, ‘FreeBSD’, ‘Solaris’], which is consipicuously omitting Windows. What I’m learning is that there isn’t any way for the packager to specify that it doesn’t run on windows… and I’m trying to suggest it would be good if there were some way to do so.

I’m gathering that the core devs are saying “yes, we know”… oh… ok… I get that it isn’t an easy thing now, I guess it might be good to put on a roadmap or something.

20

Right, but the question is what does C do if they want to say “this package, C itself, is not available on Windows”?

I think the answer is “don’t provide a Windows wheel and don’t upload an sdist”. So this is really another case where the problem is sdists being available and searched by pip. The problem is that there is no way to say “pip, this is an sdist, but for resolution purposes you should treat it as a wheel with the following set of platform tags, and ignore it if the platform doesn’t match those”.

I think that’s going a bit far. :slight_smile: If you tried to install fish in an entry-level manner comparable to pip install xattr, you wouldn’t wind up with a bunch of inscrutable build errors. You just wouldn’t find a file to download for your platform, or if you downloaded it and tried to run it it would say “this isn’t a valid Windows executable”, or something along those lines. And I think something like that in this case would be fine, like if pip install xattr just said “package not found” or “xattr is not available for your system” or something. If you tried to build fish on a Windows system you’d probably get a build failure, but the only reason that happens here with xattr is that pip thinks “install” actually means “install, or maybe try to build”.

I’m not sure all this stuff about “platforms being hard” is really the main issue here. There’s already a system of platform tags for wheels and they work fine — well, they work okay. :slight_smile: It’s only when sdists start jumping into the mix that we have these issues. My hunch is that if xattr didn’t publish sdists, you would get a normal “package not found” error on windows, just like I do if I try to install pywin32 on linux.

1 Like