Packaging and python-for-android

Packaging
1

Coming from https://github.com/pypa/pip/issues/6718.

The python-for-android project, has been making significant progress on making the process of using Python on Android better. I’m preemptively creating this topic, for a broader conversation between the Packaging folks and the python-for-android folks.

My understanding is that someone from the python-for-android would provide us with an overview of how they do packaging, to initiate the discussion; likely elaborating their workarounds/pain points with the current tooling.

4 Likes
2

Hi! I’m @jtc0de on GitHub and I contribute to python-for-android (we often call it p4a), one of my main areas being the pip- and packaging-related code of p4a.

What’s python-for-android? In short p4a is a tool which takes a python project’s source folder that can mostly be just a standard setup.py python package, and packages it as .apk (Android’s package format). In that process p4a collects, cross-compiles, and packages all native code and dependencies (including CPython itself) as part of this .apk. The result is a regular Android app that can be put on Google’s Play Store or similar.

How/why does p4a use pip?
In short: pip doesn’t really know how to make an .apk, and to our knowledge also doesn’t really know about Android or cross-compilation. p4a therefore collects the packages itself (with the help of pip & pep517) and sets up an environment where a pip install can be run that cross-compiles.
In long: I described the process in full detail here: how p4a uses pip

What are our main challenges? Our main challenges that I’m seeing right now are:

  1. There is no library to analyze python packages (and e.g. list their dependencies) without installing them, or to operate pip in general, which led to quite some overhead of code on our part that we need to maintain and take care of although right now all of this is somewhat manageable. However, it does make us sensitive to additional upstream changes which make this process even more complicated for us
  2. Any other unexpected pip or python packaging upstream change that may clash with the cross compilation is difficult for us. Currently that is build isolation, because it seems to be designed such that for pyproject.toml packages it’s apparently not meant to be disabled: doing so will make pip skip build-system.requires which without manual intervention can break package install. But we don’t know how to get build isolation to work in our environment since we install some dependencies with custom patches we call “recipes” which pip doesn’t understand. As a result, build isolation’s per-design reinstall of already present dependencies doesn’t currently work in p4a

What do we hope to get out of this communication? I can currently only mostly speak for myself, but since I do most of the python-for-android packaging code right now my main hopes are 1. being able to give some input on upcoming things that affect us (like build isolation), 2. giving useful input at a point where it suits both sides which hopefully means early enough (how would I do that? What things do I need to keep an eye on / weigh in on and where?) and 3. I’m hoping we can continuously figure out together how to keep our tools compatible so p4a can keep steering towards standard packaging approaches.

p4a has only recently moved forward to actually accept a setup.py for specifying dependencies and moved deeper into more standard ways of packaging, and it would be nice to keep that train moving forward in that direction :slightly_smiling_face:

How to reach p4a developers? I will be following things here of course and relay back what info I can that looks important to other p4a contributors, but to directly reach everyone of the team the best way I know of is to make a meta discussion ticket here: https://github.com/kivy/python-for-android/issues

Happy to answer any questions, and looking for useful input on how to keep some exchange going, and hoping to get some ideas on the build isolation issue :slightly_smiling_face::+1: thanks again to everyone for inviting me over!

Edit/personal note: I’ve also just been looking back through my personal setuptools & python packaging tickets I made, and I may have come across brash and thoughtless in some of them. Sorry for that and I hope nobody was offended by this, python packaging can be overwhelming for newcomers sometimes :see_no_evil: (and I’m saying that as an explanatory contributing reason rather than an excuse, I still should have kept a better tone)

1 Like
3

I haven’t used it (though I know people who have), but does pip-tools help with this?

Yes, this is a somewhat recognized but as yet unsolved problem.

The answer seems to be that you should build wheels for your patched dependencies and “make sure” that they are installed into the isolated environment rather than downloading from PyPI. The command line arguments for this to happen are going to vary based on your specific scenario.

4

https://github.com/pypa/pep517/blob/master/pep517/meta.py can get you the meta data which in theory contains it’s dependencies.

5

With your explanation here, I don’t follow why that’s the case. As @steve.dower says, you could simply build wheels of your patched dependencies, and make them available to pip so that they are the versions that are installed in the isolated environment - there’s no need to teach pip (actually, setuptools) how to build those patched dependencies from source, and there’s no need to reference shared copies.

If you can’t create wheels of your patched dependencies, that suggests a different issue - that your patching process isn’t compatible with the wheel format. And in that case, the solution definitely isn’t to teach pip to handle wheel-incompatible packages, but to fix the incompatibility. And switching off build isolation is not a solution to this - at best it’s a hack to work around one particular way in which not being wheel-compatible is biting you.

6

Hm, it sounds to me that this is possibly somewhat elaborate. We have no packaging expert, I’m probably the one who knows most and I don’t even know how that could possibly work. As far as I know pip doesn’t even know what Android is, and wouldn’t that need to be a recognized target architecture for wheels? Are cross-platform wheels even supported? How will pip know whether to pick wheels for a different platform? All in all this may sound like the cooler idea, but to me it sounds way more involved than just giving up on making build isolation mandatory for build-system.requires to work

I’m not saying that makes it impossible but build isolation is already shipping now and I’m doing quite ugly hacks to even get my stuff to build, and it doesn’t sound to me like something that could be made to work any time soon even if we really tried to.

The thing is also that our current patching is quirky but it’s also pretty simple, nobody maintaining it needs to really understand how python packages work and that is a great advantage for us. (and it also must work without wheels because we use it on completely non-python things like libxml, so we can’t just purely use python packaging tools for the process either, even if they had the functionality.) I understand unifying it might still be nicer at least from your side, but this could pose a significant future maintenance challenge on our side depending on how involved this will be

Edit: also, can all python projects even build wheels? I think panda3d for example for a while couldn’t and you had to use their custom installer - and our recipe wrappers can quite easily deal with this, and we want & need to support all projects out there. So if this is truly a potential issue for some projects then purely based on that it doesn’t sound like a good thing to need to rely on

If there is no other good solution available it still doesn’t sound too bad to me, I have to admit :woman_shrugging:

7

These are exactly the questions I’ve been trying to ask for some time.

Cross-compilation is (as far as I know!) a complex subject, and not one that’s well supported yet by the Python packaging stack. But the one thing that I do know is that the only way we’ll get anything even remotely like a reliable and robust solution is looking at the problem from the bottom up:

  • Forget about whether pip knows what Android is, does setuptools? (If the answer is “no”, then that’s where you should be starting).
  • Are there wheel compatibility tags for Android? (No). As there aren’t, why are they not needed? Does Android only support pure Python code, or are you (ab)using some other architecture tag? (From comments later, I guess the answer is that you’re not supporting wheels :frowning:)
  • Does setuptools know how to build binaries for one architecture on a different source platform? I.e., does setuptools support cross-compilation at all?

Pip’s role in all of this is mostly just to orchestrate other tools. If you try to treat cross-compilation as a pip-level issue, you’re bound to fail. You need to aim at the build tools (setuptools) and the binary format (wheel).

But it’s a major disadvantage when you hit places where you can’t hack the existing tools to work, as you don’t know the design principles, and you’re not familiar with the broader picture. That’s not to say you can’t raise issues and open discussions, but I don’t think it’s unreasonable to expect you to understand the background before assuming you can have a meaningful debate. So while you’ve had the advantage of working in isolation until now, you’ve set yourself up with a quite steep learning curve to get up to speed on what’s behind the design of the packaging toolset.

Handling of non-Python dependencies is a very complex issue - there are a lot of people working on mainstream platforms who are struggling with this - so you’re going to find very little support for that in current standards, and no interest in android-only solutions. On the other hand, more input into cross-platform options, and new perspectives that could help design better solutions, are definitely welcome.

We’re very definitely moving to a world where we expect that. If projects can’t build wheels, the expectation is that we’ll look at what their pain points and help them work out solutions - but no-one is particularly interested (to my knowledge) in maintaining long-term support for environments or projects that don’t support wheels. While there are (and probably always will be) people who would prefer other formats than wheel, that ship has sailed - the standard packaging toolset is built around wheels as a binary format.

Then you’re going to have significant problems fitting in with the standard packaging tools. And as a starter, you’re going to have to do some learning, because you’re going to be finding a lot of people suggesting approaches that start “build a wheel…” - and you’ll need to have much better answers than “we can’t build wheels” if you’re to even find common ground to have a discussion.

Sorry, I know most of the above sounds pretty negative. But there seems to me to be a pretty major gap in understanding, and to bridge it, I firmly believe we need to start with basics. We can get into nuanced discussions later, but for a starter, I think we have to address at least the following fundamentals:

  1. Packaging expects wheels, you can’t build them.
  2. Cross-compilation needs to be addressed at a low level, but you are patching it on top of a non-cross-compilation toolset.
  3. Packaging doesn’t have a solution for non-Python dependencies (yet) but that’s fundamental for you.

One question, which I’m hesitant to ask because I feel like it would be a failure if the standard packaging tools couldn’t support Python for Android, but have you explored conda as an alternative build option? Your goals seem like they may align better with theirs. Even if it’s not a good option for you, knowing how they address some of these issues would give an interesting perspective on where the pain points for you will lie.

Well, given that it directly solves your immediate issue, I can see how you’d feel that way. But it puts a lot of support constraints on pip that don’t align with our long term goals, and your comments here about how you don’t support wheels suggest that we’ll probably implement even more changes in pip in the future that will break your workflow. So the picture doesn’t look anywhere near as rosy from pip’s side :slight_smile:

1 Like
8

Thanks for your extensive response :slightly_smiling_face:

Would conda build all setuptools-based third party libraries out there which users want to use? p4a aims to support pretty much everything in terms of libraries out there (and actually does pretty well so except for build isolation-using things and some projects that use native code) so we can’t really “replace” setuptools if people just use that

Actually p4a does have pip build wheels for many projects, and if we set up our virtualenvs differently wheel caching might even work. But while many projects build fine with wheels, some 3rd party libraries don’t use them. So far we can integrate them with relatively little effort, and if you make this a hard requirement then we can’t. That doesn’t seem ideal to me, and it also doesn’t seem like something we can ever fix on our side

I’m not sure that is universally true. The design is nicer, but it might also make it more difficult for our teams to improve things separately. It will also increase medium term work loads for both sides a lot I think to even get there. And pip/setuptools might be way more fiddly with all this logic in them (for you guys when maintaining the code base, I mean) in the future for just this niche use

I agree. For now we have a system for that, and if we are supposed to use standard tools instead then this is a problem. However, I’m still not convinced the best approach is to try to fully move this cross-platform android & cross-compile stuff into pip & setuptools as you appear to be suggesting

I suppose that is likely. And I get the “support constraints” fear but then again e.g. the current proposed change at hand for build isolation seems not that big to me. Maybe I’m wrong, or maybe the big ugly changes are still in the future, so I definitely get that this is worrying for you

Summed up, I can see where you’re coming from, but I’m not sure deeper level cross-platform integration is realistic in any practical time frame, or even worth the effort. You’d have less external constraints, but more internal ones keeping all the logic working. Therefore I’m also not sure if this is really the way to go, even though I agree the current setup isn’t ideal either (but what is?). I am still thinking maybe making build isolation turned off have less impact by processing build-system.requires anyway might be better for possibly both sides, followed with just seeing in the future how we can keep things working as a gradual process

Edit: I need to add something which I think may also be a misunderstanding: what we are trying to achieve & maintain, as well as humanly possible (which is also why I find making wheels a hard requirement problematic), is to let p4a users and third-party libraries use standard tools, like setuptools etc. You seem to be aiming at also making more p4a internals move to standard tools as well which is a very interesting proposition, but as illustrated above I think comes with its own set of problems

9

Well yes, but over on the pip tracker you clearly stated that this doesn’t work (and your proposed change to pip seems unlikely to be accepted). So I’m not quite sure where you hope this discussion will go next.

Well, if you deliver wheels, p4a users can just do “pip install”. If you can’t build Android wheels for a given project, your users won’t be able to either, so I don’t see how that’s relevant.

10

For a start, you could take a look at what packages conda-forge provides: conda-forge feedstocks

I’m sure not every setuptools-based project out there already has a package in conda-forge, but the breadth of packages available should tell you that conda is able to package a lot of things (including things that can’t really be distributed satisfyingly as wheels).

11

Well, if would be really nice if we could get a consistent message on this issue. Elsewhere we’ve been told that it’s fine if wheels are not adequate for our project, and that the “PyPA” does not actually promote that everything should exist as wheels, so there’s nothing to actually complain about. Here you seem to be saying exactly the contrary, and that you “expect” all Python projects to build wheels.

Someone who reads your message will come up to our issue tracker or mailing-list and complain that we are not producing wheels even though we should (according to this message written by a packaging authority), and we’ll have to painfully explain how much time we lost trying to produce reliably-working wheels.

12

As for relevancy: technically we currently ship the site-packages. So we actually just do “pip install” as you propose (p4a does this internally for each dependency, after our patching was applied if necessary) except that we don’t require wheels for it. So if we did that would narrow the scope of projects that work, that is really all I wanted to explain there. Of course if they use conda instead that might still work, but requiring either wheels or conda still may exclude some projects that currently work

As far as I’m concerned, the idea behind my change is “make pip behave as such that build isolation is not required”. If we added a workaround with the current situation, it would achieve the same result. But of course if your future plans are to make even more things build isolation only then this would break things again.

In conclusion, if pip really wants to move to a build isolation only world, then I suppose moving cross-platform android handling fully into pip/setuptools is the only way. But might it not be much easier to consider treating build isolation as optional for both you and us (in terms of workload, maintaining it long term etc)? I get this would require changes like the one I proposed that may seem annoying, but wouldn’t it still be less work than adding in all the cross-platform handling and patching … into pip itself just for us? But these are just my thoughts, obviously

Edit: it would still be cool btw to have pip/wheel recognize android as architecture. But I’m just not so convinced also moving the cross-platform & patch/workaround hook/… handling into pip/setuptools is a great idea, which would be necessary to make build isolation work as far as I can tell

13

I’m not saying every project needs to ship wheels (I’m not sure if you think “build wheels” means the same as “ship wheels” - if you do, bear with me as I make a distinction between the two below). I’ve snipped a lot of context from your message, but you removed a lot of context from mine as well. Unfortunately (and I think this is at least partly the point you’re trying to make) people do quote things out of context, and often treat quotes without context as supporting their preferred position, even if that’s not the reality at all.

Let me try to clarify. I understand that you are frustrated with the pressure you see to distribute wheels, so I accept that you might not be comfortable with the picture I’m describing below.To be clear, this is my personal view, but if you want to infer “authority” here, can I ask that you keep firmly in mind that I’m offering personal opinions and a personal interpretation of what I hear the community saying. Authority comes from what’s agreed in PEPs, not what I say. (And yes, I know that whatever I say will be taken as “authoritative”, no matter how many disclaimers I add - there’s not much I can do about that except try to be as clear as I can…)

In the context of packaging, wheels tend to get discussed in two distinct contexts. As an intermediate format for transferring files between the “build” and the “install” phases of package installation, and as a public format for sharing package binaries. When I talk about pip “installing via wheel” I’m very specifically referring to the first of these (see below). When people are asking projects to publish wheels, they are asking for the second.

And yes, maybe the terminology is confusing, and maybe that makes the problem worse. If anyone wants to try to bridge the communication gap better, then great. Words are hard, expert advice is welcome.

The key distinction here is around the target platform. Wheels as installation intermediary can be as platform specific as you want, down to the level of “only works on this machine”. But wheels as binary publishing format have to tackle all sorts of portability issues. And yes, I 100% accept that not every project can viably produce portable wheels. If anyone claims that “the PyPA” is requiring projects to do so, feel free to direct them to this message - we are not (treat that statement as authoritative, if you like :slight_smile:). Unfortunately, that isn’t likely to reduce the pressure much, because what is actually happening is that users are asking for binary builds that they can “just install”. The fact that they want wheels is because they use pip (and other tools that consume the wheel format). You can suggest that they switch to conda if providing binaries via conda is viable for you - but getting people to change their preferred tools is always a challenge, regardless of technical merits.

Going back to “installing via wheel”, in that case what we need is not “portable” wheels. Machine specific ones are perfectly fine, and if you can build the project at all, you can produce machine specific wheels. With setuptools, bdist_wheel just runs build and collects the output - so as long as build works, you can get a wheel.

So install via wheel should be a non-issue for nearly all projects (the exceptions will be projects that customise the setuptools “install” command - which is pretty rare in my experience). But being able to install via wheel does not mean a project can necessarily publish wheels. And conversely, projects that only publish sources will very likely still be just as installable by pip regardless of whether we do “install via setup.py install” or “install via wheel”.

I hope this makes things clearer. Sadly, I suspect it won’t. What’s needed is a pithy one-liner that people can tweet, quote without context and make memes from. Unfortunately, not everything in life is that simple :frowning:

(BTW, this is all a complete side-issue as far as Python for Android is concerned. Their problem, as far as I understand it, is around how pip sets up the build environment when building projects from source, and has essentially nothing to do with wheels).

2 Likes
14

I think panda3d for example for a while couldn’t and you had to use their custom installer - and our recipe wrappers can quite easily deal with this, and we want & need to support all projects out there.

i think there’s a great difference beetween getting working python packages to reach a given platform ( most interesting being arm / arm64 in case of p4a/panda3d ), and distributing packaged applications which is Panda3D wants.

Why not starting with an official and universal repl + a pip command on android that can install prebuilt to first let people consider their needs and what is (not) working ?

15

Ah interesting, that explains a lot to me actually. Are these intermediary non-portable wheels guaranteed to exist afterwards for anything installed via a modern pip, even if it uses something other than setuptools? (Like something else commonly specified via pyproject.toml as build system, I’m not up-to-date on other existing alternatives)

Because if these non-portable wheels always exist after first install maybe that could solve build isolation not working for p4a: I’m pretty sure our patching is needed only for building these wheels in the first place, so if pip doesn’t rebuild them from scratch then it could just work. Hence I’m then wondering: 1. are these wheels stored in the site-packages or somewhere else? 2. does pip already do wheel caching and will reuse them if they’re in the right spot? If yes I suspect if we set the paths right build isolation could actually work already. In that case my make-disabled-build-isolation-work-better idea could be entirely unnecessary - only if I am guessing right of course

It did actually for me at least :slightly_smiling_face::slightly_smiling_face: thanks a lot for the explanations and your patience, this all sounds very helpful

Sorry I don’t fully follow where this particular idea is heading, what problem are you trying to help us solve?

16

as far as i remember android is not a supported platform neither for cpython or packaging authority : so it seems unclear to me how to install packages and run test suites on a real device to help buildings those wheels ( which i may need to remind can be built on devices with setuptools and some imagination ).

please don’t say “we can use emulators” it’s almost impossible to run graphical tests ( Kivy/Panda3d/Irrlicht/sdl2/ etc …) and compilation take ages.

i’d like to see - as first steps - something real like a repl or a “pip test env” that p4a could deploy but from an official source of course and with some kind of support ( notably because termux does not support android kitkat and prior )

17

It sounds a little like you want pip on the device, which right now is completely outside the scope of python-for-android* which uses pip on the host pc assembling the apk. So maybe this might be better to suggest in a separate thread? (Sorry I hope I don’t sound too dismissive just trying to keep things somewhat focused here)

*python-for-android really is a specific project for apk packaging, so this thread is not so much about python in general in any way on android

18

hmm not really “pip on the device”, i want pip to install packages on the device and run test suite on it ( obviously after p4a deployed a sufficent test env )

I don’t really care where packages get built nor with which toolchain/recipe ( crystax is still usefull in some cases), more like where they can run fully tested in order to prevent posting half compatible stuff for a wide audience.

i think there’s too much ways to embed cpython on android. so it may need some test framework and a connection automator independant of sdk/ndk/deploy tools used

see a real use case :
eg upip from micropython can “cross-install” packages
https://docs.micropython.org/en/latest/reference/packages.html
and testing happens via the serial port linked repl, ( but adb the android debug bridge can be far more powerfull )

19

Did you file a feature request for this already on the issue tracker? That seems like a good spot to talk about how feasible that is

20

@pf_moore if you get around to it, some input on this post I made before the slight detail would be possibly very useful to me: Packaging and python-for-android

I’m not opposed fundamentally to at least exploring the idea of pip learning cross-platform and Android related things just to stress that again, but it just doesn’t seem useful for a short/medium term fix regarding build isolation + p4a + unusual build_requires breaking right now, which together with Cython breaking setup_requires is a real problem in my eyes.

So it would be nice to figure out some intermediate solution for that which is nice enough, I suppose p4a parsing pyproject.toml’s build_requires manually could work but is this really the best approach? If with wheels caching this can work in a more pip-controlled (rather than p4a-cnontrolled) way if we just set up the folders right, that’d be nicer