PEP 730: Adding iOS as a supported platform

PEPs
1

Presented for your consideration: A PEP proposing the addition of iOS as a Tier 3 supported platform.

Any feedback on this PEP would be appreciated.

UPDATE: 23 Oct 2023

A revised version of the PEP has been submitted; a preview rendering is available. Barring significant feedback, my intention is to submit this to the Steering Council for ratification towards the end of next week (i.e., around Nov 3).

UPDATE: 24 Oct 2023

The PEP revisions have been merged and published on peps.python.org.

UPDATE: 30 Oct 2023

There’s been one piece of substantial feedback, raising a possible approach to avoid the use of the deprecated --undefined dynamic_lookup option. Steering council submission is on hold while I investigate this further.

UPDATE: 9 Nov 2023

The PEP has been revised again; preview rendering is here. Barring significant feedback, my intention is to submit this to the steering council this time next week (~Nov 16).

24 Likes
2

If you intend on having an interactive mode like running python3 in Terminal on macOS (I have no idea whether this makes any sense), I expect that you should test that much on a device.

3

I’ve just added a note about this in the rationale section.

A traditional “python.exe” command line experience isn’t really viable on mobile devices - because mobile devices don’t have a command line. iOS apps don’t have a stdout, stderr or stdin; and while you can redirect stdout and stderr to the system log, there’s no source for stdin that exists that doesn’t also involve building a very specific user-facing app that would be closer to an IDLE-style IDE experience.

To that end, I’m suggesting that “embedded only” distribution is the goal for iOS.

2 Likes
4

PEP 730 is now live at PEP 730 – Adding iOS as a supported platform | peps.python.org

1 Like
5

Reading the section about sys the suggested values reminded that other projects, like rust, have values for it as well,

$ rustc --print target-list|grep ios
aarch64-apple-ios
aarch64-apple-ios-macabi
aarch64-apple-ios-sim
armv7-apple-ios
armv7s-apple-ios
i386-apple-ios
x86_64-apple-ios
x86_64-apple-ios-macabi

While we don’t necessarily need to follow their convention, and i have nothing against the suggested names, having some consistency (or an easy way to convert between the two conventions) could be nice as rust is becoming more and more important for Python.

Note that (afaik) in the kivy project, we still need to work on the ability to build rust modules both for android and ios.

1 Like
6

In terms of the platform triples (as they appear in the PEP 11 table), those look mostly the same as what I’ve proposed here, except for the -simulator vs -sim suffix. I picked simulator mostly because of parity with the xcrun SDK identifier (iphonesimulator), and because of my personal preference to avoid abbreviations, but I don’t think it would impact much if the suffix was -sim instead to maintain parity with Rust.

As for what sys.implementation._multiarch should return - it’s unclear to me exactly what format that should be in. It’s clearly a platform triple on a lot of platforms, but it’s darwin on macOS, and WASI/Emscripten use a tuple similar to what I’ve proposed in the PEP. I’m not sure this matters too much, though - _multiarch is a bit of an internal detail; it makes sense to be a compiler triple on platforms that can actually use it as a compiler triple, but Apple platforms don’t use GNU-style compiler identification, so it doesn’t really help. Having values that can be easily decomposed into values you can pass to xcrun seems advantageous to me; plus, there’s always the HOST_GNU_TYPE value in sysconfigdata if you do want a compiler triple.

It’s also interesting to note that the Rust naming system is itself inconsistent - they have an x86_64-apple-ios target… which must be a simulator, because there has never been an x86_64 iOS device.

On the subject of building Rust modules - I’ve got some patches (not yet submitted as PRs - mostly waiting on some of the details in this PEP to be confirmed) to crossenv and setuptools-rust that are sufficient to get Cryptography compiled, and mostly working as dynamically loaded modules. They are currently failing when the dylib is moved to the framework location discussed in the PEP - that seems to be because either CFFI or PyO3 need to be patched to use a different location for a dlopen() call when loading the Rust library.

1 Like
7

My primary concern is relying on Apple’s continued support of the “–undefined dynamic_lookup”. It seems to me that the iOS security approach would be to ultimately remove support for this.

Is the libPython statically linked into the main application?

If this is required in the future, would extension modules be able to link against a Python as a framework?

2 Likes
8

I agree the need for a deprecated compiler setting is less than ideal. All I can go on is the reference in the Apple discussion that is linked in the PEP which suggests that Apple is at least somewhat sympathetic to the needs of Python as a language ecosystem.

The current reference implementation uses a statically linked libPython in applications; however, I can’t see any reason that this couldn’t be modified to a dynamic library if the need arises.

2 Likes
9

The focus of this work would be to provide an embedded distribution that IDE-style native interfaces could utilize, not a user-facing “app” interface to iOS on Python.

One thing that is a bit unclear to me after reading the PEP is how to make extra packages available for use in this distribution - especially packages that contain extension modules (for instance regex or numpy). Those packages will also need to be recompiled for iOS and included in the app that someone is building. So, clear compilation docs are prety important (as pointed out in the PEP). Tools like cibuildwheel and pip will, I assume, also need to be updated.

Related to this (with some security implication, perhaps): Will it be possible to build apps that can dynamically update some of their dependencies by direct downloads from the internet?
This could kind of bypass the AppStore review process; it seems to be a pretty different update path compared to re-installing a new version of the whole app.
(There are currently terminal apps available on the AppStore that run a Linux or Linux-like OS in a sandbox and that support downloading python from the internet, and even support installing gcc and rebuilding Python packages – all on iOS. Since this is all sandboxed, I assume this is no big deal, but I wonder if this PEP somehow changes that picture? Will this PEP make it easier to escape from the sandbox and get root access?)

For the platform specification: “iphoneos-arm64” This may be a stupid question, but is it sufficient to only specify the ABI as “arm64”? What about “arm64e”?

1 Like
10

Before upgrading iOS to Tier-3, would it be possible to add a CI right now, and so how it goes? I think that a buildbot builder would be more appropriate. Recently, a FreeBSD CI was added, but it failed randomly, and people got annoyed since it’s “only” a Tier-3 platform. Then we ran out of free CI budget and the CI simply got disabled :frowning:

1 Like
11

Just a quick comment: I don’t think you need the underscore in sys.implementation._simulator.
By including it you’re opting out of Python’s backwards compatibility policy.
(_multiarch does have an underscore, but let’s not use it as a precedent.)

12

What has been specified in this PEP should be sufficient to build extension modules (and I’ve done so - they’re currently published in a private PyPI-compatible repo). The combination of sys and sysconfig specifications is enough to give pypa/build enough detail to construct a wheel, and sysconfigdata captures required compiler arguments.

Building for iOS requires cross compilation (as you aren’t running a C compiler on the host device). I have a prototype patch for crossenv that fills in the missing pieces; that patch is dependent on some of the design decisions in this PEP, so I haven’t formalized that patch as a PR yet. Assuming this PEP gets approval, my intention is to do so. Any PEP518 build system will also require patching; setuptools seems to mostly work out of the box, depending on the quirks in setup.py; systems like NumPy’s Meson need some minor patches (again, I’ve got patches for NumPy, and some other packages). cibuildwheel will presumably also need patching, but I haven’t looked into that yet.

pip will work without modification, provided you stick to the same minimum iOS version (i.e., you can install a minimum iOS 12 wheel on a CPython that has the same iOS minimum configured). pip will need to be patched to allow macOS-style installation of any compatible version.

There will also be a need to update PyPI to allow the new wheel tags; however, my understanding is that is mostly an administration formality.

My understanding is that running pip on-device is effectively prohibited. As you’ve noted, it’s a violation of the App Store review process, because it’s indistinguishable from avoiding code review. All pip installation would need to occur as part of the app build process. It’s technically possible to use pip in an app is only ever run on the developer’s own device, but that’s a limited audience.

As for how terminal apps et al get away with this? I don’t know. I suspect if Apple ever looked closely enough at an app doing this, the app might be retrospectively banned; however, I have no specialist knowledge here.

arm64e is a seperate architecture that at present, isn’t required to support any devices. If we were to add arm64e support, it would be another platform triple in the PEP 11 list, an independent wheel tag, etc - i.e., we’d start seeing ios_12_0_iphoneos_arm64e wheels.

1 Like
13

I’m definitely sympathetic to the desire to avoid “dead” CI platforms.

In this case, we can’t just add CI - there’s a bunch of patches that need to land as well. The patch that I currently have does include iOS triples in the Tier 3 platform check in configure, but that’s not requried to run or test. We could definitely land the “functional” patches and CI config, and make the decision about formalizing Tier 3 status (updating PEP 11 and the configure check) once the buildbot has proven reliable over time.

14

Before upgrading iOS to Tier-3, would it be possible to add a CI right now, and so how it goes?

Also keep in mind that, AFAIK and at least initially, the only practical GitHub-based CI that we could use for iOS is building on macOS (no matter what) and running the Apple-provided iOS simulator on macOS (x86_64 based). So there would be no special hardware requirements for that, unlike as was the case with the FreeBSD CI.

1 Like
15

My employer (Anaconda) has offered to cover the cost of standalone runners to support buildbots, so we don’t need to rely exclusively on Github Action runners - or, at least, that’s an option, and would allow us to test on ARM hardware without needing to use Github’s paid tier runners.

16

Yes, buildbots are critical, too. But, AFAIK, we don’t depend on any buildbots in our GH CI pipeline and we probably don’t want to add any. Sorry, I wasn’t clear about that.

17

sys.implementation._multiarch will describe the ABI and CPU architecture
sys.implementation will also have an additional attribute - _simulator - storing a Boolean that is True if the device running the app is a simulator.

IMO it’s not a good home for such information, just put it at top-level sys module. I dislike “simulator” name, it’s too generic and the name is not helpful. Honestly, just make it explicit: add sys.iphone_info or something like sys.get_iphone_info(), and put whatever you want in such object: multiarch, simulator, version, etc.

Don’t worry too much about that. We have many buildbots for “no tier” platforms such as AIX and Solaris. Sometimes, we run the garbage collector and buildbots which are gone for a long time.

I suggest you to go ahead and start proposing changes, starting from the least controversial. You can put me as a reviewer.

18

In a conversation at the Core team sprint, @thomas noted that the proposed name munging scheme for Frameworks was ambigous, as a library named _bar in a package named foo, and a library named bar in a package named foo_ would generate the same Framework name (foo__bar.framework).

He suggested that we just use the dotted-path name (yielding foo._bar.framework and foo_.bar.framework in this example). I’ve tested this, and it appears to pass automated Xcode validation. On that basis, I’m proposing that the PEP be updated to replace the underscore-based name munging scheme with direct dotted-path naming.

2 Likes
19

Over the last 15 years, mobile platforms have become increasingly important parts of the computing landscape. iOS is one of two operating systems that control the vast majority of these devices. However, there is no official support for iOS in CPython.

is important for the future of Python as a language that it is able to be used on any hardware or OS that has widespread adoption. If Python cannot be used a on a platform that has widespread use, adoption of the language will be impacted as potential users will adopt other languages that do provide support for these platforms.

This is one of the main topics that I tend to re-iterate on multiple times during conference talks and generally while discussing Kivy and Python-based app lifecycle on mobile platforms.

Python on mobile platforms is definitely a thing, and can’t be ignored.

Adding iOS to Tier 3 support will certainly help to make python package maintainers aware of mobile platforms (and making package maintainers aware of mobile platforms is a key point in order to consider iOS “Really Supported”, cause as the PEP 730 describes, on iOS, some features are not available, and libraries need to better accommodate the dynamic-loading limitations of iOS. [pycryptodome here is a great example])

This imposes some constraints on the operation of CPython. It is not possible store binary modules in the lib-dynload and/or site-packages folders; they must be stored in the app’s Frameworks folder, with each module wrapped in a Framework. This also means that the common assumption that a Python module can construct the location of a binary module by using the file attribute of the Python module no longer holds.

I definitely agree with having an Umbrella Framework for each dynamically loadable library.

I thought of this solution multiple times but looked (unfortunately) “a way too big change” to fit into my

OSS (Kivy) schedule, as I have a limited amount of time budget. So it just has been kept on my wishlist.

We currently use another approach (static libraries, linked to the main executable [https://github.com/kivy/kivy-ios/blob/master/kivy_ios/recipes/python3/dynload_shlib.patch]), but sometimes this approach it’s not applicable (again, here pycryptodome is a great example) unless a lot of patching is done on the package side. (and it’s super-challenging to maintain on long term).

There may be some backwards compatibility implications on the projects that have historically provided CPython support (i.e., BeeWare and Kivy) if the final form of any CPython patches don’t align with the patches they have historically used.

I think my first PRs on Kivy project have been done on cross-compiling things, mostly on kivy-ios repository, so, trust me, I can be quite emotive about the work that has been done.

But, even if that means keeping that code only in my memories, (and git commits), I will be happy to remove (and I think the whole team will be happy too) most of the wizardry and alchemy that keeps kivy-ios up and running (but flaky), in order to have a battle-tested and future-proof solution, raised and grown by the whole Python community.

And, I will be happy to support the whole process, with code, tests and migration guides for our users.

agree the need for a deprecated compiler setting is less than ideal. All I can go on is the reference in the Apple discussion that is linked in the PEP which suggests that Apple is at least somewhat sympathetic to the needs of Python as a language ecosystem.

That can be a huge issue, but yeah, discussing about it with Apple, may help, since lately (and that’s awesome) they are pretty welcoming with the open-source community. (And they are also actively contributing to multiple projects)

But, having a working parachute, is always a nice to have.

P.S: I had the opportunity to talk with someone (I don’t recall the nickname, but I can check on my emails) from the Python Core Dev team ~6 months ago about “Making cross-compilation easier for Python packages”, and we talked about the issues we had on Android and iOS, while cross-compiling Python packages, as he was interested in writing a PEP about it.
If adding iOS to Tier 3 support is the first step, the second one could be define a “suggested” way to cross-compile packages.

1 Like
20

Was that PEP 720?

1 Like