PEP 730: Adding iOS as a supported platform

[steve.dower]

@freakboy3742

platform.is_simulator() perhaps?

Would is_emulated() work? That could also serve to answer True when running x86 or x64 on Windows ARM64, and possibly could be made to detect qemu?

[freakboy3742 (Russell Keith-Magee)]

@steve.dower

Would is_emulated() work? That could also serve to answer True when running x86 or x64 on Windows ARM64, and possibly could be made to detect qemu?

I guess that’s an option. There’s a bikeshed-coloured can of worms in the difference between a “simulator” and an “emulator”; but I can see how that naming discrepancy matters less than the bigger question of “am I on a real device or not?”.

I guess another alternative would be to avoid the “simulator/emulator” language entirely, and use something like “is_virtual_device()” or “is_real_device()”.

[encukou (Petr Viktorin)]

Avoid premature generalization. We cannot detect all of VirtualBox, QEMU, VMWare, Rosetta (?) and what not, and keep that up to date. That kind of guesswork doesn’t belong in the stdlib. What’s a “real device”, anyway?
The iOS simulator is well-defined. Keep it iOS-specific.

[steve.dower]

Let’s also avoid a sprawling API then, and avoid simply adding new public/supported APIs without a necessary use case.

If one of the existing APIs can return something that clearly indicates the current device is the simulator rather than an actual iOS device, that’s preferable.

[freakboy3742 (Russell Keith-Magee)]

As mentioned above It would be possible to avoid adding this extra API, using platform.machine() == "iPhoneSimulator". That’s a string comparison, rather than a boolean, so it is potentially error prone; but given this isn’t a feature most people will need in practice, maybe that doesn’t matter.

Another option - platform.mac_ver() exists to provide macOS-specific version details; the reference patch currently has a platform.ios_ver() for the same reason. We could make the return value of that structure a named tuple, and include an “is_simulator” boolean. That keeps the feature behind an iOS-specific interface that was going to need to exist anyway, but leaves the door open to a more generic simulator/emulator detection at a later date.

Does that sound preferable as an approach?

[mhsmith (Malcolm Smith)]

@freakboy3742

While Rust is definitely a common target, I’d argue that every binary iOS package needs to interact with Apple’s tooling; only a subset of those need to interact with Rust.

OK, now that I’ve had a look at the existing build scripts in Python-Apple-support, I see that there is some use of GNU-style triples there. I’ve done some experimentation with this file:

int main() { return 0; }

And this command:

xcrun --sdk $sdk clang -target $target -c test.c -o $sdk-$target.o

And the resulting .o files are:

302c0d9e5a6c5ab41045a03250b13303ba39ae23  iphoneos-aarch64-apple-ios.o
302c0d9e5a6c5ab41045a03250b13303ba39ae23  iphonesimulator-aarch64-apple-ios-sim.o
c5c5daa1dcafb93eeaaf2a22055cd659c328129c  iphonesimulator-aarch64-apple-ios-simulator.o
302c0d9e5a6c5ab41045a03250b13303ba39ae23  iphonesimulator-aarch64-apple-ios.o

In other words, the suffix “simulator” does make a difference, but “sim” doesn’t.

I also found that it makes no difference whether you use “arm64” or “aarch64”. But if you omit the “apple-ios” part completely, then clang generates ELF .o files which are rejected by Apple’s linker.

So based on all that, I agree that the triples currently in the PEP are the best choices.

@freakboy3742

The documentation for platform.machine() describes the value as “The machine type… empty string if it can’t be determined”. This falls to CPU architecture because that’s about all you can really tell about most laptops and servers; however, I’d argue “iPhone13,2” or “iPhoneSimulator” is a much more accurate description of “the machine type” than “arm64”, especially given that platform.architecture() and platform.processor() exist.

I think the most important factor here is not the literal meaning of the words, but the way they’re already documented and implemented on the major platforms:

• architecture is clearly documented to not return the architecture at all, but the bitness and the executable format.
• processor returns “arm” on my macOS machine, “Intel64 Family 6 Model 42 Stepping 7, GenuineIntel” on my Windows machine, and an empty string on my Linux machine.
• machine returns “arm64” on macOS, “AMD64” on Windows (the only example given in the documentation), and “x86_64” on Linux.

So I think it’s safe to say that most existing code that uses machine is expecting it to return an architecture name, and that most existing code that needs an architecture name will be getting it from machine, and expecting one of a few well-known options. I recommend we stay compatible with that, and make the model name an attribute of platform.ios_ver instead.

@freakboy3742

platform.node() and os.uname().nodename return the same value: the machine’s network name for the simulator, and the device name (“Janes-iPhone”) on device. The platform.node() value is implemented directly by the call to uname().

In that case, the special case for platform.node should be removed from the PEP, because it’s already covered by “All other values will be as returned by os.uname().”

[freakboy3742 (Russell Keith-Magee)]

@mhsmith

So I think it’s safe to say that most existing code that uses machine is expecting it to return an architecture name, and that most existing code that needs an architecture name will be getting it from machine, and expecting one of a few well-known options.

Again, I’d argue that this is at least in part because there’s no system API that will return “Dell XPS 10…” if you’re running on a Dell Laptop, making the CPU architecture a reasonable fallback descriptor.

However, on the basis that platform.platform() seems to exist largely to work around the fact that os.uname() isn’t available on non-POSIX platforms, there’s a good argument to be made that platform.machine() should be the same as os.uname().machine, and the “iPhone13,2” device identifier should be part of ios_ver().

@mhsmith

In that case, the special case for platform.node should be removed from the PEP, because it’s already covered by “All other values will be as returned by os.uname().”

Agreed. This means the only deviations from raw iOS os.uname() values would be:

1. The platform identifier (iOS vs darwin)
2. The OS Version number (returning the iOS version, rather than the Darwin kernel version).

[freakboy3742 (Russell Keith-Magee)]

Thanks to everyone for their contributions and suggestions. I’ve incorporated the discussions so far into a revised version of the PEP. An RTD preview is available; the pull request is here.

I hope I’ve accurately represented the views expressed so far. If not, let me know and I’ll make whatever corrections are required. Assuming there’s isn’t any significant further feedback, my intention is to submit this to the Steering Council for ratification towards the end of next week (i.e., around Nov 3). If there is significant feedback, then I’ll do another revision pass for feedback.

[ronaldoussoren]

To start: thanks for working on this, support for mobile platforms is a clear gap in our support matrix and doing this upstream avoids doing ongoing support multiple times.

My background for this PEP is primarily macOS, I have very limited development experience with mobile Apple systems. I have done Python-based development on ancient windows based PDAs, but that’s hardly relevant for this PEP.

I’ve only skimmed through the discussion, but my comments are based on the PEP text and might rehash earlier discussion. Sorry about that.

mobile platforms

The PEP mentions iOS support without mentioning the other closely related Apple platforms, and in particular iPadOS which AFAIK is basically just a different marketing label for iOS as far as Python support is concerned.

Having a specification that’s compatible with other platforms, such as tvOS and the upcoming visionOS would be nice. Without necessarily requiring that those platforms are immediatly supported.

libpython symbol lookup

The PEP mentions “-undefined dynamic_lookup”. Issue #103306 contains some discussion about this, including an option to using “-U ” instead of doing dynamic lookups for all symbols. That would primarily help with getting better build time errors when using missing symbols.

The reason we use “-undefined dynamic_lookup” on macOS is that extension modules must not link to libpython to support staticly linked binaries and because the framework build is older than “@rpath” support in macOS.

Have you considered specifying that libpython itself will be shared library (framework) on iOS and link extension modules to libpython with “@rpath”?

dynamic module loading

I like your solution of using a metapath finder for searching for extension modules. T.b.h. I’d have leveraged symlinks to present a “normal” filesystem layout to importlib, but your proposal is a lot cleaner.

packaging

I’m a bit conflicted about not using “universal” wheels. I’m a bit disappointed about the uptake of them by, especially, the scienentic python community for the macOS port because this complicates life for those of us that distribute bundled python applications to others (pyinstaller, py2app, …).

That said, I agree with choosing thin wheels for the iOS port for the second reason you mention, and because using universal wheels would require stripping similator support from artefacts for the app store.

The PEP indicates that wheels will have to same layout as on other platforms, with a build step for users to move files to the right location when building distribution artefacts. Have you considered adding the binary modules to the right location in the wheel archive?

building

The PEP doesn’t mention how building for iOS will work. The reference implementation uses cross-compiling with the normal build system for CPython, and it would be good to mention this in the PEP to make the ongoing impact on the build system clearer. In particular, this makes it clear that the PEP won’t result in Xcode project file that needs to be manually maintained.

tooling and examples

Will there be documentation, tooling and maybe even a full example for creating a full app that uses Python in the CPython documentation? The “how to teach this” section is not clear about this.

[ronaldoussoren]

And to respond to myself: it might be interesting to slightly broaden the scope for platform.ios_ver (and change its name).

The information that ios_ver() returns is more useful than that of mac_ver (which is basically the same interface as on classic MacOS).

It should be possible to create a platform.darwin_version that returns the same information as the proposed platform.ios_ver but for all Apple platforms.

[freakboy3742 (Russell Keith-Magee)]

@ronaldoussoren

Having a specification that’s compatible with other platforms, such as tvOS and the upcoming visionOS would be nice. Without necessarily requiring that those platforms are immediatly supported.

Agreed. FWIW, the patches that I have already do support tvOS and watchOS, and the extra changes required for those platforms are minimal (and mostly in the area of “add tvOS to the list of known platforms” type changes). I wasn’t proposing adding those to this PEP as officially supported platforms, but the advice I got during the core team sprint was that if the patches needed for tvOS/watchOS support aren’t excessive, and they don’t break anything on supported platforms, then they should be OK to land.

iPadOS is an interesting one. While Apple seems to make a strong distinction between iOS and iPadOS from a branding perspective, I can’t see any difference from an API perspective. When you create a new app in XCode, there is no “iPadOS” option; when you have an iOS project, you can declare that you want to make iPad a valid target, but you specify the iphoneos/iphonesimulator ABIs. Apple’s own documentation says “new iOS projects contain resources for both iPhone and iPad by default”; and the Xcode projects that I’ve got bear that out. Apple’s docs always say “iOS/iPadOS” - I can’t find any iPadOS-specific docs. I’ll stand corrected, but AFAICT, what I’ve proposed here should work for iPadOS without any changes.

I haven’t done any work on visionOS, so I can’t comment on what is needed there… but if someone wants to buy me a toy

@ronaldoussoren

Have you considered specifying that libpython itself will be shared library (framework) on iOS and link extension modules to libpython with “@rpath”?

I hadn’t - my approach on iOS was mostly based on copying what macOS does, modified as needed for iOS compatibility. Avoiding the deprecation warning would definitely be desirable, so I should probably look into this…

@ronaldoussoren

T.b.h. I’d have leveraged symlinks to present a “normal” filesystem layout

FWIW, I tried this - this approach triggers the iOS “invalid format” validation.

@ronaldoussoren

The PEP indicates that wheels will have to same layout as on other platforms, with a build step for users to move files to the right location when building distribution artefacts. Have you considered adding the binary modules to the right location in the wheel archive?

I hadn’t - but I’m also not entirely sure what that would look like.

My guess is that you’re thinking of something similar to the way data files are handled in wheels, allowing a wheel to explicitly specify binary components, and use a sysconfig prefix to nominate the frameworks directory. I guess I can see how that might work - but I can also see some major complications.

Most notably, it’s radically different to what every other OS does for binaries. It means every single binary that is produced for iOS isn’t “just compiled”, it needs to be compiled, stored in a different location in the wheel, annotated with metadata, and archived with metadata in the dist-info folder that describes a different installation location.

In a world where build systems were all distutils-based, this might have been plausible - we could make one set of patches to build binaries on iOS, and the entire ecosystem would gain that feature. In a PEP518 world, we’d need to teach every individual build backend a bunch of special rules for how to handle iOS-compatible wheels, which is… a lot of work.

Using the approach I’ve described in the PEP, a good chunk of PEP518 build backends “just work”, or only require relatively minor modifications to add iOS platform awareness. We can then provide a single relatively straightforward post-processing script to use as part of the build - and some sort of build step is going to be needed regardless, because the end-user experience for Python on iOS isn’t going to be able to avoid Xcode.

A different wheel format would also require some fairly heavy convention-based standardisation. Installing an iOS package doesn’t just require knowledge about the target site-packages and some “system-specific” detail like the data directory - it also needs knowledge about the Xcode project the package is being installed in. I’m not sure how we’d communicate the “location of the Xcode project’s frameworks folder” - which isn’t a concept Xcode has - so we’d need to come up with some sort of convention for how that would be interpreted, and provide guidance on how to add those frameworks to your project.

On the other side - the proposed setup allows for runtime-installed binary wheels to work as-is. There’s a huge question mark over whether an app that does this would be allowed in the App Store - but if you don’t move binaries, then they “just work”. The post-processing becomes an App Store compliance issue, rather than something baked into the Python part of the problem.

Also, AIUI, the packaging format issues are strictly a separate issue (especially since the removal of distutils). The changes to metapath loading are independent of the packaging format; so at least at some level, we can defer exact formatting decisions until later. AFAIK, Emscripten and WASI are Tier-3 platforms, but PyPI doesn’t allow WASM binaries yet because packaging issues are still being resolved. That doesn’t stop people from using WASM today as a useful platform for Python.

@ronaldoussoren

The PEP doesn’t mention how building for iOS will work. The reference implementation uses cross-compiling with the normal build system for CPython, and it would be good to mention this in the PEP to make the ongoing impact on the build system clearer. In particular, this makes it clear that the PEP won’t result in Xcode project file that needs to be manually maintained.

Good note; I’ll update to reflect this.

An Xcode project will be needed; but only for the purposes of running the test suite.

@ronaldoussoren

Will there be documentation, tooling and maybe even a full example

My intention was mostly to follow the WASM example here. Based on that precedent, I’d infer that Tier 3 compliance requires some details about platform quirks, but mostly leans on external projects for usage instructions. I can elaborate this section of the PEP to make this more clear; or, I can commit to more comprehensive HOWTO-style documentation if required. I’ll add an “Open Issues” section about this.

[freakboy3742 (Russell Keith-Magee)]

@ronaldoussoren

It should be possible to create a platform.darwin_version that returns the same information as the proposed platform.ios_ver but for all Apple platforms.

I wouldn’t object to unified API that can be used on all Apple platforms - I’ll admit I don’t love the ios_ver name, and it’s even worse once tvOS/watchOS/visionOS are on the table. However, I’d push back on darwin_version() as a proposed name.

I’ve also never understood the intention behind “darwin” as a platform identifier in the Python context. It’s “technically correct” - yes, the kernel is Darwin. But nobody outside of kernel programmers actually know or care about this detail, and I can’t think of a single end-user API that Apple promotes that talks about “Darwin”. And, in this case, the APIs needed to satisfy these calls aren’t Darwin APIs - they’re UIKit on iOS, tvOS, watchOS and visionOS. I don’t what the APIs would be on macOS, but I presume they’re on AppKit somewhere.

Knowing there’s a Darwin kernel is occasionally useful, and I can’t argue this is what uname() returns. However, I’d strongly push back against adding user-facing APIs for iOS involving the use of the darwin name. It’s just alien (and therefor unintuitive) to the iOS user experience. In my ideal world, sys.platform wouldn’t be darwin on macOS… but I appreciate this is near impossible to change now.

There’s an analogous issue with “Android” and “linux” - but that’s an issue for a different PEP

[ronaldoussoren]

I’m not a fan of darwin_ver(), but didn’t know a better name while typing my message. I don’t know if there’s a name of the shared bits of the the various Apple operating systems.

I’m also not a fan of having ``sys.platform == “darwin”’ on macOS. I’ve proposed changing the latter in the past, but too late to make this a realistic proposal. There’s too much code that depends on the status quo.

On the other hand, if we had changed the value at that time we’d now have a value that’s also not correct (“macosx”).

[ronaldoussoren]

@freakboy3742

iPadOS is an interesting one. While Apple seems to make a strong distinction between iOS and iPadOS from a branding perspective, I can’t see any difference from an API perspective. When you create a new app in XCode, there is no “iPadOS” option; when you have an iOS project, you can declare that you want to make iPad a valid target, but you specify the iphoneos/iphonesimulator ABIs. Apple’s own documentation says “new iOS projects contain resources for both iPhone and iPad by default”; and the Xcode projects that I’ve got bear that out. Apple’s docs always say “iOS/iPadOS” - I can’t find any iPadOS-specific docs. I’ll stand corrected, but AFAICT, what I’ve proposed here should work for iPadOS without any changes.

I agree. That’s something to explicitly mention in the PEP: sys.platform == 'iOS' will be used for both iOS and iPadOS. That matches the Swift compiler (see the #if os(...) control statement in the swift reference manual

@freakboy3742

I haven’t done any work on visionOS, so I can’t comment on what is needed there… but if someone wants to buy me a toy

Same here. I’ve started the simulator once, but nothing beyond that. I expect that your proposal works as-is on visionOS, which would make adding support later fairly trivial once this PEP lands.

@freakboy3742

@ronaldoussoren

The PEP indicates that wheels will have to same layout as on other platforms, with a build step for users to move files to the right location when building distribution artefacts. Have you considered adding the binary modules to the right location in the wheel archive?

I hadn’t - but I’m also not entirely sure what that would look like.

Having thought a little more about this: keeping your current proposal is better. My proposal requires changes the layout of binary wheels and would require changing all tooling that can generate wheels.

@freakboy3742

@ronaldoussoren

The PEP doesn’t mention how building for iOS will work. The reference implementation uses cross-compiling with the normal build system for CPython, and it would be good to mention this in the PEP to make the ongoing impact on the build system clearer. In particular, this makes it clear that the PEP won’t result in Xcode project file that needs to be manually maintained.

Good note; I’ll update to reflect this.

An Xcode project will be needed; but only for the purposes of running the test suite.

A project for running tests is fine.

@freakboy3742

@ronaldoussoren

Will there be documentation, tooling and maybe even a full example

My intention was mostly to follow the WASM example here. Based on that precedent, I’d infer that Tier 3 compliance requires some details about platform quirks, but mostly leans on external projects for usage instructions. I can elaborate this section of the PEP to make this more clear; or, I can commit to more comprehensive HOWTO-style documentation if required. I’ll add an “Open Issues” section about this.

As long as there’s documentation somewhere ;-). This is a feature that I’d like to play with without having to do a lot of research myself.

[mhsmith (Malcolm Smith)]

@ronaldoussoren

The PEP mentions “-undefined dynamic_lookup”. Issue #103306 contains some discussion about this

There’s also discussion in #97524, where we received a clear response from Apple explaining that the warning is gone in XCode 14.3, and the option is safe to use. So I think that concern can be removed from the PEP.

@ronaldoussoren

Have you considered specifying that libpython itself will be shared library (framework) on iOS and link extension modules to libpython with “@rpath”?

If we were starting completely from scratch then that might be the best solution. My main concern is making sure we leave open the option for BeeWare and Kivy to share a repository of iOS wheels in the future.

As @misl6 said above, Kivy currently links libpython and all extension modules statically into the main executable. If they ever switch to loading extension modules dynamically, would they be happy to start loading libpython dynamically at the same time?

[freakboy3742 (Russell Keith-Magee)]

@mhsmith

There’s also discussion in #97524, where we received a clear response from Apple explaining that the warning is gone in XCode 14.3, and the option is safe to use.

The warning is gone for Xcode on macOS. The warning still definitely appears for non-macOS targets. What that means for being able to rely on the option is anyone’s guess - I’d agree that it’s indicative, but it’s not exactly a firm commitment.

EDIT: Since it came up in an offline conversation, the warning reported on iOS is:

ld: warning: -undefined dynamic_lookup is deprecated on iOS

@mhsmith

If we were starting completely from scratch then that might be the best solution.

I guess the question is whether this PEP is “starting from scratch” from the perspective of CPython as a project. There’s an argument to be made that BeeWare and Kivy have been in uncharted waters until now, and as such, the decision CPython makes as a project is not bound by the decisions those projects have made.

On that basis, it could be argued that official CPython builds on iOS must be framework builds, and projects like BeeWare and Kivy need to adapt. That’s obviously disruptive to those projects, but any official CPython iOS support is going to be at least marginally disruptive. I’d argue it’s better to use this as an opportunity to do things right from the outset.

That said, the reason macOS currently uses this option is exactly the same reason that iOS needs it: although most uses of libPython are framework builds, it is possible to build a static library of libPython for macOS, and binary modules have to support both configurations. Another approach to this problem might be to use -undefined dynamic_lookup for now, and if/when #103306 is resolved on macOS, or the option is removed the iOS compiler toolchain, adopt the -U approach that is described on that ticket.

@ronaldoussoren / @misl6 do either of those approaches sound preferable to you?

[misl6 (Mirko Galimberti)]

As @misl6 said above, Kivy currently links libpython and all extension modules statically into the main executable. If they ever switch to loading extension modules dynamically, would they be happy to start loading libpython dynamically at the same time?

We would be happy to do anything that benefits our users and the community. kivy/kivy-ios will undergo an epochal change, but even if Kivy’s resources are limited, I do not see any blocking to also start loading libpython dynamically.

That said, the reason macOS currently uses this option is exactly the same reason that iOS needs it: although most uses of libPython are framework builds, it is possible to build a static library of libPython for macOS, and binary modules have to support both configurations. Another approach to this problem might be to use -undefined dynamic_lookup for now, and if/when #103306 is resolved on macOS, or the option is removed the iOS compiler toolchain, adopt the -U approach that is described on that ticket.

Since we’re starting “from scratch” (as iOS is not a supported platform ATM), I do not see why we need to hurt ourselves with something that is already deprecated, risking that another project/user starts to rely on this behavior. Maybe could be an option, but not the default (IMHO).

[ronaldoussoren]

@freakboy3742

That said, the reason macOS currently uses this option is exactly the same reason that iOS needs it: although most uses of libPython are framework builds, it is possible to build a static library of libPython for macOS, and binary modules have to support both configurations. Another approach to this problem might be to use -undefined dynamic_lookup for now, and if/when #103306 is resolved on macOS, or the option is removed the iOS compiler toolchain, adopt the -U approach that is described on that ticket.

I’d prefer to avoid using a deprecated option on iOS.

The macOS port has to deal with backward compatibility and cannot drop support for static builds. I do hope to find some time to look into the -U approach there because that’s a nicer development experience. That would still end up using dynamic_lookup for the libpython symbols though.

[freakboy3742 (Russell Keith-Magee)]

I’ve just submitted another round of updates to PEP730, incorporating feedback since the last update.

• As a result of previous discussions about the deprecated --undefined dynamic_lookup flag, I’ve done some testing and determined that a dynamic framework build of libPython is a viable option. The PEP now proposes that we link binary modules against libPython, and only support dynamic framework builds.
• I’ve added an explicit note about iPadOS
• I’ve clarified the relationship between the CPython configure build system and the artefacts that will be needed for practical usage.

Once again, feedback would be most appreciated (especially from @misl6 and @ronaldoussoren, as you’re the closest to this from a technical perspective).

This resets the clock on steering council submission; barring significant feedback, I’ll submit this for ratification this time next week (~Nov 16).

[freakboy3742 (Russell Keith-Magee)]

Last call for comments - unless I hear voices to the contrary, I’ll be proposing this to the steering committee for ratification in about 24 hours from now.