Poll: Public/internal handling — what behavior do we want?

Ideas
1

Following the recent discussions around PEP 842, PEP 843, and PEP 844, and using two namespaces, I’d like to get a broader sense of what semantics people actually want from a public/internal distinction.

The goal of this poll is to understand what behavior people would find most useful, not to choose a particular syntax or implementation.

Please answer based on the behavior you would prefer, rather than on what you think is easiest to implement.

Access to internal names: How should code outside a module access its internal symbols?
  • Through the same interface as public symbols, e.g. module.internal_name (current approach)
  • Through a distinct interface, e.g. module.__internal__.internal_name
  • Internal symbols should not be externally accessible
  • No preference
  • Other
0 voters
Default visibility: Should names (e.g. declared functions) be
  • Internal-by-default, with public names explicitly marked
  • Public-by-default, with internal names explicitly marked
  • Neither: every name explicitly classified
  • Internal-by-default and public-by-default should both be possible, configurable per module
  • No preference
  • Other
0 voters
Declaration locality: Where/how should the public/internal distinction be declared?
  • Centrally at the top of the module
  • Locally next to the definitions
  • Configurable per module
  • Both central and local declarations can be mixed
  • No preference
  • Other
0 voters
Package-level visibility: Should modules in the same package get direct access to each other’s internals?
  • Yes, same-package modules can directly access each other’s internals
  • No, internals should be module-level scoped. Other same-package modules will need to go through the same opt-in mechanism for internal access just like any other external code
  • Introduce a separate package-internal visibility level, distinct from public and module-internal
  • No preference
  • Other
0 voters

Propsals for different behavior and why you chose “Other” as an answer are welcome below. Please refrain from going into discussing concrete solutions and implementation aspects. I’d like to keep this focussed on requirements engineering.

11 Likes
PEP 843: Export Statement for DRY Re-exports
Use a with block to control export (public/private/__all__)
PEP 844: `public` and `private` builtins
A two-namespace model for Python modules to distinguish public/internal
Namespace blocks
2

You haven’t provided a “The way things are right now.” option for anything, but based on the solutions so far I don’t really see them helping with any problem I have and they’re all more complex than the status quo, so I don’t particularly want any of them.

Type checkers already work well enough for the level of visibility being discussed. As a library author, I need to be able to define the public API, without affecting runtime access to it, at both the module and class level, and I want that to be a single consistent solution.

The status quo is:

  • Access to external names: directly, regardless of if they’re marked public/private somehow.
  • Default visibility: public if not _ prefixed name, or public if in __all__ if __all__ is defined, or public if re-xported import name as name.
  • Declaration locality: _ prefixed name, or __all__ somewhere, or import name as name somewhere.
  • Package-level visibility: same as first question: direct access, regardless of public/private mark
31 Likes
3

Regarding default visibility, where I voted “other”:
I think imports should be internal-by-default, as well as symbols that start with a underscore. All other symbols should be public-by-default.
I have some doubt about the leading underscores, but using the same convention as we do for __all__ there is probably the right choice.

2 Likes
4

Thanks for the feedback.

The first question has the option “Through the same interface as public symbols […] (current approach)”. The last question has “Yes, same-package modules can directly access each other’s internals”. These describe the status quo.
The other questions indeed lack a direct entry fort the current state (´_´ prefix or __all__ or reexport). Sorry for the omission. Please use the “Other” option there.

2 Likes
5

This probably would have been a good place to use ranked choice voting. There’s also several details that I consider important to which behavior I’d prefer in some cases, and it can’t be seperated.

I prefer at top of module declaration if it has to be managed. I don’t find keeping __all__ up to date hard, it’s not a common action to add something to the public API of a module, and you should know when you do it. A single extra thing that might need updating when adding something new to a module is not a large burden, and it’s simple.

I wouldn’t want to use a decorator to “manage this for me”, unless it did something more than just add it to some export list, like add a message users would get on attempting to access something private, but I also think private should be the default for modules, so that’s a tiny side-concern if private by default happens.

However, if we are getting syntax for this, then that calculus shifts a bit. Depending on the specifics of the syntax (are the names chosen appropriately extendible to classes in the future?) I would expect it to be in-line for consistent application.

I think you’re trying to condense too much of the discussion into a set of polls here, so I hope anywhere you take it after this doesn’t leave out any of the nuance that these polls do.

7 Likes
6

No preference to how (it’s the exception, after all), but definitely it should remain possible. Therefore other

1 Like
7

I think that the option “Internal-by-default” would be preferable (clearly state what belongs to the API), but it is obviously not backwards compatible, unless it is opt-in in some way. I am not sure what the difference between “Internal-by-default with opt-in” and “Internal-by-default and public-by-default should both be possible, configurable per module” would be.

I voted for the latter one in order to make clear that Internal-by-default would need to be opt-in. The opt-in should be explicit, a solution like that proposed by PEP 842, where the use of a single export keyword somewhere in the whole module turns the default from everything-public to internal-by-default seams to be too implicit for me.

3 Likes
8

I replied as best as I could, but there were a lot of places where “I don’t understand the question” or “none of the above” were what I would have preferred. Rather than just answering “other” to everything, I made a best attempt. I hope that doesn’t distort the result too much, although there are certainly cases where you could implement what I voted for and I’d hate it…

The biggest issue for me is that it’s hard to answer in the absence of information on how we’d get to the final state. For example, I voted for access via a distinct interface, but a lot depends on what that interface is, and whether it’s designed to make accessing internals feel like you’re doing something wrong (as opposed to just making a choice on whether you’re comfortable with using something unsupported).

Internal by default is unacceptable to me if it affects existing modules, which weren’t designed with that in mind. But as a long term goal with a viable transition plan, it might be reasonable. I wasn’t able to choose it even if I wanted to, without more details.

As for declaration locality, my real answer is “alongside the documentation that explains what the public API is”. If that’s a module docstring, then “at the top” makes sense. If it’s function/class docstrings, “at point of definition” is best. If it’s in a separate document, neither is ideal and you’re in effect duplicating information. Whatever is easier to ensure docs and code are kept in sync applies then. So I said “both” - but randomly mixing the two approaches in the same module is (obviously, to me) a really bad idea, so the option I voted for (“can be mixed”) implies more freedom than I think is acceptable in practice.

Package level visibility is the one where I had the worst case of “I don’t understand the question”. I want to substitute “project” for “package”, but the two concepts aren’t quite the same and I don’t know how that would affect my answers. But taking “package” as the best we can do, it still depends. I’d want something like an “internal” pkg.utils module to be directly accessible everywhere in pkg, but I wouldn’t want private support functions in pkg.cli to be usable in pkg.networking, for example. And unit tests are a whole other issue - having to jump through extra hoops to access the internal support function that you’re specifically testing seems like unnecessary complexity. I said “same-package modules can directly access each other’s internals” because I think the status quo is the best we can realistically hope for here (especially if we are considering backward compatibility). And even that doesn’t cover tests, so I’d extend it to “same-package and test modules can directly access internals”.

One key thing that drives my views here. Within a single project, code as written now - with direct reference to internal names from anywhere within the project - is working code. It’s not wrong for a project to access its own internals. It may be a nasty bit of technical debt that the project would like to fix, but it’s not the same as an external user calling an API that’s not declared as public and supported. Therefore, in my view, backward compatibility requires that we don’t break this working code without the usual deprecation process. That means that (for example) the combination of “private by default” and “internals should be module level scoped” is not allowed without a deprecation and transition cycle under Python’s current backward compatibility policy.

This is where I get stuck with the whole “I don’t understand the question” problem. Options are being presented as if they are all equal, and the four questions are presented as independent. But they aren’t, once you take things like the compatibility policy into account. So as it stands, the poll is hiding (not intentionally, I’m sure!) a lot of important nuances around feasibility and complexity.

13 Likes
9

Thanks for the detailed feedback.

As the previous discussions have shown the topic is complex with a lot of nuances and dependencies. So much that I got the impression that people partly talk past each other because of different mental models and unspoken assumptions; and also that discussions drift off into technical details of specific solutions, loosing the bigger picture of what we actually want or need.

The poll is an attempt to delineate key aspects to foster and guide further discussion. First by stating them as individual questions, and second by getting a high-level sentiment. Doing this is of course a strong simplification.

I acknowledge that the poll may feel vague and runs the danger of being too abstract, but not proposing concrete solutions yet is intentional. There can be different solutions to the same requirement. It’s not guaranteed that consensus on a solution can be reached or even that there is a technically feasible or justifiable (cost, compatibility, …) solution. - I’m deferring all the nitty gritty details to subsequent solution discussions.

The poll should rather serve as a muter or enhancer for follow-up discussions: If e.g. most people would say, they want to access internal and public symbols through the same interface, that discussion is done because that’s the status quo. If however, there is a strong interest in making access to internals different to access to public API, that’s only the starting point for figuring out details.

In that sense, yes please voter under a “best case” assumption: Vote for an entry if you see at least one appealing way of realizing it, and even if you think it’s generally a good idea worth discussing further, even if you are not sure whether it can be done or is worth it.

You are absolutely right with these observations. I’m aware of several nuances (though certainly not all). I’m not “hiding” them to gloss over issues and make people believe the world is simple. It’s not. Instead, leaving them out of the poll is intentional to be able to generate a coarse structured picture. It would not be possible to create a poll reflecting all the nuances and dependencies. The nuances will all resurface when we move back to discussing concrete solutions.

We’ll have to see whether something useful comes out of this approach.

5 Likes
10

The scope is to me the most tricky one. I think importing from below should not have to use __internal__, though I guess that can be OK since a submodule can have everything public for modules above yet not have its contents exposed, if the submodule itself is not public.

But that leaves the example mentioned by Paul, of an pkg.utils module where you’d like to use its contents everywhere throughout pkg but not necessarily expose it to users who import from pkg. But perhaps even then from pkg.utils.__internal__ import ... actually states intent more clearly (… will change my vote a third time…)

Marten

11

I think authors will generally have many different views on what “internal” means, contrary to public.

Public is easy: it’s what I expose to users, and maintain to be free of breaking-changes.

Internal on the other hand can mean that the access at runtime is enforced or not, depending on a specific scope, whether bound to the language or how humans organize: same package, same module, same class, or same organization, same team, same project.

An example: you might want to completely prevent users from accessing an internal name, while allowing the same org uncomfortable access with a static analysis and runtime warning, while allowing the same team comfortable access with only a static analysis warning, while allowing the same project complete access without any warning. Or maybe something even finer grained, with package, module and class scopes.

What I mean is that maybe we should only standardize the public vs. internal declaration here (emphasis on public), and leave out what exactly “internal” means and implies. We’d drop questions 1 and 4.

The other option is of course to gather data around how projects handle and would like to handle internal APIs, and specify that too. But IMO there’s a clear cut and this can be done later.

2 Likes
12

Is that not the same as voting “Through the same interface as public symbols, e.g. module.internal_name (current approach)” for (1) and “Yes, same-package modules can directly access each other’s internals” for (4)?

13

Sure, that’s actually what I voted for :slightly_smiling_face: If these options end up being voted by the majority, the result is the same. My point was to avoid asking the question in the first place (or to avoid continuing the discussion around what the questions ask), and focus solely on defining the mechanism to distinguish between public and internal, dropping aspects relevant to internal APIs only.

2 Likes
14

What I want:
Everything is public. Everything is accessible.

If an utility function is useful to the library author you can bet it will be useful to the user. Same thing with attributes.

It helps with learning (by exploration), it helps with tinkering.

How many times I had to fight with C++ or Java because public/private got in the way. There has never been a time where public/private was useful to me.

Please don’t import this to Python.

Having the possibility to document what is intended API and what to use at your own risk is fine.

10 Likes
15

So what distinguishing behaviour would you expect of something that’s considered “public”, if you intend to leave internal names behaving as now? As far as I can see, you’d be left with public being just an annotation on certain names, with no distinct runtime behaviour at all.

Which, to be clear, I’m perfectly fine with - I’m happy with leaving the public/private distinction as a matter of documentation (and something tools flagh for the user). But others might not be.

5 Likes
16

The only option which an outright majority took is “public/internal should be declared locally next to the definition”. I interpret this as people liking the @public decorator from atpublic (the one that appends the name of the class or function to the module’s __all__).

1 Like
17

I only voted for that because it’s closest to the status quo of using underscore prefixes. I withdraw that vote if anyone is going to interpret that as support for something __all__-based (which I really don’t want to use).

2 Likes
18

No worries! I did select Other for a few things because – as is probably no surprise – I like Python’s current semantics (for namespace access, “publicness”, etc.). just fine so I’m personally not in favor of changing anything there.

Perhaps, but as I mentioned elsewhere, features don’t get added to Python by a DPO vote. Polls like this are useful to PEP authors to get a sense of where their proposals might have gaps or YAGNI, but ultimately I write the PEP for what I want and let the SC have the final say, which often includes suggested or required modifications.

For me as an SC member, I use polls as a data point, but only to some extent. DPO in general greatly under-represents the wider Python community. Discussion threads, and external feedback (say, from 3rd party Python projects that may not engage much w/DPO) are also super helpful.

That all said, thanks for putting this poll together @timhoffm

7 Likes
19

A lot of my preference hinges on what happens to the existing methods. I’m quite sympathetic to wanting something better than import sys as _sys for single file modules.

But I also think the status quo for packages or using underscore-prefixed submoules and an __init__.py to export the public stuff trumps all these proposals, anything involving __all__, and anything I’ve used in other languages.

So anything that breaks that or even just insinuates that __all__ (or whatever else we come up with) is the preferred/official/correct/modern way to publicise an API gets an automatic -1 from me.

2 Likes
20

This is one of the issues with the poll as it stands. Declaring that a name is private by using an underscore is entirely in line with “public/internal should be declared locally next to the definition”. So is @public. And yet, people can like one and hate the other - so how do we interpret their vote?

4 Likes