PEP 842: Module Exports (new revision)

PEPs
1

I’ve created a new revision of PEP 842 based on the feedback from the first discussion thread. In short, the PEP now proposes an export (soft) keyword, while still using the __export__ list as the backend.

Get it while it’s fresh:

3 Likes
2

Other than feedback you’ve already gotten from me, with the new version, I’m not sure there’s a good reason to allow “anything that implements contains” for __export__, but then further constrain what must be implemented in specific cases.

I think requiring it be a python list is fine (meets the requirements of __export__, export syntax, and implicit __all__), and less likely to run into edge cases or confused users, and being more permissive here isn’t something that is needed.

3 Likes
3

Sounds reasonable to me.

1 Like
4

Could you add a Rejected Idea as to why export doesn’t work for methods? PEP 842 – Module Exports | peps.python.org sort of covers it, but it’s specific to the private idea.

1 Like
5

A common use case for assignment with unpacking is that some values of the unpacked sequence need to be ignored:

a, b, *_ = 'abcde...'

I believe the spec says that making this an export assignment will include ‘_’ in _exports_. I think that is unlikely to be what is typically desired. So, rather than:

export a, b, *_ = 'abcde...'  # exports a, b, _

to avoid exporting ‘_’, it could be written as:

a, b, _ = 'abcde...'
export a, b

Should export assignments special case ‘_’ (or any identifier that starts with ‘_’) to not include them in the export? Of course, it is possible the intent is to export ‘_’ (or something prefixed with it), and it would have to be manually added to _exports_.

I don’t have a strong feeling on it either way, but do think the it should be addressed in the spec as ‘_’ is commonly used to ignore uninteresting values when unpacking and I think there is a chance of surprise if ‘_’ appears in _exports_ when doing this.

2 Likes
6

I have a few questions.

  1. Are “modifiers” order sensitive?
export async def foo(): ...  # valid
async export def bar(): ...  # invalid
  1. What happens when a previously exported name is reused?
export class Foo: ...
class Foo: ...  # which one is exported?
  1. Are * exports a thing? from .module export *
1 Like
7

Good questions. I’ll answer them here and clarify in the PEP.

I see two options:

  1. Do nothing – if a user explicitly put _ in an export statement, then that’s their fault.
  2. Disallow unpacking in export statements.

I don’t think it’s a good idea to make _ special. I’ve actually written a library before that made use of _ as a public name:

from pointers import _

ptr = _&"hello world"

I think we should continue to support that in the new API.

Yes, they are. The grammar specified in the PEP only allows for export to be a prefix.

This is functionally the same as:

class Foo:
    ...
__export__.append("Foo")
class Foo:
    ...

So the name Foo is redefined, but "Foo" still shows up in the export list. Maybe we should consider raising an exception for this case?

Currently, yes, per the current specification. I haven’t thought about this too hard, though. It might be worth requiring that the module defines __all__.

1 Like
8

Or perhaps requiring that it defines __export__?

1 Like
9

I note that this proposal still doesn’t cover allowing access within a project, but keeping symbols private for external use.

I’d argue that this should be explicitly stated in the PEP.

4 Likes
10

It might be worth mentioning how this feature looks in other languages. PEP 695’s appendix did it more exhaustively than I think you’d need here.

The main comparison I’d see as useful would be JavaScript’s export

1 Like
11

Didn’t comment in the previous thread in time, but I think the webbrowser module is a prime example of __all__ differing from the public, documented names, which includes the browser controller classes. Maybe this can have a place in the “Reuse __all__ for exports” section under the rejected ideas?

12

I think the biggest difficulty with doing that is that most other languages default to “not exportable” for names. So export typically (like in Javascript) means “allow others to import me”. Because Python makes everything visible, export has a very different (and more complex) effect. It means something along the lines of “change the default to be that symbols warn when used, and then exempt this symbol from that warning”.

With that said, I suspect that this is something that should be covered in “How to Teach This” - how do we explain to people coming from other languages that Python’s export mechanism is different, and export doesn’t do what you’re probably expecting it to?

11 Likes
13

It’s unclear to me whether those controller classes are meant to be public/importable. I think you’re supposed to use webbrowser.get(name).

Sure, I can add a section. I don’t follow the second part, though – what would be the “expected” behavior for export in Python? Everyone I’ve spoken to offline has intuitively figured this out; Python has everything visible by default, so things only become hidden once export is used. I think that’s simple enough to learn and to follow.

14

@dangotbanned suggested comparing Python’s export to the one in Javascript. The key point would be to explain that unlike in Javascript, you don’t need to add export in Python in order to import a name from another module. It’s optional, to explicitly express intent, not mandatory to enable importing at all.

Once you have made that point, though, you then need to go on to explain that if you do add export for one name, you need to add it for all others in that module that you want to export, because the export keyword has a global effect on what is visible when importing from that module.

The second part is less critical[1] - although I expect some people will be caught out by it. The first part is important (IMO) to ensure that we don’t get a bunch of newcomers with Javascript (or Rust, or similar) experience thinking that export is necessary in Python.

  1. Although arguably the more confusing point ↩︎

2 Likes
15

export feels like the wrong approach to me.

  • There’s the logical inconsistency: Since everything is public, export as a concept does not make sense. This results in the somewhat quirky logic that the proposed export doesn’t mean “make this public” but “make everything private except for this”.
  • The global effect of a local keyword is counter-intuitive and can easily lead to errors: You have to know whether the module uses export to write correct code (if in doubt you need to search the module for “export” to be sure). If you add an exported function in a module that doesn’t use export, you implicitly hide all other symbols. If you didn’t know the module uses export and add you a function without it, your assumption that the function is public is false.

IMHO it would make more sense to invert the logic and introduce a private concept/keyword.

12 Likes
16

Regarding the effect on dir(), I’d like to note that unittest and pytest use dir() on module and class objects to discover tests. This means that if export is used in a test file, tests that are not exported will be silently ignored.

If this turns out to be undesirable, perhaps pytest could implement its own dir(), however this is not currently possible cleanly (see Expose special method lookup at Python level - #13 by bluetech ).

17

A valid concern, but off the top of my head, I can’t think of a case where you would want to import anything from a test file, meaning export would likely not be used in the first place.

1 Like
18

Honestly I would just declare methods (and whatever else lives in a class) out of scope here. That’s a completely different discussion.

3 Likes
19

It doesn’t make sense when you apply the JavaScript meaning. In Python, it’s what you described: “make everything private except for this”.

I’m not convinced this will be a big problem in practice. In any given project, it’s either going to use export or not, so you have to look for it just one time.

I’ve thought about this a lot, but I strongly feel that this is not a good solution.

The purpose of this proposal is to avoid leaking private names into the public namespace, but with private, you have to know all the names anyway. We don’t want to need to think about every little thing that defines a name – export lets you think about just the public API, and then you never have to think about privacy again when adding helpers and imports to improve that public API. Inevitably, developers will forget to add private somewhere, and that will almost certainly lead people to think something is public because the definition is not private.

I think it’s also mechanically more difficult. For large modules (which are the primary beneficiaries of this proposal), there are often many more private names than public names. Codebases where this matters will be littered with private at every corner.

I acknowledge that export can be a bit counterintuitive when you first hear about it, but I think it takes about 2 seconds to learn what it does and why. private is more intuitive, but it costs you time in the long run.

1 Like
20

I’m sorry, but this feels very much like the “export is not pythonic” discussion from the previous thread. I don’t want to re-open that debate, as I think it was adequately covered before, but I do think that you should acknowledge that what you want to do here[1] is to have a way of changing, on a per-module basis, the long-established principle that Python modules are “public by default”.

Whether people agree with that idea or not, I think it needs to be very clear - and my feeling is that you’re too close to the problem, and as a result aren’t seeing how hard it is for other people to make the association between “exporting public symbols” and “switching to private by default”.

Honestly, I think that the debate should start with the question “do we want to have a way for modules to opt into being private by default, and what should that opt in look like?” Once that question has been resolved, the question of how we mark public names will be a lot simpler.

If we take that question as a starting point, I think there are some basic considerations that come from it:

  • Having an export statement somewhere in the module feels insufficiently visible. We do something similar with generators, which are identified by having a yield statement in the function body, but functions are much smaller than modules (which can be hundreds of lines long). And in that case, the mere existence of yield (even if hidden behind if False:) changes the interpretation. It’s not clear to me whether the same would be true of export.

  • Having a special variable (whether __export__ or something more explicit like __private_module__ = True) is better, because it’s easier to make it visible. But semantically, there are a lot of edge cases to consider, and being a runtime value makes it impossible for static tools to cover all cases. Consider

    def sneaky_definition(optional = False):
    if optional:
        global __export__
        __export__ = []

sneaky_definition()

    How would a static analysis tool be able to know what the public names in this module are?

Let’s get these (and similar) matters sorted out before we worry too much about the exact spelling of “how do I make this name public?”

  1. Assuming I’ve read your intentions correctly! ↩︎

7 Likes