Can you catch warnings around your uses of optparse? Like we do in the test suite?
Probably (although our usage is complex, so I don’t think it would be easy). But why should we have to? Warnings are supposed to be to let people know of things they need to take action on - maybe not now, but at some point. That’s why people test with -Werror, to make sure they don’t get caught out by changes they were warned about. In this case, there is no action required from users of optparse. Rather, the action is for people who aren’t using optparse, but are considering doing so, to think about the implications.
Warning users of optparse is targetting the wrong group of people.
5 Likes
Which means we want it in the docs, but not in code (which has been suggested and, based on the discussion since then, rejected).
Perhaps we also want a neat list in the docs so that linters are able to provide warnings if they choose? Some of these “do-not-use” will be because of the security implications of relying on the functionality (e.g. urllib.parse is fairly exploitable, but only if you use it wrong, and we’re not going to rip it out because it’s still useful in every other case). A tool like Bandit may well want to warn on these, and we can make it easier for linters.
It might also be more useful to define a module attribute __deprecated__ = ["list", "of", "deprecated", "members", "or", "*", "for", "everything"] so that interactive tools can have access to the list. e.g. this would let most code completion tools discourage using anything marked in this way.
A warning that only triggers when you use these methods seems the least useful approach.
1 Like
Under model I had suggested, the optparse module would raise a DeprecationWarning.
Wouldn’t it be possible to filter the warning for the optparse module prior to the import of the module in pip ?
If not, then perhaps the optparse deprecation should be reviewed and possibly reverted.
pip is a central part of the Python ecosystem and if this cannot easily be changed to use argparse instead, then this is a good sign that argparse is not “a better alternative to optparse”.
Part of being better means there’s a clear and easy to follow upgrade path. This could mean that argparse would need to gain better upgrade path support for optparse users, before optparse can be deprecated.
1 Like
I don’t think this follows. If something is deprecated, even if it will not be removed, current users of it potentially will want to take action, depending on the reasons for the deprecation, better options available that will be better supported in future, etc. Or perhaps they will want to just keep using the deprecated thing. I think current users of a deprecated API are the right audience to be alerted to the deprecation, and they should also have an easy way to silence that warning once they’ve considered it. I think that the existing filterwarnings support provides this?
Whether optparse in particular should really be deprecated at all is a separate question. If there aren’t solid reasons why people shouldn’t use it, and clearly superior options to migrate to, then it likely shouldn’t be. Lots of the stdlib is not actively updated with new features, but that doesn’t make it deprecated.
My proposal above would allow doing that using the existing deprecated-removed directive (by adding the ability to specify the removal version as notplanned, and other defined values if desired, as well as extra useful optional metadata like a deprecation reason, one of which could be, e.g., security).
I’m slow-burn working on tooling that will parse this statically out of the docs, but as a more immediate option it should be pretty straightforward to have the existing directive save each entry to a central dict and output that during the docs build as JSON and CSV. This takes maximal advantage of what we have already and minimizes unnecessary churn and manual effort in making docs deprecations and key information about them (API affected, deprecation version, removal version/plans, plus reason and any other optional metadata we add) easily ingestable by tools, and with a bit more work could even be output into a human-readable table on a page of the docs themselves, as well as in the current sections of the What’s New (Deprecated, Pending Removal, Removed).
1 Like
I honestly haven’t looked. Yes it would be possible (obviously) but I don’t know how easy it would be. We can probably suppress it in the main module. Let’s be honest, it’s not going to be the end of the world even if it is fiddly.
Personally, I think its current state (“you’re advised not to use it in new code, but there’s nothing you need to do if you’re already using it beyond being aware that it won’t be updated”) is fine. But if you want to describe that as a deprecation, and then require all deprecations issue warnings, that’s a change in behaviour, which is (a) not the same as the current state, and (b) IMO, in contradiction to the statement that it will simply sit there unchanged.
Certainly pip cannot use argparse. I can’t recall precisely what bit of our syntax argparse doesn’t support, but I know there’s something. I’m not sure I’d argue that pip’s syntax is necessarily good, but changing it would be a backward compatibility problem. The thing is, pip’s use of optparse is precisely the scenario that justifies the current status of optparse - the CLI was developed when optparse was the state of the art, it uses syntax that argparse decided (for whatever reason) not to support, so pip sticks with what it has, accepting the limitations of using a no-longer-supported library.
We’re likely to switch to click, if and when we move off optparse. I don’t think we’d care much if argparse got an upgrade path from optparse.
Disclaimer: These are my views, not any sort of “official pip stance”…
I think that’s a very nice idea – if we were starting from scratch. Unfortunately, we’ve been teaching python users for years that deprecation equals removal. I’m not even just talking about CPython here, “deprecation” is used as a marker of impending removal in many open source projects (numpy, scipy, pandas, scikit-learn, etc.), so the effect is compounded.
I don’t think it’s realistic to change users’ expectation around that word anymore, or at least not within a small amount of years. Whether this conceptual clarity is worth a multi-year period of confusion, I don’t know.
3 Likes
Though as mentioned above, there are other words that mean “not recommended, but no removal planned” (and which aren’t encumbered by expectations arising from existing appearances in Python code). For comparison, here’s what the dictionary you referenced has to say about obsolete:
- no longer in use or no longer useful; an obsolete word
- of a kind or style no longer current: old-fashioned; an obsolete technology, […] methods that are now obsolete
The second one in particular is spot-on IMO.
1 Like
It seems like issuing a warning cause a major issue: warnings usually call for action. Here, a soft deprecation does not call for any action. The urgency is to no nothing! It’s perfectly fine to keep a soft deprecated function forever.
It was asked how linters can provide feedback on users who do care about soft deprecations. Since a warning doesn’t fit the use case, I propose adding a new API to query Python stdlib:
PR gh-106240: Add stdlib_deprecations module. I implemented an API to list deprecated modules, functions and C API, and an API to query the stdlib to check if a symbol is deprecated or not.
With such API, no warning is issued, running pip with -Werror is unaffected. But people who cares about deprecations will have more accurate and more up to date information: linters don’t have to maintain their own list of deprecated APIs anymore. The information is now provided by Python itself.
Example of the current API:
>>> stdlib_deprecations.is_deprecated('asyncore.loop')
True
>>> stdlib_deprecations.is_deprecated('asyncio')
False
>>> stdlib_deprecations.is_capi_deprecated('Py_VerboseFlag')
True
>>> stdlib_deprecations.is_capi_deprecated('Py_Initialize')
False
Extract of the implementaion:
# Python 3.6
_deprecate('asyncore', '3.6', remove='3.12'),
_deprecate('asynchat', '3.6', remove='3.12'),
_deprecate('smtpd', '3.6', remove='3.12'),
_deprecate('ssl.RAND_pseudo_bytes', '3.6', remove='3.12',
replace='os.urandom()'),
(...)
# C API: Python 3.12
_deprecate_capi('PyDictObject.ma_version_tag', '3.12', remove='3.14'),
I don’t understand this PR. It’s adding a new way to find out what’s deprecated in the current sense (of “to be removed”). I didn’t think anyone had a problem with how the current deprecations work. On the other hand, it doesn’t cover optparse (or any of the “soft deprecations” that are the cause of the debate) at all. So I don’t see how this will improve things from the current situation at all…?
I definitely agree that it would be great to have a centralized place to store this information in a machine-readable form.
However, could you explain a bit more the advantages of adding a new stdlib module with its own list of deprecations that linters would need to dynamically query the Python runtime for, over the approach proposed above that would leverage a modest extension to the existing deprecated-removed directives in the docs to provide the same information (and more, e.g. prose description) as a static JSON (alongside the existing human-readable form, and possibly new automatically generated deprecations/removals page(s) and/or What’s New sections).
Vs. a new stdlib module, ISTM that extending the existing deprecated-removed directive:
- Seems a good deal simpler for us to implement (only requires ≈a few dozen lines of docs-only code and uses existing metadata)
- A lot easier for linters to use, just reading a static JSON file (on the web or vendored offline) vs loading and querying dynamic code in the desired version of the Python runtime (especially for linters like Ruff that are not themselves written in Python)
- Avoids the need to duplicate all our existing deprecations and their metadata in a separate place and format, and keeping that in sync
- Provides a superset of the information (e.g. the prose description of the deprecation, whether the deprecation is “soft” or removal is planned, optional args like
reason, etc, and keeps that in sync with what is displayed in the human-readible docs) - Allows tools to use the latest info immediately without actually needing them to be compatible with the latest in-development Python versions, nor users to have those versions installed
- Also can be feasibly backported to the docs of existing bugfix branches (since it only requires internal-only tweaks to the existing docs config)
- Simpler for core devs and contributors to read, update and extend, since its just a modest extension of the existing well-used docs directive vs. a whole new code module
- Opens up a much wider array of use cases, e.g. we can use it to generate a centralized page listing deprecations or specific lists of deprecations/removals for the What’s New, and other websites, APIs and tools beyond just Python linters can easily consume the online JSON file (or CSV, etc) as input for their own tailored content
That being said, it could well be that I’m missing some reasons/use cases for considering a new stdlib module instead?
2 Likes
The idea is to provide an API to query if a module or a function is deprecated or soft deprecated: deprecations issue a warning, whereas soft deprecations don’t (new proposal). So linters can detect usage of soft deprecated APIs: linter users can decide how to handle this category of warnings.
How do you provide these information? With which kind of API? A new stdlib module or function?
Someone proposed to document deprecated functions in each module: the problem is that importing a module has side effects. Some modules cannot be imported on some platforms. Or just not available on some Python versions. A central dedicated module with no side effects sound safer to me.
Do we know if linters want an API versus a data file they can ingest into their own format?
4 Likes
See the proposal linked in the above quote for the full details, but TL;DR is its just a JSON data file automatically compiled during docs build from the metadata and content of our existing deprecated-removed directive, with some modest extensions that would be mostly needed anyway to handle the concept of a docs-level “soft deprecation” and further improve/clarify the messaging per feedback on this thread (allowing a notplanned value for the “removal” arg to signal a soft deprecation, and a new optional :replacement: and :reason: param).
A JSON file with the same information, included in the online docs for each version and easily regeneratable offline from the Python docs source, seems to be even simpler, safer and more easily usable, no? No need to have the latest bleeding-edge development Python runtime available, run dynamic code and query the module for resutls (especially for linters not actually written in Python to begin with, like Ruff). Just grab the latest JSON from the URL or drop a static copy into your artifact and boom, you can get all the exact same info in a language or website in a line or two of code. Plus, less work for us to create and maintain, too.
1 Like
I agree with @CAM-Gerlach. With the static file approach, a linter could ship with multiple versions of the file, and tell you: “hey, I know you’re using 3.8, but this API was soft deprecated in 3.10”.
That said, I think the whole idea of soft deprecations is of marginal utility.
1 Like
In my proposed stdlib_deprecations module implementation, I plan to keep APIs which are already removed (full deprecation history since Python 3.0), so a linter can detect usage of API which was already removed.
One issue with extracting deprecations from the doc is that the message is formatted with Sphinx format which may not be convenient for linters. Maybe a basic formatter can render the message as plain text without any Sphinx formatting.
The advantage of a JSON file is that you can copy the Python 3.13 file without having to install Python 3.13.
I don’t know which option would fit better linters usage.
Speaking as pylint maintainer, but certainly not on behalf of all Python linters:
I think we would prefer a JSON file.
We currently implement this with the following code:
We would probably just parse the JSON file into this format whenever some user notifies us that it is outdated or we ourselves note this. (Or change our code to parse the JSON, but that is probably more effort. Perhaps a good first issue? 
)
The issue with the API is that we’re just too reliant on the Python interpreter the user is running. This will probably cause a lot of confusion for (newer) users and also makes it so that pylint behaves differently on different interpreters. That last point is unavoidable to some degree, but if we can avoid it (by just hardcoding data instead of relying on the interpreter) we tend to go that route.
3 Likes
While I’m not sure what the best way to provide a list of deprecated APIs, it seems like the idea of issuing a warning for soft deprecated API is not liked so I abandon this idea and I closed my PR adding SoftDeprecationWarning: gh-106137: Add SoftDeprecationWarning category by vstinner · Pull Request #106142 · python/cpython · GitHub.