It’s unclear to me that pip itself doesn’t also do this approach of having codebase structure that tries to hide implementation. Otherwise why is almost all of pip’s non vendored code written inside a folder called _internal? If documented public api is sufficient then why not avoid having internal folder structure and have more direct pip/commands.py, pip/install_command.py, etc? For pip there’s also extra special thing of it not having any programmatic public api. For other libraries they commonly have some public api and want to have some thing comparable to pip’s usage of _internal to make implementation clear as not intended for usage.
CPython core itself also has some maintainance burden from this issue. There are many undocumented internal methods in standard library. If a method has some mild usage in other open source codebases it is common for there to be avoidance/debate to not just remove it without deprecation period. There is a conflict between being able to follow statement that internal undocumented apis are free to be refactored and do not need to be maintained with minimizing issues for your user base. I don’t think this is an open source vs commercial dichotomy either. Usual examples of reasons to not just remove undocumented CPython method come from looking at other pypi packages usage of it/open source grep like scan (source graph).