Python and OpenSSL FIPS mode

Ideas
1

Hi folks,

I’d like to start a discussion regarding CPython’s compatibility with OpenSSL’s FIPS mode.

FIPS-140 is a set of security standards for cryptographic modules, mainly used in US Federal agencies, contractors and vendors.

FIPS sets various restrictions in regards to the permissible digests, key sizes etc, with one of the prominent one is disabling the md5 algorithm, and it’s implemented system-wide. Some more information is provided here.

Any developer, developing or deploying applications on a FIPS enforced environment needs to be using FIPS compliant software. While I’m not advocating for fully supporting an upstream FIPS compliant codebase, I’d like to work on a best-effort initiative, as the required changes, I believe, are not that intrusive and do not require significant maintenance.

Motivation is coming from maintaing a patch downstream mainly for Red Hat Enterprise Linux to provide said compatibility. We’d like to share that patch with others.

Currently FIPS mode can be determined through _hashlib.get_fips_mode() which was added in Python 3.9, along with various fixes around that.

Now I’d like to move more of the bits of our downstream patch upstream with the main point being making CPython more compilable and usable for system utilizing OpenSSL’s FIPS mode.

On a high level, the patch:

  • exposes blake2b and blake2s from OpenSSL (although they are limited in their interface, and using them for security is not FIPS compliant),
  • disables Python’s blake2 interfaces in case we are in FIPS mode
  • disables the hash-based .pyc validation in FIPS mode, due to the non-fips-compliant sip24 digest
  • adds various test fixes.

The patch is assuming that Python has been compiled with --with-builtin-hashlib-hashes=blake2 only and it needs to be generalized for all the other cases, main one being disabling falling back to the internal implementations in case they are built if we are under FIPS.

I’m also already providing, for some time now, two buildbots compiling Python under FIPS enabled systems:

4 Likes
2

In the hopes of the issue getting more traction I’m cc’ing some folks who might be interested into that or could provide some more feedback.

@tiran @encukou @vstinner @gpshead

3

Initial thoughts on the patches themselves:

The OpenSSL ones don’t provide the complete API necessary to be hashlib compatible so we should not use them - I don’t want to see new code that doesn’t support the entire API added to CPython.

Just have them raise an error if usedforsecurity=True when in “FIPS mode”. There is no added value in exposing them from OpenSSL builds that oddly support them.

I don’t like this one. Hash-based .pyc validation is not a security feature. The fast hash is not being used for security. Disabling it just hurts the rare-ish users who actually use validated hash-based .pyc files (it isn’t the default) in their deployments. If you are disabling just this, you really should disable all use of .pyc files entirely. A timestamp + size comparison is just as much a not-for-security hash as sip24 is. It’s just that nobody put “hash” in the name of that comparison.

hashlib is moving in the direction of potentially not using OpenSSL at all in the future in favor of only supporting the new HACL* based built-ins. If we do get to that point, keeping _hashopenssl.c around won’t make a lot of sense…

4

This patch set seems to be going beyond OpenSSL by trying to prevent things not in OpenSSL from doing their job. At that point it ceases to really be “OpenSSL FIPS mode” and turns into a feature “CPython FIPS mode sponsored by $Vendors X,Y,&Z”. Which is plausible as a feature request but needs an explicit definition of scope and buildbot(s) that explicitly tests such a build and environment validating that things that should not work do not, and things that should do. With engaged contributors who’ll work to keep that stable as the codebase evolves.

It isn’t the kind of thing that the open source community is likely interested in maintaining on their own (it’s someones interpretation of one government’s bureaucracy). So a goal I’d recommend is to make understanding of the scope and maintenance of code that dances around such a self-limiting feature as easy as possible. Most people never want to think about it at all. All of the existing OpenSSL FIPS mode support within CPython today came from people specifically employed to make it happen, RedHat primarily.

Note that the HACL* hash implementations are also in the formally verified camp, just not by the US Government’s National Institute of Standards and Technology. Someone $enterprising could presumably make that happen so that they get a US NIST seal of approval similar to the one the OpenSSL fips module claims to have - assuming that is what this is really all about.

5

I’m quite biased; I don’t think I should be making decisions for CPython in this area.

But for the record, I think the current state draws a reasonable boundary between what CPython should do and what should be the job of a redistributor (who can both ensure OpenSSL is available, and enact and test system-wide restrictions).

6

I’m working for Red Hat with @stratakis and I’m fine to care of maintaining these FIPS changes upstream. We are already maintaining these changes downstream for years. The idea is to make it available to other users who have to respect FIPS constraints.

If tomorrow, hashlib gets rid of OpenSSL, that would be great, and we can adapt the FIPS patch then.

1 Like
7

I agree that exposing them from OpenSSL is undesirable as they lack features. Raising an error sounds reasonable here. On a semi-relevant note here, compiling Python without the internal blake2 implementation results in various test failures.

I’m in the process of the removing this part of the patch as it was an overkill and siphash is not used in a security context here.

Reaching that point is a long-term goal and replacing the internal digests implementation is a different thing than stopping supporting openssl or other cryptographic libraries. And if that point is reached the discussion will be entirely different in scope.

As noted in the initial post, I’m already providing 2 buildbots that test the FIPS mode functionality. And I’m quite well versed in this part of the codebase and as mentioned by @vstinner we’ve been maintaining those patches for years. With the buildbot testing I don’t think it’s gonna be a big maintenance trouble.

Would starting with a PR with (almost) all the changes incorporated to provide a better overview of the scope make sense?

1 Like
8

Dumping here some notes about FIPS

  • FIPS is about interoperability; safe implementation; certified implementation

  • It means it picks winners (blake2 lost, keccak one to be crowned SHA-3)

  • it means implementation detail matter (inside openssl default & fips provider implementation of the same algorithm are often different with more safety checks)

  • it is time dependent (i.e. key lengths acceptable yesterday, might not be acceptable tomorrow, or be subject of usage type - verification & decryption, often are allowed for longer than signing & encryption).

  • exact implementation policy is subject to a security policy from a given vendor (i.e. FIPS 140-2 module from Rhel may have different runtime properties to a FIPS 140-2 module from RHEL on another release and different from FIPS 140-2 module from Canonical, etc)

  • each cryptographic module is stand alone (i.e. it is possible to create a chroot with non-fips openssl, on a host that has kernel in fips mode, meaning FIPS mode matters of the cryptographic module one is using directly; rather than anything else)

  • ideally higher level programming languages like Python, Node, Php, Ruby should delegate policy enforcement to the system provided OpenSSL in FIPS mode. And most do.

  • ideally try not to second guess what the particular runtime vendor does

Next I will point out implementation details for this exact same problem in other interpreter languages that too try to integrate with and expose OpenSSL FIPS mode higher up. As it helps a lot for end users when “openssl FIPS + nodejs” behaves the same as “openssl FIPS + python” and same as “openssl FIPS + php”. Because consistent runtime policy across different languages leads better results.

I will also provide different common ways for people to configure OpenSSL (with just default provider, base+fips providers, default+fips providers allowing opt out).

9

It’s explicitly not about safe implementation, it’s about the implementation being certified, and the value judgment on “certification” here is not in favor of safety but of legal requirement.

Commonly, it’s resulted in software that is FIPS certified to receive fixes less frequently as any change to fix something requires an expensive recertification process. [1]

There have been major issues with specific FIPS to call into question whether the regulation is even well-intentioned.[2] Using FIPS140-2 as an example here, it mandated that initial points for elliptical curves were only using ones that everyone has a strong reason to distrust.[1:1]

If you don’t have a government-mandated reason to enable fips compliant modes in software, you probably shouldn’t.

  1. Further reading on both of the above is available in the aftermath of DUAL_EC_DRBG for openssl here: 'Flaw in Dual EC DRBG (no, not that one)' - MARC ↩︎ ↩︎

  2. All reasonable signs point to these curves being intentionally weak, but more reading available here: https://safecurves.cr.yp.to for the evaluation of elliptical curves for cryptographic purposes. ↩︎

1 Like
10

FIPS 140-2 is obsolete, superseeded by ISO/IEC 19790:2012 standard. The curves you mention have been deprecated and removed in FIPS 186-5. And shipping updates doesn’t cost anything, and is possible - see for example 300+ Chainguard FIPS images that are all with near zero CVEs. So I have no idea what you are talking about.

Note, python doesn’t implement a cryptographic module nor is attempting to get certified. This discussion is about which OpenSSL C APIs make sense to bubble up and expose in python hashlib. If you are not interested, that’s fine, please do not derail this thread on how to improve hashlib for those that choose to use an ISO/IEC 19790:2012 cryptographic module with python.

11

I have no problem with those who need to comply with this doing so and enough being added to allow that to be possible, but please be careful not to mislead people about what the benefits of FIPS compliance actually is with specific claims that it is about safety when there is no reason to believe that the use of FIPS-compliant modes increase safety. I would have gladly not said anything in this thread if not for such a claim.

The history was relevant to the point even with the superseding standards.

I’d also encourage you to read the relevant linked footnotes, it wasn’t an idle or unsubstantiated claim. Quote from one of the footnotes on openssl-announce list

Even if we wanted to fix it our options are severely constrained by
the fact that the CMVP process forbids modifications of any
kind (even to address severe vulnerabilities) without the substantial
time and expense of formal retesting and review.

I’ll leave the authoritative link with current details about CMVP costs about this as well.

12

Here is a proposed flowchart to improve usedforsecurity flag decision making process for which backend to initialize for a given hash implementation.

One can click the diagram to go to mermaid online editor to edit the diagram. Both TD and LR directions render small in discourse. Go Full Screen

OpenSSL wants FIPS?

Note that OpenSSL can be default to fips with or without Linux kernel operating in FIPS mode (depends on the vendor of a given OpenSSL). However the below APIs work correctly with all known vendors of OpenSSL on Linux-like platforms, but also Windows & Mac OS.

  • For OpenSSL 1.1.1 call FIPS_mode() api
  • For OpenSSL v3+ call EVP_default_properties_is_fips_enabled() api

Example Implementations:

in OpenSSL*?

Note that OpenSSL v3+ has implemented Blake MAC API that allows to specify size, customization, key, salt parameterers. OpenSSL does not have blake tree implementation. I wonder if for blake we can make the choice slight better - i.e. requested constructor sequential (most common case) or not; and then appropriately use OpenSSL v3; or fallback to builtin implementation. This likely needs work to implement using Blake via OpenSSL MAC api. See EVP_MAC-BLAKE2 - OpenSSL Documentation

Load extra providers side-quest

Separately, one can take a side-quest. If one landed at this node of flowchart, and one is using OpenSSL v3+ one can attempt to load additional providers - for example default or legacy; to access more hashes. The reason being that MD5 and SHA1 might not be available in the default library context; if only base+fips providers or just default provider are loaded. And it may help loading legacy or default to gain access to additional hashes. This is effectively alternative way to access fallbacks.

This also can be used to load other 3rd party providers such as blake3 xxhash gost and so on.

Use OpenSSL

And it may fail at runtime. Note that in case openssl is in FIPS mode and usedforsecurity=True we land on this node, regardless of whether or not a requested digest is available in OpenSSL. The reason being is as follows. If the user went out of their way to configure OpenSSL to be in fips mode, it means they are trying hard to use only things allowed by the OpenSSL policy. In such cases, lets only use openssl irrespective of whether or not it provides MD4, MD5, Blake2B and so on. As it’s not our call to decide what the system configured OpenSSL policy actually is. Also note it can be very selective. For example, HMAC-MD5 could be allowed, whilst pure MD5 is blocked. Similarly for SHA1 and other algorithms.

in Builtin?

Once we are here, check if a builtin fallback was compiled in, and use it if available. Otherwise display a friendly error saying things like

Requested hash was not found in openssl. Please consider installing or loading legacy and/or default providers.

Requested hash was also not found in builtin fallback. Consider compiling python with a builtin fallback

Extra functionality

In golang-openssl and in nodejs they have ability to force loading FIPS provider (i.e. turn off fallbacks, ensure FIPS is enabled in OpenSSL and fail if it is not). I’m not too sure why such things were implemented, and if anybody uses them. All security policies from all vendors suggest that security officers configure the system-wide openssl, rather than each individual application. Thus telling the user to configure system openssl.cnf or provide a custom one is probably best.

Current issues

Whilst most things behave correctly, especially in the case of non-fips and usedforsecurity=False, there are some gaps in the current upstream implementation of hashlib:

  • Blake2 is used from builtin, even when OpenSSL wants FIPS and usedforsecurity=True
  • When python compiled without a builtin fallback, when OpenSSL is in FIPS mode and usedforsecurity=False default provider is not loaded to gain access to MD5
  • Friendly Errors are not very friendly - improving messaging to the user as to what has happened, and why things fail would help

ps. diagram is big in edit preview; but not site. Sorry about this, seems like there are hard caps placed on image size.

13

It’s explicitly not about safe implementation, it’s about the implementation being certified, and the value judgment on “certification” here is not in favor of safety but of legal requirement.

Something else that people often fail to acknowledge in these discussions is that FIPS is a standard put forward by one singular national government, dictating the rules they expect their own agencies and contractors to abide by. Many nations have similar standards of their own. Community-developed open source projects are increasingly global in both the scope of their users and maintainers, so placing a lot of importance on the standards of a single nation’s government is pretty out-of-place these days.

2 Likes