Add option to ignore CRC for bzip2

smiRaphi · July 22, 2026, 6:06pm

I would like an option to ignore the CRC in the bzip2 module as it would be really helpful for either broken bzip2 streams that need to be recovered or for decompressing proprietary formats based on bzip2 but without a CRC.

Here’s a small example of what this would look like:

import bz2

d = b'test'
z = bz2.compress(d)
z = z[:10] + b'\0'*4 + z[14:] # nullify crc

u = bz2.BZ2Decompressor(ignore_crc=True).decompress(z)
assert u == d
# raises regular "OSError: Invalid data stream" exception
bz2.BZ2Decompressor(ignore_crc=False).decompress(z)

My use case would be to decompress the Xceed BWT format that’s basically a bzip2 stream but without the signature & CRC:

github.com/smiRaphi/UniPyX

lib/file.py

0392f9264


      
          
              ibuf = (ctypes.c_uint8 * isz).from_buffer_copy(i)
              obuf = (ctypes.c_uint8 * us)()
          
              r = GDEFLATE.DecompressData(ibuf,isz,obuf,us)
              if r == 0: raise ValueError('Failed to decompress')
              return bytes(obuf)
          case 'xmem'|'xmem_lzx': return uxx().decompress_xmemlzx(i,kwargs['usize'])
          case 'xb'|'xbcompress': return uxx().decompress_xb(i)
          case 'capcom_yz2': return uxx().decompress_capcom_yz2(i,kwargs['usize'])
          case 'xceed_bwt':
              us = kwargs['usize']
              if us == 0: return b''
              import bz2
          
              # I'm too lazy to write a whole bzip2 decompressor so this coaxes bz2 into decompressing without a known valid crc
              be = by2bi(i[-2:])
              be = be[:be.rfind('00010111') + 8] + by2bi(b'rE8P\x90')
              bd = b'BZh' + i[:2] + b'AY&SY'
          
              if us > 1:

jamestwebber · July 22, 2026, 7:16pm

Why does this need to be in the standard library? A rare proprietary format seems like a good use-case for a package, no?

I’m not sure how you’d use this to debug a broken stream…

smiRaphi · July 22, 2026, 7:33pm

It’s not about adding support for some obscure format, it’s just about adding an option to ignore the CRC so you can hack together & decompress a valid (besides the CRC) bzip2 stream without having to compile/write a whole custom bzip2 decompressor.
And for the broken stream “debugging”, it’s less debugging and more data recovery, for example if you have a malformed stream because your hard drive crashed (just as an example), you could still decompress it and recover some of the data.

barry-scott · July 22, 2026, 7:53pm

Are you sure? It is doubtful that a hole in the data will allow the stream to be decoded.

Have you experimented with damaging bzip2 files and seeing if you can recover any data after the damage?

Rosuav · July 22, 2026, 8:07pm

Hmm. I would tackle this in the opposite direction. If you have a zip file but without its surrounding information, what you actually have is a deflation stream. If there’s something that’s basically a bz2 file but without its signature, that sounds like it’s a more raw way to use the underlying compression algorithm.

That’s about as far as I can take it though, since the Python bz2 module calls on the C _bz2 module, which just directly calls the bzDecompress function from libbzip2, and it’s that function that returns BZ_DATA_ERROR for failed CRCs. So this is likely impractical, unfortunately. IMO there ought to be a way to access the compression stream independently of that extra info, but it might require some hackery.

smiRaphi · July 22, 2026, 8:07pm

It’s unlikely but sometimes it is possible, especially if the error is with the CRC or the magic/signature bytes

smiRaphi · July 22, 2026, 8:15pm

I would more compare it with zlib + deflate and not zip + deflate, but the issue with that comparison is that deflate is a standard standalone format but raw bzip2 (or whatever you wanna call it) is not, I’ve did a quick google and yeah libbzip2 doesn’t seem to have a “ignore crc” flag but I feel like it wouldn’t be too hard to add one but I’m also not well versed in licenses and all that so not sure if there are any issues with that.

Rosuav · July 22, 2026, 8:25pm

Yeah, that’s the issue, that’s why it would require hackery. Whether it’s “access the underlying stream” or “disable the CRC check while decompressing”, it would require support from the underlying libbzip2. That means changing the library, making sure the new version of the library is used, making sure it works on all platforms, and a ton of other effort, most of which is outside of CPython altogether.

I’m not entirely sure what your code is doing there, but line 948 is clearly adding the header. You aren’t calculating a correct checksum and putting it in there, which is what I would have thought would be necessary. Maybe there’s a “null checksum” value?? In any case, though, what you have is (unfortunately) likely the best you’re going to get.

But given that this is closely related to repairing/recovering damaged files, it may be of value to look into how existing bzip2 recovery works. Might be some useful tricks to pick up.

smiRaphi · July 22, 2026, 8:29pm

My code has 2 routes, if you supply it with a check function, it’ll just bruteforce the last byte until that function returns true, if that’s not there, it’ll bruteforce the last byte, calculating the bzip2 crc32 for that and inject it into the stream and then try to decompress it again, if it works, return the decompressed bytes, if not, try the next byte.