WebPerf Xz format inadequate for long-term archiving

eva2000 · Oct 23, 2016

Interesting piece titled Xz format inadequate for long-term archiving. Whether it's true or not compared to bzip2, lzip and gzip formats not sure.

This article started with a series of posts to the debian-devel mailing list [Debian], where it became clear that nobody had analyzed xz in any depth before adopting it in the Debian package format. The same unthinking adoption of xz seems to have happened in major free software projects, like GNU Coreutils and Linux. In my opinion, it is a mistake for any widely used project to become an early adopter of a new data format; it may cause a lot of trouble if any serious defect is later discovered in the format.
Click to expand...

For those who just want the conclusion

4 Conclusions
There are several reasons why the xz compressed data format should not be used for long-term archiving, specially of valuable data. To begin with, xz is a complex container format. Using a complex format for long-term archiving would be a bad idea even if the format were well-designed, which xz is not. In general, the more complex the format, the less probable that it can be decoded in the future by a digital archaeologist.

Xz is unreasonably extensible; it has room for trillions of compression algorithms, but currently only supports one, LZMA2, which in spite of its name is not an improved version of LZMA, but an unsafe container for LZMA data. Additionally, the xz format lacks a version number field, which makes xz's extensibility problematic. Xz is also fragmented by design. Xz implementations may choose what subset of the format they support. Interoperability among xz implementations is not guaranteed.

Error detection in the xz format is less accurate than in bzip2, gzip and lzip formats mainly because of false positives, and specially if an overkill check sequence like SHA-256 is used in xz. Another cause of false positives is that xz tries to detect errors in parts of the compressed file that do not affect decompression, like the padding added to keep the useless 4 byte alignment. In total xz reports several times more false positives than bzip2, gzip or lzip, and every false positive may result in unnecessary loss of data.

Xz uses variable-length integers unsafely, specially when they are used to store the size of other fields or when they are concatenated together. But probably the worst defect in xz is that it fails to protect critical fields like length fields and flags signalling the presence of optional fields. These defects make xz fragile, meaning that most of the times when it reports a false positive, the decoder state is so mangled that it is unable to recover the decompressed data. They also reduce the value of xz as a general-purpose format because anybody wanting to archive a file already compressed in xz format will have to either leave it as-is and face a larger risk of losing the data, or waste time recompressing the data into a format more suitable for long-term archiving.

The weird combination of fragile constructs (like unprotected length fields) and overkill check sequences can only be explained by the inexperience of the designers of xz. It is said that given enough eyeballs, all bugs are shallow. But the adoption of xz by several GNU/Linux distributions shows that if those eyeballs lack the required experience, it may take too long for them to find the bugs. It would be an improvement for data safety if compressed data formats intended for broad use were designed by experts and peer reviewed before publication. This would help to avoid design errors like those of xz, which are very difficult to fix once a format is in use.
Click to expand...

Log in / Register

Forums

Want more timely Centmin Mod News Updates?

WebPerf Xz format inadequate for long-term archiving

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Want more timely Centmin Mod News Updates?

WebPerf Xz format inadequate for long-term archiving

eva2000 Administrator Staff Member

.