We're implementing checksum providers into DDF. Our initial benchmark testing is intended to give a brief overview into the performance of each of the checksums tested.
Assumptions and Decisions
Certain assumptions and decisions were made during this testing:
- The file sizes tested (~5mb,~50mb,~200mb) are sufficient for giving us an idea of performance for each algorithm tested.
- 5 tests for each size is a sufficient sample size for benchmarking.
...
| Algorithm | ~Avg MB/s |
|---|---|
| CRC32 | .00134 |
| MD5 | .003956 |
Result Analysis
- There seems to be a skew in the data for > ~50MB and the first run taking significantly longer than other test runs.
- CRC32 has the lowest average approximate MB/s.
- MD5 seems to be outperformed in every metric in each of its size categories.
...
Decision
Based on customer requirements, added Adler32 for performance (faster than CRC32) and SHA-256 for secure checksums.
https://github.com/codice/ddf/tree/2.27.x/libs/checksum/src/main/java/org/codice/ddf/checksum/impl