Checksum Algorithm Benchmarking

We're implementing checksum providers into DDF. Our initial benchmark testing is intended to give a brief overview into the performance of each of the checksums tested.

Assumptions and Decisions

Certain assumptions and decisions were made during this testing:

The file sizes tested (~5mb,~50mb,~200mb) are sufficient for giving us an idea of performance for each algorithm tested.
5 tests for each size is a sufficient sample size for benchmarking.

CRC32 Benchmark Results (in seconds)

Test #	~5MB	~50MB	~200MB
1	.005	.106	.576
2	.004	.082	.216
3	.005	.054	.24
4	.005	.052	.401
5	.005	.053	.239

MD5 Benchmark Results (in seconds)

Test #	~5MB	~50MB	~200MB
1	.015	.208	1.038
2	015	.208	.731
3	.016	.214	.959
4	.016	.21	.705
5	.015	.258	.963

~5MB Comparison (in seconds)

Algorithm	Min	Max	Avg
CRC32	.004	.005	.0048
MD5	.015	.016	.0154

~50MB Comparison (in seconds)

Algorithm	Min	Max	Avg
CRC32	.052	.106	.06974
MD5	.208	.258	.2196

~200MB Comparison (in seconds)

Algorithm	Min	Max	Avg
CRC32	.216	.576	.3344
MD5	.705	1.038	.8792

~MB/s Comparison

Algorithm	~5MB	~50MB	~200MB
CRC32	.00096	.001388	.001672
MD5	.00308	.004392	.004396

~ Average MB/S Comparison

Algorithm	~Avg MB/s
CRC32	.00134
MD5	.003956

Result Analysis

There seems to be a skew in the data for > ~50MB and the first run taking significantly longer than other test runs.
CRC32 has the lowest average approximate MB/s.
MD5 seems to be outperformed in every metric in each of its size categories.

Decision

Based on customer requirements, added Adler32 for performance (faster than CRC32) and SHA-256 for secure checksums.

https://github.com/codice/ddf/tree/2.27.x/libs/checksum/src/main/java/org/codice/ddf/checksum/impl

DDF