Is calculating an MD5 hash less CPU intensive than SHA family functions?_问答_开发者

Is calculating an MD5 hash less CPU intensive than SHA-1 or SHA-2 on "standard" laptop x86 hardware? I'm interested in general information, not specific to a certain chip.

UPDATE: In my case, I'm interested in calculating the hash of a file. If file-size matters, let's assume its 300K.开发者_JAVA百科

Yes, MD5 is somewhat less CPU-intensive. On my Intel x86 (Core2 Quad Q6600, 2.4 GHz, using one core), I get this in 32-bit mode:

MD5       411
SHA-1     218
SHA-256   118
SHA-512    46

and this in 64-bit mode:

MD5       407
SHA-1     312
SHA-256   148
SHA-512   189

Figures are in megabytes per second, for a "long" message (this is what you get for messages longer than 8 kB). This is with sphlib, a library of hash function implementations in C (and Java). All implementations are from the same author (me) and were made with comparable efforts at optimizations; thus the speed differences can be considered as really intrinsic to the functions.

As a point of comparison, consider that a recent hard disk will run at about 100 MB/s, and anything over USB will top below 60 MB/s. Even though SHA-256 appears "slow" here, it is fast enough for most purposes.

Note that OpenSSL includes a 32-bit implementation of SHA-512 which is quite faster than my code (but not as fast as the 64-bit SHA-512), because the OpenSSL implementation is in assembly and uses SSE2 registers, something which cannot be done in plain C. SHA-512 is the only function among those four which benefits from a SSE2 implementation.

Edit: on this page (archive), one can find a report on the speed of many hash functions (click on the "Telechargez maintenant" link). The report is in French, but it is mostly full of tables and numbers, and numbers are international. The implemented hash functions do not include the SHA-3 candidates (except SHABAL) but I am working on it.

On my 2012 MacBook Air (Intel Core i5-3427U, 2x 1.8 GHz, 2.8 GHz Turbo), SHA-1 is slightly faster than MD5 (using OpenSSL in 64-bit mode):

$ openssl speed md5 sha1
OpenSSL 0.9.8r 8 Feb 2011
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              30055.02k    94158.96k   219602.97k   329008.21k   384150.47k
sha1             31261.12k    95676.48k   224357.36k   332756.21k   396864.62k

Update: 10 months later with OS X 10.9, SHA-1 got slower on the same machine:

$ openssl speed md5 sha1
OpenSSL 0.9.8y 5 Feb 2013
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              36277.35k   106558.04k   234680.17k   334469.33k   381756.70k
sha1             35453.52k    99530.85k   206635.24k   281695.48k   313881.86k

Second update: On OS X 10.10, SHA-1 speed is back to the 10.8 level:

$ openssl speed md5 sha1
OpenSSL 0.9.8zc 15 Oct 2014
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              35391.50k   104905.27k   229872.93k   330506.91k   382791.75k
sha1             38054.09k   110332.44k   238198.72k   340007.12k   387137.77k

Third update: OS X 10.14 with LibreSSL is a lot faster (still on the same machine). SHA-1 still comes out on top:

$ openssl speed md5 sha1
LibreSSL 2.6.5
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              43128.00k   131797.91k   304661.16k   453120.00k   526789.29k
sha1             55598.35k   157916.03k   343214.08k   489092.34k   570668.37k

As someone who's spent a bit of time optimizing MD5 performance, I thought I'd supply more of a technical explanation than the benchmarks provided here, to anyone who happens to find this in the future.

MD5 does less "work" than SHA1 (e.g. fewer compression rounds), so one may think it should be faster. However, the MD5 algorithm is mostly one big dependency chain, which means that it doesn't exploit modern superscalar processors particularly well (i.e. exhibits low instructions-per-clock). SHA1 has more parallelism available, so despite needing more "computational work" done, it often ends up being faster than MD5 on modern superscalar processors.
If you do the MD5 vs SHA1 comparison on older processors or ones with less superscalar "width" (such as a Silvermont based Atom CPU), you'll generally find MD5 is faster than SHA1.

SHA2 and SHA3 are even more compute intensive than SHA1, and generally much slower.
One thing to note, however, is that some new x86 and ARM CPUs have instructions to accelerate SHA1 and SHA256, which obviously helps these algorithms greatly if the instructions are being used.

As an aside, SHA256 and SHA512 performance may exhibit similarly curious behaviour. SHA512 does more "work" than SHA256, however a key difference between the two is that SHA256 operates using 32-bit words, whilst SHA512 operates using 64-bit words. As such, SHA512 will generally be faster than SHA256 on a platform with a 64-bit word size, as it's processing twice the amount of data at once. Conversely, SHA256 should outperform SHA512 on a platform with a 32-bit word size.

Note that all of the above only applies to single buffer hashing (by far the most common use case). If you're fancy and computing multiple hashes in parallel, i.e. a multi-buffer SIMD approach, the behaviour changes somewhat.

The real answer is : It depends

There are a couple factors to consider, the most obvious are : the cpu you are running these algorithms on and the implementation of the algorithms.

For instance, me and my friend both run the exact same openssl version and get slightly different results with different Intel Core i7 cpus.

Update 2021 Ran openssl speed sha1 md5 on a Ryzen 9 3900x : Sha1 is now 2-3 times faster than md5 and the difference increases as the data size increases

The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
md5             171084.26k   373867.24k   660204.56k   783808.17k   840138.75k   843743.23k
sha1            309769.46k   772013.89k  1523885.48k  2017251.67k  2226836.82k  2251024.61k

End update

My test at work with an Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz

The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              64257.97k   187370.26k   406435.07k   576544.43k   649827.67k
sha1             73225.75k   202701.20k   432679.68k   601140.57k   679900.50k

And his with an Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz

The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              51859.12k   156255.78k   350252.00k   513141.73k   590701.52k
sha1             56492.56k   156300.76k   328688.76k   452450.92k   508625.68k

We both are running the exact same binaries of OpenSSL 1.0.1j 15 Oct 2014 from the ArchLinux official package.

My opinion on this is that with the added security of sha1, cpu designers are more likely to improve the speed of sha1 and more programmers will be working on the algorithm's optimization than md5sum.

I guess that md5 will no longer be used some day since it seems that it has no advantage over sha1. I also tested some cases on real files and the results were always the same in both cases (likely limited by disk I/O).

md5sum of a large 4.6GB file took the exact same time than sha1sum of the same file, same goes with many small files (488 in the same directory). I ran the tests a dozen times and they were consitently getting the same results.

It would be very interesting to investigate this further. I guess there are some experts around that could provide a solid answer to why sha1 is getting faster than md5 on newer processors.

MD5 also benefits from SSE2 usage, check out BarsWF and then tell me that it doesn't. All it takes is a little assembler knowledge and you can craft your own MD5 SSE2 routine(s). For large amounts of throughput however, there is a tradeoff of the speed during hashing as opposed to the time spent rearranging the input data to be compatible with the SIMD instructions used.

sha1sum is quite a bit faster on Power9 than md5sum

$ uname -mov
#1 SMP Mon May 13 12:16:08 EDT 2019 ppc64le GNU/Linux

$ cat /proc/cpuinfo
processor       : 0
cpu             : POWER9, altivec supported
clock           : 2166.000000MHz
revision        : 2.2 (pvr 004e 1202)

$ ls -l linux-master.tar
-rw-rw-r-- 1 x x 829685760 Jan 29 14:30 linux-master.tar

$ time sha1sum linux-master.tar
10fbf911e254c4fe8e5eb2e605c6c02d29a88563  linux-master.tar

real    0m1.685s
user    0m1.528s
sys     0m0.156s

$ time md5sum linux-master.tar
d476375abacda064ae437a683c537ec4  linux-master.tar

real    0m2.942s
user    0m2.806s
sys     0m0.136s

$ time sum linux-master.tar
36928 810240

real    0m2.186s
user    0m1.917s
sys     0m0.268s

[
Is MD5 faster or SHA1? ]

It's implementation dependent:

|*|
[
Theoretically the MD5 algorithm would do less work than SHA1, but the design of MD5 itself determined that the algorithm cannot effectively exploit computation parallelism (i.e. cannot effectively utilize a multi-processor system; or processors that utilize instruction-level parallelism). While SHA1 would provide better opportunity for so.

This is part of the reason why in some implementations SHA1 would outperform MD5. ]

|*|
[
There are also processors that provide dedicated hardware acceleration support for SHA1.

When properly utilized, such implementations tend to easily outperform software based MD5 implementations:

[ Quote dr-js @ CE 2021-01-28 10:31 UTC:
https://security.stackexchange.com/a/95697

2021 update with OpenSSL 1.1.1d: now we see md5 is often slower on newer CPU, and for larger chunks:

[
## PC i7-1165G7 @ 2.80GHz (2020)
OpenSSL 1.1.1d  10 Sep 2019 / built on: Mon Dec  7 20:44:45 2020 UTC
type      16 bytes    64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
md5     189018.70k  418310.85k   712090.28k   890189.14k   956293.12k   962560.00k
sha1    287134.62k  746529.17k  1474064.38k  1973607.08k  2197842.60k  2192179.20k
sha256  222301.71k  603962.47k  1213340.33k  1665262.59k  1849016.32k  1847388.84k

## Server AMD EPYC 7571 (2018)
OpenSSL 1.1.1d  10 Sep 2019 / built on: Mon Dec  7 20:44:45 2020 UTC
type      16 bytes    64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
md5      93668.33k  213979.18k   378971.56k   467472.38k   501205.67k   504064.68k
sha1    165020.82k  442991.72k   888443.48k  1188591.62k  1319236.95k  1330080.43k
sha256  142886.55k  375612.63k   791567.70k  1095950.34k  1234381.48k  1246827.86k

## Server E5-2682 v4 @ 2.50GHz (2016)
OpenSSL 1.1.1d  10 Sep 2019 / built on: Mon Dec  7 20:44:45 2020 UTC
type      16 bytes    64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
md5     101505.24k  207422.92k   393158.83k   453332.99k   527085.34k   490711.72k
sha1     98091.83k  249828.79k   389640.36k   675694.25k   686966.33k   721021.61k
sha256   55421.86k  130103.33k   251929.17k   302571.86k   296977.81k   338439.56k
] ]

Worth noticing that even SHA-256 could be faster than MD5 in such cases. ]


To put it in a simple (though not so accurate) statement:

|*| For high-end processors, SHA1 tends to be faster.
|*| For low-end processors, MD5 would be faster.


[ Quote Nyan @ CE 2020-12-10 10:18 UTC:
https://stackoverflow.com/a/64928816

Note that all of the above only applies to single buffer hashing (by far the most common use case). If you're fancy and computing multiple hashes in parallel, i.e. a multi-buffer SIMD approach, the behaviour changes somewhat. ]