[RFC] Change OpenSSL derived digest functions to return boolean values

[Jun Kuriyama]

Hi,

I put a patch to Phabricator and Github.

https://reviews.freebsd.org/D1542
https://github.com/kuriyama/freebsd/compare/openssl-digest-return-value

Any comments are welcome!


DESCRIPTION

OpenSSL changed return value type of *_{Init,Update,Final}() functions
at 2001 [1].  Our implementations at libmd do not follow these changes.

[1] https://github.com/openssl/openssl/commit/2dc769a1c17e1e0c7aef6e11496c8ba2c1db2e28


BACKGROUND

I noticed this problem during using install(1) with net/nss_ldap,
net/openldap24-client and ldaps:// protocol.  While install(1) is
linked with libmd, but ports libldap is compiled with libcrypto to
expect OpenSSL's SHA1_Update() function.  When using install(1) in this
situation, install(1) uses libmd's SHA1_Update(), but wrapping
functions in OpenSSL expects SHA1_Update() to return boolean. This
causes sometimes fails SHA1_Update() (which depends on value of EAX
register?) call.

Problem is, we have SHA1_Update() functions in libmd and libcrypto,
and both has different return value types.  This should be same if they
provides identical functionality.


TBD

Should adjust {SHA{256,512}_,MDX}{Init,Update,Final}() functions, too?


-- 
Jun Kuriyama <kuriyama at FreeBSD.org> // FreeBSD Project
         <kuriyama at s2factory.co.jp> // S2 Factory, Inc.