Sudden trouble with net/rdesktop

[Greg Veldman]

On Fri, Nov 13, 2020 at 05:53:06PM +0300, Yuri Pankov wrote:
> size is 128 now, so with digest of 256 we fail the ((len * 2) <  size) 
> assertion, 257 works though (or changing the assertion to be <=).

The call to gnutls_x509_crt_get_fingerprint adjusts the size
of buf.  I looked at the size immediately before the call to
_utils_data_to_hex() for a couple of servers I have access to,
and it was much smaller than 128 (20 and 32).  I am not familiar
enough with the internals of GnuTLS to know if it can be adjusted
up as well, but as I stated before I think the intention was that
128 would be the largest size and the actual value of buf_size
in the calls to _utils_data_to_hex() would depend on the actual
size of the fingerprints of the cert.  That appears to be why
it worked before when digest was defined as 128 chars.

My guess is Microsoft changed some parameters of the certificates
in newer Windows versions and the resulting fingerprints are
longer.

> Actually, this does not look like a real fix.  I have looked more into 
> it and something is still very wrong -- _utils_cert_get_info() calls 
> gnutls_x509_crt_get_fingerprint() and does NOT check the return value 
> while it is -72 (GNUTLS_E_ASN1_VALUE_NOT_VALID), while the only 
> *documented* return values are 0 (got fingerprint successfully) and -51 
> (GNUTLS_E_SHORT_MEMORY_BUFFER, not enough buffer space).  I don't know 
> if it was the same previously as I don't have a system to test against. 
>   If yes, then this patch could be used in ports at least until upstream 
> fixes the problem properly; if no, then we would need a proper fix first.
> 
> Also note that sha1 and sha256 fingerprints reported are the *same*, 
> which doesn't look right with this approach.

Indeed, it would seem as though a good chunk of this code path
makes assumptions that may no longer be valid and is perhaps in
need of some rework.  But for the purposes of patches in the
ports tree, it seems like the best approach would be to change
things by the minimal amount necessary to fix the issue.

-- 
Greg Veldman
greg at gregv.net