[nfsv4] testing of RPC over TLS

[grarpamp]

On 11/26/20, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> The implementation of RPC over TLS for freebsd is now basically
> done and setting up a test system is fairly straightforward.

Nice.

> If you are interested see
> https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt
>
> If you have any questions about it, feel free to ask, rick

How do users and administrators pin down the
public key fingerprint of the NFS server's TLS certificate
so that the hash is checked and thus a MITM
attack is prevented?


>From March on list...
Implementations must provide users with TLS public key
fingerprint pinning options. Implementations that do not offer
those options are exposing their users to MITM risks (those
risks are not mitigated by the legacy wrong answer of 'trust
the [rogue and or MITMable] "CA" system').
MITM attacks are in play by many parties over many networks.
*ALL* applications that speak TLS today must make these
pubkey fingerprint pinning options available for their users.


Cheers.





https://tools.ietf.org/html/draft-ietf-nfsv4-rpc-tls

People appear to be talking about using and
"authenticating / verifying" TLS certs now with at least
perhaps this NFSv4, and certainly with other apps.

If so, it's required critical thing for the admins and users to have
the option to pin the certificate pubkey fingerprints in four ways...

- Ignore the CA chain / expiry / etc, validate only the fingerprint.
- Validate the CA chain / expiry / etc, and validate the fingerprint.
- Validate the CA chain / expiry / etc, ignore the fingerprint.
- A TOFU mode, with some management by fingerprint options.

No application that uses TLS should be considered completely
featured and security capable without fingerprint pinning functions.

The "SHOULD implement" loophole in 5.5.2 will end up with many OS
and implementations lacking such basic security feature option.

For some background reasons on why pubkey fingerprint pinning
implementations are now showing up in softwares that speak TLS,
and for sample code, and related infos, see the links...

https://owasp.org/www-community/controls/Certificate_and_Public_Key_Pinning
https://cheatsheetseries.owasp.org/cheatsheets/Pinning_Cheat_Sheet.html

https://curl.haxx.se/libcurl/c/CURLOPT_PINNEDPUBLICKEY.html
--pinnedpubkey <hashes | file>
Tells curl to use the specified public key file (or hashes) to verify the
peer. This can be a path to a file which contains a single public key in PEM
or DER format, or any number of base64 encoded sha256 hashes preceded by
'sha256//' and separated by ';'.
When negotiating a TLS or SSL connection, the server sends a certificate
indicating its identity. A public key is extracted from this certificate and
if it does not exactly match the public key provided to this option, curl will
abort the connection before sending or receiving any data.

Please note this option is rightly very specific covering only the
isolated pubkey, not the DER form of the entire "CA signed" cert
(ie: not the typically referenced coverage of "openssl x509 -fingerprint").
This allows additional adaptability to some cert environments.

Complete fingerprint implementations need both modes: pubkey, and cert DER.

Also note the use of sha-256, not the broken and now deprecated sha-1,
sha-3 could function as a backstop.

When fully implemented, fingerprint pinning enables a local admin and
user environment of more flexible certificate validation service cababilities
and security model hardening when subject to various third party things
and adversaries like...

- Environment of rogue / forced / spy MITM CA's, TLS termination / proxy
cloud MITM, VPN / overlay / WiFi networks MITM, etc.
- Annoying "expired" certs awaiting tax revenue from their captured audience.
- Assigning pinned trust to intermediate CA's such as Lets Encrypt, Google,
and corporate schemes, to let edge server certs they sign be freely
rotated and or freshly signed without need to update pin.
- Avoid need to update pin every "expiry" period.
- Avoid CA's by using cert owners publicly available and out of band self
certified hash attestations found on keybase, social, observatories, PGP, etc.
- As mentioned above, optionally in combination with other CA / expiry / etc
checks, or ignoring the CA altogether.
- CRL checks are a massive metadata privacy and user monetization
leak that some users might not want exposed to.
- Pinning one or both of: pubkey (herein) and or CA (openssl x509 -fingerprint)

Another very useful security feature to have is a trust on first use TOFU
mode that stores, pins, and subsequently validates against those fingerprints,
similar to SSH model. This is useful for both known comms partners such as
client-server model, and in more distributed group or even p2p applications
to help keep things a bit more locked down by default.

Defense (like this pubkey fingerprint pinning) in depth... you can use it :)


References (obviously TLS_1.3 is todays version to use)...

https://www.netcraft.com/internet-data-mining/ssl-survey/
https://www.ssllabs.com/ssl-pulse/
https://arstechnica.com/gadgets/2018/10/browser-vendors-unite-to-end-support-for-20-year-old-tls-1-0/
https://www.bleepingcomputer.com/news/security/ietf-approves-tls-13-as-internet-standard/
https://en.wikipedia.org/wiki/Transport_Layer_Security
https://tools.ietf.org/html/rfc8446

https://github.com/OWASP/www-community/blob/master/pages/controls/Certificate_and_Public_Key_Pinning.md
https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Pinning_Cheat_Sheet.md

https://github.com/curl/curl/blob/master/docs/cmdline-opts/pinnedpubkey.d
https://github.com/curl/curl/blob/deb9462ff2de8e955c67ed441f5f48619a31198d/docs/libcurl/opts/CURLOPT_PINNEDPUBLICKEY.3
https://github.com/curl/curl/blob/51fde337471c9125e7bf425e7ce0a0bf53691992/docs/TODO#L728

On 3/30/20, Black, David <David.Black at dell.com> wrote:
> This looks interesting - is there a BCP (Best Current Practice) RFC or
> similar reference on what MUST/SHOULD to be implemented and how implementers
> ought to think about this to make good choices?  Or is there one being
> prepared?

Not aware of one, maybe the list knows of some more resources?
Would be good to have some more documents in the space.
In particular herein regarding fingerprinting... differentiating the
two different cert DER vs pubkey forms, their various features and
capabilities in real world cert schemes scenarios, and re MITM.

There is a general encryption proponent mandate RFC,
commensurate with Snowden events, the number escapes
at this moment.

Yes, more development of such a BCP as you mention regarding
TLS fingerprint capabilities, and TLS extent forward in general,
would be nice. That would then extend to many protocols
in a default guide of reference way as to what is possible,
including say NFS.

It is difficult to say "MUST" or "SHOULD" or "MAY"
regarding this fingerprinting business for NFS.
But it may be reasonable that 5.5.2 be expanded to mention
the two different fingerprint classes, the four modes, as reasonable
desireables. And to then reference out to the OWASP links at the
bottom of the RFC.

Maybe then leverage that effort into a BCP with other
application RFC groups regarding TLS.

> For example, such a document ought to cover the tradeoffs among
> the four validation options, and how a safe default be chosen.   In general,
> this looks like a good direction as trust anchors and trust anchor
> management have always been a weak point in the web PKI
>
> The OWASP pages are a good start, but leave revocation for future work and
> are weak on certificate replacement - in particular, the comment that the
> "application would need to be updated regularly" appears to punt the
> underlying app validation problem upward to the app store operators (e.g.,
> to defend against a rogue app that accesses the server via an MITM proxy
> server), which doesn't mesh well with "An application which pins a
> certificate or public key no longer needs to depend on others - such as DNS
> or CAs".  Some dependence on the app store operator may be unavoidable for
> first-use.

It is assumed, and guaranteed under public embarrassment
of news media, that revocation will results in a physical regen
and change of cert (triggering both cert DER and pubkey
hard physical swapouts). Thus revocation, and the whole
revocation "authority" repository CRL OCSP checks etc,
is not at all coming into an issue in this fingerprint context,
since they are entirely independant and fixed and local,
and detect and resolve all such revocation swaps by
very nature of fingerprint failure and fingerprint updates.

The "application being updated regularly" is of course a nice
option for such app shipping devs to do gratuitously for users.
But has no bearing on users subsequent needed capability
to pin whatever they want and feel is necessary independantly.

If I were an NFS users in particular infrastructures and risks,
I might want to pin the end server pubkey (or cert DER),
or intermediate instances of the same (leaving server cert
free to float) regardless of what my vertical or lateral silo,
or other peer model, or CA, or rogue, says to accept.

Is there a reasonable way to more strongly suggest implementation
of those two fingerprint methods, and even some of the four modes,
as options between server-client in the NFS TLS RFC?


> There is a general encryption proponent mandate RFC...

One might also try full text RFC search for "fingerprint"
references in context of "TLS" (or the older "SSL").

The following incomplete list all in manner of spirit do
generally give rise and encourage the offering and enablement
of such security option sets in apps / protocols utilizing TLS...

Recommendations for Secure Use of Transport Layer Security (TLS)
https://tools.ietf.org/html/rfc7525
Pervasive Monitoring Is an Attack
https://tools.ietf.org/html/rfc7258
Privacy Considerations for Internet Protocols
https://tools.ietf.org/html/rfc6973
Certificate Transparency
https://tools.ietf.org/html/rfc6962
Strong Security Requirements for Internet Engineering Task Force
Standard Protocols
https://tools.ietf.org/html/rfc3365
Guidelines for Writing RFC Text on Security Considerations
https://tools.ietf.org/html/rfc3552
IETF Policy on Wiretapping
https://tools.ietf.org/html/rfc2804
IAB and IESG Statement on Cryptographic Technology and the Internet
https://tools.ietf.org/html/rfc1984


"It is the consensus of the IETF that IETF standard protocols MUST
make use of appropriate strong security mechanisms."


Privacy Requirements for IETF Protocols
https://tools.ietf.org/html/draft-cooper-ietf-privacy-requirements-01
   It is the consensus of the IETF that our protocols be designed to
   avoid privacy violations to the extent possible.

Handling Pervasive Monitoring in the IETF (perpass) (WG)
https://www.ietf.org/proceedings/88/perpass.html
https://www.ietf.org/mailman/listinfo/perpass


Not requiring app (NFS) spec implementations to have some options
for some of the previously noted modes of fingerprint checking to help
avoid some TLS MITM attacks, would seem to be in conflict with all above.


> the premise that pinning by fingerprint is the
> strongest security/authentication mechanism possible in many cases.

It is certainly among the set of them that might be applied
in various ways depending on the users security model.

> In IETF documents we essentially universally leave the deployment's
> selection of trust anchor(s) as a matter of local policy, and thus it is a
> bit out of character for us to be recommending specific trust anchors or
> classes of trust anchor.  We provide robust security mechanisms and
> describe how they can be safely used, but leave policy to others.

Allow to better define below the context as being in
supporting the presence of options for policies...

> I'm not sure I understand your point here.  You seem to be saying that
> CA-driven revocation is orthogonal to key pinning, and thus that when a
> revocation event occurs, the legitimate counterparty will generate a new
> key (and fingerprint), with the resulting connection errors being detected
> and used as a trigger to update local configuration.
> But that doesn't make sense to me!  If there's a revocation event and
> presumed key compromise, then the attacker that got the key can continue to
> use it against the deployment that pinned that key until the configuration
> is changed.  This seems to be ignoring the signal from the CA that
> "something bad happened" and could lead to a long period of vulnerability.
> It's far from clear to me that key pinning provides superior properties to
> CA-based setups in the face of such revocation events.

It is risky and then embarrassing to continue using a compromised
private key. The compromise is the preceding and defining element,
and rightly triggers a security rekey event. The rekey hash mismatch is,
the and only required, signal when a users application is watching the
fingerprint, if that is how the user configured their options.
This is independant, and if so chosen "orthagonal", from any assertion
or not as such by or to any CA signal crl scheme, that assertion being
at that point, equally the same as pinning check, a separate function
from the security compromise and restoration itself.
Fingerprints, "CA signals", other checks, are independant cases, and
can be combined in some users environments, provided they have
options to do so.
Further, many environments use CA only to avoid software CA nag
warnings and have little knowledge or care for CRL systems, and
many users have little knowledge or care to check them.
People also often try to trot out this false argument of relying "CA signal"
or revocation as some means to moot or demur the implementing of
fingerprinting options in the app code, or to avoid configuring them.
That is not the case.



As to NFS... in this suggestion, nfsv4-rpc-tls in 5.2.2
needs to REQUIRE all conformant implementations
to have fingerprint configuration OPTIONS for the user
to use as they deem appropriate. This does NOT
require users to use it, only that it be present
as option in conformant implementations.

If CA method of 5.2.1 is a REQUIRED security thing, then
similarly, so can fingerprint security OPTION things of 5.2.2 be.

If nfsv4-rpc-tls anywhere REQUIRES any things
of any conformant implementations, then so can it
require some fingerprint OPTIONS things.

If nfsv4-rpc-tls 5.2.2 is already specifying a
SHOULD-hint as to a "via"-form of SHA-256 being
over the cert-DER form (with or without including
reference to just what spec that is), it really must
also fairly provide a SHOULD-hint as to the "via"-form
of pubkey. That is probably simple enough, leaving
the four modes and other option combinatrics up
to the application teams, the possibility for such
potentially hinted in note in section 7.

External references are already possible by example
of URI list in 9.3. Additionally, perhaps something in
RFC5280 or similar could be noted or pointed to
as RFC5280 was noted and or pointed to in 5.2.1.

RFC's even have places available for further discussion
and strong words encouragement, such as section 7.

> fingerprint pinning is hardly ubiquitous in those protocols.

Unfortunately that is largely because fingerprint pinning is a
hardly understood black magic of additional TLS defence, in
depth, hardly mentioned, referenced, documented, implemented,
or educated on, etc. Most don't really understand MITM modes.
And with all the noise droning on about CA, $fees, "validation"
of corporate entites, advertising, who can blame them.
This issue is a similarly common copout / chicken egg to the
above CA CRL excuse for not creating fingerprint options.

Beneficial ubiquity can be fostered by more rfc's,
internet application docs, etc becoming a part of the
set of same that begins and further integrates more
mentions and supports for pinning options in their
various works.

It would suck to effort to add this TLS to NFS but
not enjoy some fingerprint pinning options along with it,
because, well, it was only a SHOULD, so your OS
skipped them resulting in less than featureful. Oops.

> It would perhaps be compatible to focus on the choice of trust anchor and
> the possibility to pin intermediate certificates, but this again starts to
> stray into areas that the IETF historically leaves as a matter of local
> policy.

Requiring pinning OPTIONS would enable this making of local
policy choices among such options. No options, no choices.
Users like having options and choices.



Opportunistic Security: Some Protection Most of the Time
https://tools.ietf.org/html/rfc7435.html