pkg audit false negatives (was: Perl upgrade - 5.20.x vulnerable)

[Mark Felder]

On Tue, Aug 16, 2016, at 11:41, Roger Marquis wrote:
> 
> There's also an issue with older versions (perl 5.1*) no longer showing
> up in the vuln.xml at all.  I've seen perl, php and other critical
> network components still in use because the site depended on 'pkg audit'
> but did not know that expired OR deprecated ports are not audited.
> Apparently this is intentional and by policy.  IMO it is a serious flaw
> in pkg audit's design.
> 

This is hard to keep track of, but I also agree it is a problem we need
to be more conscious of. Unfortunately it adds burden to our
ports-secteam because often times upstream will EoL software and not
state if older versions are actually vulnerable. Short of a PoC or
auditing the source code of something that is EoL / removed from the
ports tree there isn't much we can do but guess. It's entirely possible
something EoL is not actually vulnerable to newer CVEs depending on if
it's in new/refactored code or shared code. Even then it's possible that
the way the code is used in the EoL version it's still not vulnerable.

> A better policy would include expired AND deprecated ports in the output
> of 'pkg audit' for at least a year after they are removed from the ports
> and/or pkg trees.  If a port had no known vulnerability when removed it
> should at least indicate 'no longer audited' in place of 'vulnerable'.
> 

Yes, a solution to create vuxml entries for EoL/removed ports with a
simple statement that they're EoL and could be vulnerable is a sane
approach. This way we aren't incorrectly listing software as vulnerable
to CVEs we haven't validated. I welcome this, but we will need help from
users and the community to let us know when this happens as we aren't
always aware of the EoL schedules of software in the ports tree.


> This is, IMO, one of 3 remaining weaknesses in the otherwise excellent
> freebsd audit framework.  The other two issues have to do with base not
> being packaged (so not really being 'audit'able) and the 'general rule'
> announced on Aug 10 that 'the FreeBSD Security Officer does not announce
> vulnerabilities for which there is no released patch'.  This is
> particularly problematic as there are usually mitigations that do not
> require patches.
> 

I already solved your #2 problem:
https://blog.feld.me/posts/2016/08/monitoring-freebsd-base-system-vulnerabilities-with-pkg-audit/

#3 is being reviewed by secteam/core, so I think we're well on our way
to solving these concerns.


-- 
  Mark Felder
  ports-secteam member
  feld at FreeBSD.org