or similar? That might be neat. But, it would create two different meanings for this attribute: (a) editorial description, i.e. to disambiguate multiple similar references to CVE names, source code, whatever; (b) the referenced item's description, e.g. <title> for HTML documents, Subject: for email. I'm not sure I like that. As you seem to point out, (b) could be largely automated, so I'm not sure it is worth storing it in the document. Then when rendering a document, how will a reader know that what she sees is (a) or (b)? One more thing: I suggested `description' as the attribute name, but I have second thoughts. I believe it might be better to pick something that is not also an element name, and perhaps `description' is a little too much typing. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Mon Aug 23 07:18:25 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Mon Aug 23 07:19:33 2004 Subject: making <description> optional In-Reply-To: <272AEBD2-F486-11D8-8CAA-00039312D914@fillmore-labs.com> References: <20040822213232.GE17478@madman.celabo.org> <272AEBD2-F486-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <20040823141803.GN27355@madman.celabo.org> On Sun, Aug 22, 2004 at 11:56:42PM +0200, Oliver Eikemeier wrote: > Jacques A. Vidrine wrote: > 60 (in words: sixty) entries in portaudit have the description `Please > contact the FreeBSD Security Team for more information'. There are > references, so when you care to add a quote, feel free, in fact this > might be a job for the security team. You can frown on them as often as > you like, the question is whether you just want to have an optional > <description> entry as an easy to spot sign that an editor is needed, or > if you prefer to search for and similar constructs. I'm not sure what you are talking about. I don't see any such entries in VuXML ... but you said `portaudit' so maybe you are talking about your personal database? > >However, I must admit that I have some doubt the value of the > ><discovery> date in any case. What I'd really like to hear are some > >arguments for keeping it or getting rid of it! I think it is useful > >information of itself to many reading VuXML content, and that combined > >with <entry> it provides a good metric about our response time. But I > >could be overestimating the value of it, and if it somehow puts people > >off to need to provide this information, then maybe it loses. > > Oviously we have a different opinion what is useful here. I expect most > users to be simple consumers, not security researchers. They need > information about the serverity of a vulnerability, and maybe > remote/local exploitability, whoever cares about the discovery date > could check the references. Often I find the discovery date > entertaining, but not useful. So I'll take that as a vote for not keeping it (<discovery>). Such a change (dropping required content) would need to take place in a `major' update e.g. VuXML 2.0. We'll revisit it then, maybe someone else will add some opinions before then. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Mon Aug 23 07:22:48 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Mon Aug 23 07:23:36 2004 Subject: database tables for VuXML In-Reply-To: <EC0A88B8-F487-11D8-8CAA-00039312D914@fillmore-labs.com> References: <20040822213304.GF17478@madman.celabo.org> <EC0A88B8-F487-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <20040823142227.GO27355@madman.celabo.org> On Mon, Aug 23, 2004 at 12:09:22AM +0200, Oliver Eikemeier wrote: > >The current version of VuXML does not allow these to be null, and I it > >is probably good to reflect those constraints in the database. > > Which means you possibly have to change the database description when > upgrading. I guess you have some point. You must like Tcl :-) No required elements will be dropped in any VuXML 1.x, but if/when there is a VuXML 2.0 there will likely be some changes to be made in applications (duh). > <architecture> should be stuff like i386? I'm not sure that there are > many applications for that. Usually the description is sufficient for > those purposes. Or did you have something different in mind? Yeah, stuff like `i386'. I don't see it (or <category>) as useful for the FreeBSD Project today, either: these are for use by other projects. In the case of <architecture>, I'm not yet sure that it will make it through pre-1.2. I'm still working with the project to understand their needs. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Mon Aug 23 07:29:31 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Mon Aug 23 07:29:33 2004 Subject: database tables for VuXML In-Reply-To: <4129070D.28233.7C2E4023@localhost> References: <41279E59.13631.76AD2AB9@localhost> <4129070D.28233.7C2E4023@localhost> Message-ID: <20040823142910.GP27355@madman.celabo.org> On Sun, Aug 22, 2004 at 08:50:21PM -0400, Dan Langille wrote: > A thought which came to me overnight: does anyone have suggestions as > to how existing FreshPorts pages should be modified to include VuXML > messages? Please feel free to take an existing FreshPorts page, do a > file-save-as, and alter the HTML to suit. Then give us the URL where > we can find your version. > > My idea: Take http://www.freshports.org/sysutils/bacula/ as an > example. I think adding a special icon to the left hand colum under > Commit History. Alternatives welcome. First something off-topic: Dan, how do you generate the package names that are seen in that left-hand column? I've often thought it would be handy to see the "history" of package names, and that seems to be exactly what you have there. Personally, I think a special icon with a link to vuxml.freebsd.org/<id>.html (or internally rendered page) and an ALT="<topic>" would be just swell! It might also be handy to have a bright warning in the top, main section if the HEAD is currently affected... similar to how you handle FORBIDDEN, BROKEN, IGNORE, etc ports. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From dan at langille.org Mon Aug 23 07:54:03 2004 From: dan at langille.org (Dan Langille) Date: Mon Aug 23 07:54:06 2004 Subject: database tables for VuXML In-Reply-To: <20040823142910.GP27355@madman.celabo.org> References: <4129070D.28233.7C2E4023@localhost> Message-ID: <4129CCC9.11697.7F32B4B4@localhost> On 23 Aug 2004 at 9:29, Jacques A. Vidrine wrote: > On Sun, Aug 22, 2004 at 08:50:21PM -0400, Dan Langille wrote: > > A thought which came to me overnight: does anyone have suggestions as > > to how existing FreshPorts pages should be modified to include VuXML > > messages? Please feel free to take an existing FreshPorts page, do a > > file-save-as, and alter the HTML to suit. Then give us the URL where > > we can find your version. > > > > My idea: Take http://www.freshports.org/sysutils/bacula/ as an > > example. I think adding a special icon to the left hand colum under > > Commit History. Alternatives welcome. > > First something off-topic: Dan, how do you generate the package names > that are seen in that left-hand column? I've often thought it would > be handy to see the "history" of package names, and that seems to be > exactly what you have there. The package names? I think you are referring to PORTVERSION and PORTREVISION. For example, 1.34.6 is PORTVERSION, and 2 is the PORTREVISION. This information is generated from the following: make -V PORTREVISION -V PORTVERSION Is that what you are referring to? > Personally, I think a special icon with a link to > vuxml.freebsd.org/<id>.html (or internally rendered page) and an > ALT="<topic>" would be just swell! It might also be handy to have > a bright warning in the top, main section if the HEAD is currently > affected... similar to how you handle FORBIDDEN, BROKEN, IGNORE, etc > ports. I welcome example icons! I was thinking to link people to the internal FreshPorts pages, and from there to the official vuxml.freebsd.org pages if they want additional information. -- Dan Langille : http://www.langille.org/ From eikemeier at fillmore-labs.com Mon Aug 23 08:03:22 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Mon Aug 23 08:03:25 2004 Subject: portaudit wishlist In-Reply-To: <20040823140949.GL27355@madman.celabo.org> Message-ID: <941610FA-F515-11D8-8CAA-00039312D914@fillmore-labs.com> Jacques A. Vidrine wrote: > On Sun, Aug 22, 2004 at 11:39:47PM +0200, Oliver Eikemeier wrote: >> It should be trivial to deal with the <alternate> tag in XSLT, the same >> should be possible with <optional>, and for entering them into >> databases >> you have to render the stuff anyway. > > One of your concerns was bloat, and I think using > <alternate>/<optional> like this would be just as much bloat. The > only redeeming quality of the csh-style syntax was brevity (but as I > said, I don't think that's a sufficient reason to use it). Bloat as in `linear search'. It's easy to render <alternate> to csh-style braces. >> Readability of the XML code is a >> non-issue, since it is designed to be machine-readable, not >> human-readable. > > No, that's really not correct. XML is not necessarily user-readable, > but it may certainly be human-readable. Or do you think that all our > doc committers that work on the web site or hand book and so forth > work in "write-only" mode? (rhetorical question) Users should have tools. I may write a `make entry' when we settled on a format suitable for our purposes. >> portaudit is designed to be lightweight and work without >> a database, so it does linear searches on all systems. I might change >> that, but that's the way things currently are, so what's the problem >> here? > > I don't know, what *is* the problem here? What problem are you trying > to solve with this new syntax? It doesn't seem to be readability > (the csh syntax and <alternate>/<optional> elements are both worse > for readability). You mentioned space, but I don't believe that's an > issue, and you seem to not either. I don't care about the size of the vuxml file, but about the rendered version. I'm fine with openldap-{client,server}, but you had concerns that this isn't unsable for xml tools, so I suggested an alternative in xml. >> [...] >>> So basically I think we should introduce this in VuXML 1.2. I'd like >>> to hear some comments from others about it, especially from the point >>> of view of the user reading content generated from VuXML. >> >> Uhm, you don't need vuxml 1.2 for that, this is perfectly legal in >> vuxml >> since 1.0. > > No, it is not. You can of course have a *valid* VuXML document with > `1.*' or even `@#$@% mary had a little lamb' as content for the <range> > child elements, but that doesn't make it a legal or proper VuXML > document. The DTD is only part of the specification. The other part is > described in the DTD comments. I can only find `The version ranges given must not overlap.' Other pointers? I still think this is a proper vuxml range specification. How a platform defines (and sorts) version numbers isn't handled in the vuxml specification. >>>> [...] >> I can't really follow your rationale here. You claim that because it >> can't be done perfectly, it shouldn't be done at all. > > I don't think you read and understood what I wrote. Let me try to > be clearer: the security team does not currently assign severity to > security advisories or VuXML entries, and has no plans to do so. > Rather, we believe that all of these security notices should be acted > upon. Theory and practice. There are seven unfixed vulnerabilities on freefall: ssh <login>@freefall.freebsd.org pkg_info | cut -f 1 -d ' ' | xargs portaudit It is useful to have the ability to mark vulnerabilities as `minor', like for example CAN-2004-0777. I did make an entry because we can't currently rank them. And insisting that this is not useful for `the security team' supports the theory that portaudit and vuxml need different databases. >> I would find it >> incredibly useful to get some categorization into `dangerous', >> `important' and `minor', even when it's wrong sometimes. As discussed >> before: You usually have a pretty good idea whether a vulnerability is >> severe or not, you just don't want to tell anybody. >> >> I consider this valuable to give users a chance to customize the >> notification scheme of portaudit, and why shouldn't this information >> find its place in the vuxml database? Make this tag optional, so you >> don't have to fill in anything when you on't feel like it. > > I think that any severity rating system should be seperate from the > VuXML documentation to allow for multiple such systems and policies, > particularly as practices in this area evolve. I don't think so. Either we have a common view on vulnerabilities, or we don't. I guess we should make vuxml the official database format, portaudit the official tool and be set. What are `multiple systems and policies' in this context? a severity rating is advisory only, and could be used or ignored however the system thinks is fit. What's the point when FreshPorts has an other severity level that portaudit? Or should it use the portaudit severity ranking? I can't understand your problems here. >>>> - add a classification into remote/local exploitable >>> >>> On the one hand, I feel this should be handled in the narrative >>> description and that it is otherwise just another facet of the >>> severity rating. On the other hand, I can imagine someone deciding >>> that it was OK e.g. to install any ports that did not have a >>> *remotely* exploitable vulnerability. >>> >>> My instinct tells me this too should be adjunct, at least >>> for now. Why would we include remotely/locally, but not >>> authenticated/unauthenticated, application privileges/system >>> privileges, user privileges/superuser privileges, or other such >>> aspects? If you have a server with a vulnerability that lets someone >>> who has a local account to get some other access, would you record >>> that as local or remote? >> >> umh, local of course? > > Funny you should say `of course'. I would have classified it as > remote. I picked this example from some correspondence involving > several colleagues, and guess what: the answers were mixed. Which is > `of course' my point. You have border cases in every system. Just denying to add useful features because some entries could be erroneous isn't helpful. Users could use this information, even when it's not 100% correct. Even the <range> entries in vuxml are not 100% correct, so what? >>> Yes, I think it is misleading to apply such tags which a user might >>> take as an absolute judgement when in fact they just need to read the >>> description. >> >> Not everyone has the time to review every description. Besides, the >> description might be as wrong or misleading as the tags mentioned. If >> you say "users have to understand the system fully or they shouldn't >> run >> the software" you basically state "FreeBSD is only for experts". I'm >> just trying to make some often asked questions machine readable. For >> example when I run portaudit on a server with no users, I might decide >> to care for local exploitable vulnerabilities only ever friday, while I >> have to handle remote exploitable vulnerabilities immediately. This >> system is not perfect, but usable. You give users basically no way to >> filter the information, which would be a valuable feature. One one hand >> you state users have to be knowledgeable to run a system, one the other >> you claim they might take tags `as an absolute judgement'. In this case >> reading the (possibly wrong) description might not improve anything. > > Your ``reasoning'' makes me dizzy. > > Look Oliver, knock yourself out: come up with your own severity rating > scheme and implement it. Stop bugging the security team to do it, > I've already explained that we will not at this time. Ok, back to my own database specification then? We have just a different view on our user base, and I think you fail to address some needs. Not everybody is a purist here, some `just want to have the job done', even when this means to err once or twice. >>>> - add a `fixed' field that lists a version where the vulnerability is >>>> fixed. This could be used for a recommendation message, like "upgrade >>>> to >>>> version xxx" or "no upgrade is available, please deinstall the port >>>> or >>>> proceed with caution". >>>> This could also realized as an alternate <lt> tag. >>> >>> I guess I don't understand this request. That is the purpose of the >>> <affected> element and children. >> >> There is no information whether there is an update available (and since >> which date the vulnerability is fixed), or if I simply have to >> deinstall >> the software and use a different product. > > Maybe you can describe this a bit more fully? (BTW, Can you make a > separate thread for each of your proposals?) I felt that the affected > ranges were sufficient, but if there is a better way let's hear it. There is no way to deduce from the VuXML document alone whether a fixed version exists (like in port<1.5: is there a port-1.5 in the tree or not?). So basically the user has to do the research whether s/he could just upgrade or has to deinstall the port completely (or live with the vulnerability). > [... snip: adding `description's to each reference ...] >> xfdb does it that way, and I like it. It's especially useful for >> mailing >> list posts, to see whether they contain an advisory or an exploit, for >> entries that cover multiple vulnerabilities (to distinguish the CVE >> references) and to filter out those `Updated packages fix multiple >> vulnerabilities' references for other operating systems. >> >>> The whole point of VuXML is to give enough information but not too >>> much :-) ``Too much'' is anything that is not likely to be supplied >>> in >>> the vast majority of cases. >> >> This could be automated by the tool that makes entries, or even by a >> tool that adds missing descriptions, so it is likely to be supplied. > > Well, I'm not sure I'm convinced. But let's try an example. > > <references> > <cvename description="BMP heap overflow">CAN-2004-0691</cvename> > <cvename description="XPM crash">CAN-2004-0692</cvename> > <cvename description="GIF crash">CAN-2004-0693</cvename> > > <url>http://www.trolltech.com/developer/changes/changes-3.3.3.html</url> > <url>http://scary.beasts.org/security/CESA-2004-004.txt</url> > </references> > > Is this the kind of thing you mean? More or less. Since these CVE references have no title yet, they would be shown as simple URLs. But you could have <url description="Trolltech - Changes 3.3.3"> http://www.trolltech.com/developer/changes/changes-3.3.3.html</url>. > I wonder if it will really be > used much *shrug* (by authors or users). I'd like to know what you > mean by `a tool that adds missing descriptions'. Are you thinking > of following the references and snarfing the <title> or similar? Jup, with a possibility to edit or deny it when it is too meaningless. > That might be neat. But, it would create two different meanings > for this attribute: (a) editorial description, i.e. to disambiguate > multiple similar references to CVE names, source code, whatever; (b) > the referenced item's description, e.g. <title> for HTML documents, > Subject: for email. I'm not sure I like that. As you seem to point > out, (b) could be largely automated, so I'm not sure it is worth > storing it in the document. Then when rendering a document, how will > a reader know that what she sees is (a) or (b)? Probably (b) or the url is fine here. (a) is too much work in the general case. > One more thing: I suggested `description' as the attribute name, but I > have second thoughts. I believe it might be better to pick something > that is not also an element name, and perhaps `description' is a > little too much typing. Whatever you like. Since entries should be done by tools, I don't care how much typing is necessary. -Oliver From eikemeier at fillmore-labs.com Mon Aug 23 08:10:08 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Mon Aug 23 08:10:10 2004 Subject: making <description> optional In-Reply-To: <20040823141803.GN27355@madman.celabo.org> Message-ID: <86914F26-F516-11D8-8CAA-00039312D914@fillmore-labs.com> Jacques A. Vidrine wrote: > On Sun, Aug 22, 2004 at 11:56:42PM +0200, Oliver Eikemeier wrote: >> Jacques A. Vidrine wrote: >> 60 (in words: sixty) entries in portaudit have the description `Please >> contact the FreeBSD Security Team for more information'. There are >> references, so when you care to add a quote, feel free, in fact this >> might be a job for the security team. You can frown on them as often as >> you like, the question is whether you just want to have an optional >> <description> entry as an easy to spot sign that an editor is needed, >> or >> if you prefer to search for and similar constructs. > > I'm not sure what you are talking about. I don't see any such entries > in VuXML ... but you said `portaudit' so maybe you are talking about > your personal database? I have a supplementary databases that are as much `personal' as vuxml is. The portaudit text database has been announced and documented as mentioned in a previous discussion. Anyway, I think making the entry optional would be the best solution, but if you prefer a placeholder, we can keep `Please contact the FreeBSD Security Team for more information'. >>> However, I must admit that I have some doubt the value of the >>> <discovery> date in any case. What I'd really like to hear are some >>> arguments for keeping it or getting rid of it! I think it is useful >>> information of itself to many reading VuXML content, and that combined >>> with <entry> it provides a good metric about our response time. But I >>> could be overestimating the value of it, and if it somehow puts people >>> off to need to provide this information, then maybe it loses. >> >> Oviously we have a different opinion what is useful here. I expect most >> users to be simple consumers, not security researchers. They need >> information about the serverity of a vulnerability, and maybe >> remote/local exploitability, whoever cares about the discovery date >> could check the references. Often I find the discovery date >> entertaining, but not useful. > > So I'll take that as a vote for not keeping it (<discovery>). Such > a change (dropping required content) would need to take place in a > `major' update e.g. VuXML 2.0. We'll revisit it then, maybe someone > else will add some opinions before then. Whatever you like. Simply using dummy values is fine with me. -Oliver From eikemeier at fillmore-labs.com Mon Aug 23 08:13:11 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Mon Aug 23 08:23:05 2004 Subject: database tables for VuXML In-Reply-To: <20040823142227.GO27355@madman.celabo.org> Message-ID: <F42627C5-F516-11D8-8CAA-00039312D914@fillmore-labs.com> Jacques A. Vidrinew wrote: > On Mon, Aug 23, 2004 at 12:09:22AM +0200, Oliver Eikemeier wrote: [...] >> <architecture> should be stuff like i386? I'm not sure that there are >> many applications for that. Usually the description is sufficient for >> those purposes. Or did you have something different in mind? > > Yeah, stuff like `i386'. I don't see it (or <category>) as useful > for the FreeBSD Project today, either: these are for use by other > projects. In the case of <architecture>, I'm not yet sure that it > will make it through pre-1.2. I'm still working with the project to > understand their needs. FWIIW, I would find a <system> tag that enables the sharing of entries across operating systems far more useful. -Oliver From eikemeier at fillmore-labs.com Mon Aug 23 08:21:18 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Mon Aug 23 08:40:05 2004 Subject: portaudit wishlist In-Reply-To: <941610FA-F515-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <15E125E6-F518-11D8-8CAA-00039312D914@fillmore-labs.com> [...] >>>> Yes, I think it is misleading to apply such tags which a user might >>>> take as an absolute judgement when in fact they just need to read the >>>> description. >>> >>> Not everyone has the time to review every description. Besides, the >>> description might be as wrong or misleading as the tags mentioned. If >>> you say "users have to understand the system fully or they shouldn't >>> run >>> the software" you basically state "FreeBSD is only for experts". I'm >>> just trying to make some often asked questions machine readable. For >>> example when I run portaudit on a server with no users, I might decide >>> to care for local exploitable vulnerabilities only ever friday, >>> while I >>> have to handle remote exploitable vulnerabilities immediately. This >>> system is not perfect, but usable. You give users basically no way to >>> filter the information, which would be a valuable feature. One one >>> hand >>> you state users have to be knowledgeable to run a system, one the >>> other >>> you claim they might take tags `as an absolute judgement'. In this >>> case >>> reading the (possibly wrong) description might not improve anything. >> >> Your ``reasoning'' makes me dizzy. >> >> Look Oliver, knock yourself out: come up with your own severity rating >> scheme and implement it. Stop bugging the security team to do it, >> I've already explained that we will not at this time. > > Ok, back to my own database specification then? We have just a > different view on our user base, and I think you fail to address some > needs. Not everybody is a purist here, some `just want to have the job > done', even when this means to err once or twice. Thinking a little about it, I believe this should be discussed in a place where portaudit users are present, either ports@ or security. freebsd-vuxml@ has too few subscribers to get an useful picture of what features desired by users are. -Oliver From eikemeier at fillmore-labs.com Mon Aug 23 10:59:06 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Mon Aug 23 11:00:08 2004 Subject: database tables for VuXML In-Reply-To: <4129CCC9.11697.7F32B4B4@localhost> Message-ID: <218B20FB-F52E-11D8-8CAA-00039312D914@fillmore-labs.com> Dan Langille wrote: >> Personally, I think a special icon with a link to >> vuxml.freebsd.org/<id>.html (or internally rendered page) and an >> ALT="<topic>" would be just swell! It might also be handy to have >> a bright warning in the top, main section if the HEAD is currently >> affected... similar to how you handle FORBIDDEN, BROKEN, IGNORE, etc >> ports. > > I welcome example icons! > > I was thinking to link people to the internal FreshPorts pages, and > from there to the official vuxml.freebsd.org pages if they want > additional information. When you want to render your own pages: portaudit-db does this. -Oliver From dan at langille.org Tue Aug 24 14:38:57 2004 From: dan at langille.org (Dan Langille) Date: Tue Aug 24 14:50:39 2004 Subject: database tables for VuXML In-Reply-To: <20040822192915.GA17478@madman.celabo.org> References: <41279E59.13631.76AD2AB9@localhost> Message-ID: <412B7D29.31547.1472F80@localhost> On 22 Aug 2004 at 14:29, Jacques A. Vidrine wrote: > [Added freebsd-vuxml@] > > On Sat, Aug 21, 2004 at 07:11:21PM -0400, Dan Langille wrote: > > Hi folks, > > > > I'm in the early stage of incorporating VuXML information into > > FreshPorts. > > Alright! That'll be cool. > > > I have a preliminary database table design which appears > > below. If you think I've missed any essential information out of > > this design, please let me know. > > > > First: has anyone already worked on creating database tables for the > > VuXML data? > > I have an old prototype application which uses SQLite. FWIW, here is > the schema I used then. (the formatting is a little funny because this > was embedded in some other code) > > CREATE TABLE vuxml ( > vid VARCHAR PRIMARY KEY UNIQUE > , topic VARCHAR > , description VARCHAR > , discovery DATE > , entry DATE > , modified DATE > -- computed max(entry, modified) for convenience in > -- other expressions > , last DATE > ); This is pretty much identical to what I have. > > CREATE TABLE refs ( > vid VARCHAR > , type VARCHAR > , text VARCHAR > ); Ahh, yes, that repeating group. I'm not storing that. I should be > CREATE TABLE affected ( > vid VARCHAR > , type VARCHAR > ); What is this type field? I didn't follow what the affected table contains. > CREATE TABLE names ( > affected INTEGER > , name VARCHAR > ); >From your example below, the names table would have entries for mutt and ja-mutt. I think I'll do that as well. See the end of my message for how I plan to relate VuXML entries to specific ports. > CREATE TABLE ranges ( > affected INTEGER > , lowop VARCHAR > , low VARCHAR > , glow0 INTEGER > , glow1 INTEGER > , glow2 INTEGER > , glow3 INTEGER > , highop VARCHAR > , high VARCHAR > , ghigh0 INTEGER > , ghigh1 INTEGER > , ghigh2 INTEGER > , ghigh3 INTEGER > ); > > The `ranges' table perhaps looks erm interesting. In this case, > I encoded each package version number into four 32-bit integers. > This allowed one to lookup an affected package completely in SQL. > However, the encoding I used may now be incorrect in some ways, since > pkg_install et al were recently changed to treat version numbers > differently. FreshPorts stores PORTVERSION and PORTREVISION as separate text fields. I take it that the ranges equations should compare only PORTVERSION and ignore PORTREVISION. > Also, SQLite could not really optimize the complex SQL > statement that resulted. So I think it is interesting for further > study, but I doubt I'd do it that way again unless I had a very > particular need to use pure SQL or the database was huge. SQLite is a great little database. I like what I've seen of it (mostly via my work on Bacula). I'll be using PostgreSQL for this. > > > create table vuxml > > ( > > id serial not null, > > -- internal FreshPorts ID > > vid text not null, > > -- the vuln vid. e.g. c4b025bb-f05d-11d8-9837-000c41e2cdad > > topic text not null, > > description text not null, > > date_discovery date , > > date_entry date , > > date_modified date , > > status date not null, > > primary key (id) > > ); > > > > create table vuxml_port_xref > > ( > > id serial not null, > > vuxml_id integer not null, > > port_id integer not null, > > primary key (id) > > ); > > > > create table vuxml_port_xref_range > > ( > > id serial not null, > > vuxml_port_xref_id text , > > range_operator_start text , > > range_operator_end text , > > range_version_start text , > > range_version_end text , > > primary key (id) > > ); > > > > If you consider the output from: vxquery -t vuxml ~/VuXML/vuln.xml > > tnftpd, the entry in vuxml_port_xref_range for tnftpd might be: > > > > (1, 'lt', 20040810, NULL, NULL) > > Seem reasonable. For what its worth, storing the range operators as > `>', `=>', etc lets you get something human readable as the output of a > SELECT, e.g. > > SELECT range_version_start, range_operator_start, port_name, > range_operator_end, range_version_end FROM ... WHERE ...; > > ('20040810', '<', 'tnftpd', '', '') > > I found that handy for manual inspection of the database. Whatever I store, need to be able to run queries upon it. I was thinking of keeping the existing values as that might be easier with perl. Not sure yetp. > You may notice from my example that I used separate `name', `ranges', > and `affected' tables. I found this easier when filling the database, > since the <package>/<system> elements are kind of a cross-product > operator. For example, when processing something like > > <package> > <name>mutt</name> > <name>ja-mutt</name> > <range><ge>1.4</ge><lt>1.4.2</lt></range> > </package> This is what I do with my vuxml_port_xref table. That table relates a vid to a port. > I first created an `affected' entry to contain the other references. > Then, as I hit each `name' I just associated it with the `affected' > entry, and as I hit each `range' I did the same. (Otherwise one > needs to construct a few lists, and then make all entries once > the </package> end tag is seen.) I think this is also easier to > modify if we get new children of <package>, e.g. the <category> and > <architecture> tags that are currently in the VuXML pre-1.2 DTD. My plan is to populate empty the vuxml_* tables each time there is a commit to the data file. This keeps the vuxml system totally separate from FreshPorts. Each row under Commit History (e.g. http://beta.freshports.org/sysutils/bacula/) relates to a row from the commit_log_ports table. That looks something like this: create table commit_log_ports ( commit_log_id integer not null, port_id integer not null, needs_refresh smallint not null, port_version text , port_revision text , primary key (commit_log_id, port_id) ); I'll probably create another table commit_log_ports_vuxml: create table commit_log_ports ( commit_log_id integer not null, port_id integer not null, vuxml_id integer not null , primary key (commit_log_id, port_id) ); Looking at the current data, there's about 140 affected ports, but I haven't broken that with respect to ranges, which is what the above will do. I'll either do that, or add the vuxml_id column to the commit_log_ports table, but I'd rather keep it separate. Time will tell. -- Dan Langille : http://www.langille.org/ From eikemeier at fillmore-labs.com Tue Aug 24 15:35:30 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Tue Aug 24 15:51:18 2004 Subject: database tables for VuXML In-Reply-To: <412B7D29.31547.1472F80@localhost> Message-ID: <EA56FB26-F61D-11D8-91E7-00039312D914@fillmore-labs.com> Dan Langille wrote: > FreshPorts stores PORTVERSION and PORTREVISION as separate text > fields. I take it that the ranges equations should compare only > PORTVERSION and ignore PORTREVISION. Please not. pkg_version -t is the reference here, when you get different results something is wrong (and ignoring PORTREVISION will get you different results). Also keep in mind that PORTEPOCH is an essential component. -Oliver From dan at langille.org Tue Aug 24 17:53:29 2004 From: dan at langille.org (Dan Langille) Date: Tue Aug 24 17:53:31 2004 Subject: database tables for VuXML In-Reply-To: <EA56FB26-F61D-11D8-91E7-00039312D914@fillmore-labs.com> References: <412B7D29.31547.1472F80@localhost> Message-ID: <412BAAC8.10821.1F94E6D@localhost> On 25 Aug 2004 at 0:35, Oliver Eikemeier wrote: > Dan Langille wrote: > > > FreshPorts stores PORTVERSION and PORTREVISION as separate text > > fields. I take it that the ranges equations should compare only > > PORTVERSION and ignore PORTREVISION. > > Please not. pkg_version -t is the reference here, when you get different > results something is wrong I'm confused. FreshPorts obtains information primarily via "make - V". pkg_version works on installed ports only. FreshPorts is not gong to install a port to find out what version we have. Are we talking about the same thing? > (and ignoring PORTREVISION will get you different results). > Also keep in mind that PORTEPOCH is an essential component. That's news to me. Thanks. I see it is used to create PKGNAME. Looking at range data of vuxml, I see now that PORTEPOCH is used. OK. Thanks for the heads up. -- Dan Langille : http://www.langille.org/ From eikemeier at fillmore-labs.com Wed Aug 25 00:41:04 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Wed Aug 25 00:41:06 2004 Subject: database tables for VuXML In-Reply-To: <412BAAC8.10821.1F94E6D@localhost> Message-ID: <1F0717EB-F66A-11D8-91E7-00039312D914@fillmore-labs.com> Dan Langille wrote: > On 25 Aug 2004 at 0:35, Oliver Eikemeier wrote: > >> Dan Langille wrote: >> >>> FreshPorts stores PORTVERSION and PORTREVISION as separate text >>> fields. I take it that the ranges equations should compare only >>> PORTVERSION and ignore PORTREVISION. >> >> Please not. pkg_version -t is the reference here, when you get >> different >> results something is wrong > > I'm confused. FreshPorts obtains information primarily via "make - > V". pkg_version works on installed ports only. FreshPorts is not > gong to install a port to find out what version we have. Are we > talking about the same thing? <http://www.freebsd.org/cgi/man.cgi?query=pkg_version> -Oliver From nectar at FreeBSD.org Wed Aug 25 09:33:56 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 09:39:35 2004 Subject: making <description> optional In-Reply-To: <86914F26-F516-11D8-8CAA-00039312D914@fillmore-labs.com> References: <86914F26-F516-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <825857C8-F6B4-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 23, 2004, at 10:10 AM, Oliver Eikemeier wrote: > I have a supplementary databases that are as much `personal' as vuxml > is. The portaudit text database has been announced and documented as > mentioned in a previous discussion. I'm sorry, I couldn't find that text (`Please contact [...]') anywhere in portaudit.txt or related files. Now I realize that is generated in the pages at ~eik/portaudit. I think we're on the same page now. > Anyway, I think making the entry optional would be the best solution, > but if you prefer a placeholder, we can keep `Please contact the > FreeBSD Security Team for more information'. I prefer an actual description, but OK :-) > Whatever you like. Simply using dummy values is fine with me. For the sake of harmony, let's agree that setting <discovery>unknown</discovery> is acceptable for now. Gimme a few days to check that stuff doesn't break when processing said entry. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Wed Aug 25 11:05:49 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 11:16:03 2004 Subject: portaudit wishlist In-Reply-To: <941610FA-F515-11D8-8CAA-00039312D914@fillmore-labs.com> References: <941610FA-F515-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <56E15C66-F6C1-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 23, 2004, at 10:03 AM, Oliver Eikemeier wrote: > Jacques A. Vidrine wrote: > >> On Sun, Aug 22, 2004 at 11:39:47PM +0200, Oliver Eikemeier wrote: >>> It should be trivial to deal with the <alternate> tag in XSLT, the >>> same >>> should be possible with <optional>, and for entering them into >>> databases >>> you have to render the stuff anyway. >> >> One of your concerns was bloat, and I think using >> <alternate>/<optional> like this would be just as much bloat. The >> only redeeming quality of the csh-style syntax was brevity (but as I >> said, I don't think that's a sufficient reason to use it). > > Bloat as in `linear search'. It's easy to render <alternate> to > csh-style braces. OK, now I understand what you are talking about. I'm sorry, I think you tried to tell me several times already, but I didn't understand. Today there are about 245 <name> elements, and I'd guess that maybe 40 could be replaced with a syntax such as you suggest. However, I don't think a 17% reduction in entries to speed up a linear search is worth the uglification. The first rule of optimization is, well, "don't do it", but one of the other top rules is to optimize by selecting the right algorithm, rather than attempting to speed up an inherently slower one. [...] >>> Uhm, you don't need vuxml 1.2 for that, this is perfectly legal in >>> vuxml >>> since 1.0. >> >> No, it is not. You can of course have a *valid* VuXML document with >> `1.*' or even `@#$@% mary had a little lamb' as content for the >> <range> >> child elements, but that doesn't make it a legal or proper VuXML >> document. The DTD is only part of the specification. The other part >> is >> described in the DTD comments. > > I can only find `The version ranges given must not overlap.' Other > pointers? I still think this is a proper vuxml range specification. > How a platform defines (and sorts) version numbers isn't handled in > the vuxml specification. You are right, this could only be specified per platform. I guess I'm confusing it with VuXML point releases, because changing the syntax also means tools have to adapt. For example, vxquery uses code from pkg_install which does not "understand" `1.*'. This just meants that vxquery needs to be updated. portaudit already groks it, and I don't think it matters for vuxml.org's rendered pages. So, I'll update vxquery and then it should be OK to use. Tangent --- maybe the version comparison code should be available in a library. [...] > It is useful to have the ability to mark vulnerabilities as `minor', > like for example CAN-2004-0777. Hmm, I think that was a mistake. I feel quite certain that a significant number of installations actually have DEBUG_LOGIN set. [...] >> I think that any severity rating system should be seperate from the >> VuXML documentation to allow for multiple such systems and policies, >> particularly as practices in this area evolve. > > I don't think so. Either we have a common view on vulnerabilities, or > we don't. Well, we disagree, Oliver. > I guess we should make vuxml the official database format, portaudit > the official tool and be set. What are `multiple systems and policies' > in this context? A FreeBSD Project mantra is `tools, not policy'. `multiple systems and policies' means multiple tools. > a severity rating is advisory only, and could be used or ignored > however the system thinks is fit. What's the point when FreshPorts has > an other severity level that portaudit? Perhaps FreshPorts (or whatever) has a rating system that better fits what some users need to accomplish. > Or should it use the portaudit severity ranking? I can't understand > your problems here. Conversely, I do not understand yours. :-( >>>>> - add a classification into remote/local exploitable >>>> >>>> On the one hand, I feel this should be handled in the narrative >>>> description and that it is otherwise just another facet of the >>>> severity rating. On the other hand, I can imagine someone deciding >>>> that it was OK e.g. to install any ports that did not have a >>>> *remotely* exploitable vulnerability. >>>> >>>> My instinct tells me this too should be adjunct, at least >>>> for now. Why would we include remotely/locally, but not >>>> authenticated/unauthenticated, application privileges/system >>>> privileges, user privileges/superuser privileges, or other such >>>> aspects? If you have a server with a vulnerability that lets >>>> someone >>>> who has a local account to get some other access, would you record >>>> that as local or remote? >>> >>> umh, local of course? >> >> Funny you should say `of course'. I would have classified it as >> remote. I picked this example from some correspondence involving >> several colleagues, and guess what: the answers were mixed. Which is >> `of course' my point. > > You have border cases in every system. I think it is not a border case, but fundamental. The fact that you think it is a border case and I think it is fundamental is indicative of our lack of agreement in this area. > Just denying to add useful features because some entries could be > erroneous isn't helpful. You think it is useful, I don't. I almost feel it is harmful, in fact :-) You have the freedom to implement those features you think are useful, and I have the freedom to ignore your insistence that I need to do something I consider not useful. > Users could use this information, even when it's not 100% correct. Sure. There is always the possibility of flaws. I suggest treating every issue as severe unless one knows better (for herself). This is called "fail closed", and I think that it is appropriate for a project such as ours. > Even the <range> entries in vuxml are not 100% correct, so what? Such entries should be considered bugs. Or was this a rhetorical question? [...] > Ok, back to my own database specification then? We have just a > different view on our user base, and I think you fail to address some > needs. Not everybody is a purist here, some `just want to have the job > done', even when this means to err once or twice. You already have your own *database* specification, and you should continue to use and evolve it. I think that issues should be *documented* in VuXML. Did you mean something else? [...] >> Maybe you can describe this a bit more fully? (BTW, Can you make a >> separate thread for each of your proposals?) I felt that the affected >> ranges were sufficient, but if there is a better way let's hear it. > > There is no way to deduce from the VuXML document alone whether a > fixed version exists (like in port<1.5: is there a port-1.5 in the > tree or not?). So basically the user has to do the research whether > s/he could just upgrade or has to deinstall the port completely (or > live with the vulnerability). I understand what you are saying, but I was looking for a proposal here. Frankly, what we have seems appropriate. Either a non-vulnerable version --- older or newer --- is available, or it isn't. But of course I haven't seen a good alternative approach. [...] >> Well, I'm not sure I'm convinced. But let's try an example. >> >> <references> >> <cvename description="BMP heap overflow">CAN-2004-0691</cvename> >> <cvename description="XPM crash">CAN-2004-0692</cvename> >> <cvename description="GIF crash">CAN-2004-0693</cvename> >> >> <url>http://www.trolltech.com/developer/changes/changes-3.3.3.html</ >> url> >> <url>http://scary.beasts.org/security/CESA-2004-004.txt</url> >> </references> >> >> Is this the kind of thing you mean? > > More or less. Since these CVE references have no title yet, they would > be shown as simple URLs. For my part, I would want to *always* show the URLs, and the description separately (extra line or column, "hover" pop-up, whatever). > But you could have <url description="Trolltech - Changes 3.3.3"> > http://www.trolltech.com/developer/changes/changes-3.3.3.html</url>. >> I wonder if it will really be >> used much *shrug* (by authors or users). I'd like to know what you >> mean by `a tool that adds missing descriptions'. Are you thinking >> of following the references and snarfing the <title> or similar? > > Jup, with a possibility to edit or deny it when it is too meaningless. > >> That might be neat. But, it would create two different meanings >> for this attribute: (a) editorial description, i.e. to disambiguate >> multiple similar references to CVE names, source code, whatever; (b) >> the referenced item's description, e.g. <title> for HTML documents, >> Subject: for email. I'm not sure I like that. As you seem to point >> out, (b) could be largely automated, so I'm not sure it is worth >> storing it in the document. Then when rendering a document, how will >> a reader know that what she sees is (a) or (b)? > > Probably (b) or the url is fine here. (a) is too much work in the > general case. So I guess what you are proposing is a tool that reads a VuXML document, snarfs all the remote reference titles, and spits out a new VuXML document with the titles inserted? In effect, this is just caching remote reference titles, isn't it? It seems to me (a) is useful to include in VuXML since it can only be provided by authors, while (b) doesn't seem to be very useful except possibly in rendering... and I have a feeling that a large number (maybe majority) of references have titles that are not well suited to being lifted out of context. CVE names are a particularly good example. All of them will say 'CAN-YYYY-XXXX (under review)' or 'CVE-YYYY-XXXX', which is useless. Advisories are often plain text, and so have no title that can be automatically snarfed. Web-based mailing list archives have additional spam at best, and often are completely useless (e.g. ``SecurityFocus BUGTRAQ Mailing List: BugTraq'' or ``Neohapsis Archives - Red Hat Linux - #0003 - [RHSA-2004:158-01] Updated cadaver package fixes security vulnerability in neon''). Yes, upon further reflection, I think automatically filling out the information in (b) is not a great idea. But I think (a) could be really useful in some cases... except that, I think authors probably would not fill it out often. >> One more thing: I suggested `description' as the attribute name, but I >> have second thoughts. I believe it might be better to pick something >> that is not also an element name, and perhaps `description' is a >> little too much typing. > > Whatever you like. I suppose we'll wait for inspiration, then :-) > Since entries should be done by tools, I don't care how much typing is > necessary. Well *in fact* manual editing is how VuXML documents are created/edited, along with billions of other XML and SGML documents, regardless of the availability of tools. This is kind of beside the point, but you seem to like to say this in justification of one thing or another when it is simply wrong. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From trhodes at FreeBSD.org Wed Aug 25 11:05:57 2004 From: trhodes at FreeBSD.org (Tom Rhodes) Date: Wed Aug 25 11:16:04 2004 Subject: making <description> optional In-Reply-To: <825857C8-F6B4-11D8-9236-000A95BC6FAE@FreeBSD.org> References: <86914F26-F516-11D8-8CAA-00039312D914@fillmore-labs.com> <825857C8-F6B4-11D8-9236-000A95BC6FAE@FreeBSD.org> Message-ID: <20040825140459.36c75b42@localhost.pittgoth.com> On Wed, 25 Aug 2004 11:33:34 -0500 Jacques Vidrine <nectar@freebsd.org> wrote: > On Aug 23, 2004, at 10:10 AM, Oliver Eikemeier wrote: > > I have a supplementary databases that are as much `personal' as vuxml > > is. The portaudit text database has been announced and documented as > > mentioned in a previous discussion. > > I'm sorry, I couldn't find that text (`Please contact [...]') anywhere > in portaudit.txt or related files. Now I realize that is generated in > the pages at ~eik/portaudit. I think we're on the same page now. > > > Anyway, I think making the entry optional would be the best solution, > > but if you prefer a placeholder, we can keep `Please contact the > > FreeBSD Security Team for more information'. > > I prefer an actual description, but OK :-) > > > Whatever you like. Simply using dummy values is fine with me. > > For the sake of harmony, let's agree that setting > <discovery>unknown</discovery> is acceptable for now. Gimme a few days > to check that stuff doesn't break when processing said entry. Ok, so we're on our way to closing this entire discussion. I seen Oliver's wishlist but have been too busy to reply. I'll try to get it done in the next few days. Thanks guys! -- Tom Rhodes From nectar at FreeBSD.org Wed Aug 25 11:09:02 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 11:17:40 2004 Subject: portaudit wishlist In-Reply-To: <15E125E6-F518-11D8-8CAA-00039312D914@fillmore-labs.com> References: <15E125E6-F518-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <CB00FE41-F6C1-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 23, 2004, at 10:21 AM, Oliver Eikemeier wrote: > Thinking a little about it, I believe this should be discussed in a > place where portaudit users are present, either ports@ or security. > freebsd-vuxml@ has too few subscribers to get an useful picture of > what features desired by users are. Certainly if you are looking for feedback on portaudit features, another list might be more appropriate. This list is for discussing VuXML format/contents/and so on. So I'd expect for example that you'd know "I need to implement feature X in portaudit" and make a proposal here on what might be needed in VuXML to make feature X possible for you to implement. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Wed Aug 25 11:59:14 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 12:12:10 2004 Subject: database tables for VuXML In-Reply-To: <412B7D29.31547.1472F80@localhost> References: <41279E59.13631.76AD2AB9@localhost> <412B7D29.31547.1472F80@localhost> Message-ID: <CDBB0430-F6C8-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 24, 2004, at 4:38 PM, Dan Langille wrote: [...] > What is this type field? I didn't follow what the affected table > contains. Type is either `package' or `system'. You are probably concerned only with the former. [...] > FreshPorts stores PORTVERSION and PORTREVISION as separate text > fields. I take it that the ranges equations should compare only > PORTVERSION and ignore PORTREVISION. One can think a version number as a tuple as follows (most significant first): (PORTEPOCH, PORTVERSION, PORTREVISION). Comparisons are done on these tuples. > SQLite is a great little database. I like what I've seen of it > (mostly via my work on Bacula). I'll be using PostgreSQL for this. Yeah, SQLite is great for single-user things. I especially like the way it is so easy to add SQL functions from C. > [...] > My plan is to populate empty the vuxml_* tables each time there is a > commit to the data file. This keeps the vuxml system totally > separate from FreshPorts. > > Each row under Commit History (e.g. > http://beta.freshports.org/sysutils/bacula/) relates to a row from > the commit_log_ports table. That looks something like this: > > create table commit_log_ports > ( > commit_log_id integer not null, > port_id integer not null, > needs_refresh smallint not null, > port_version text , > port_revision text , > primary key (commit_log_id, port_id) > ); You'll need a `port_epoch' here as well. So these get created as commits "come in"? > I'll probably create another table commit_log_ports_vuxml: > > create table commit_log_ports > ( > commit_log_id integer not null, > port_id integer not null, > vuxml_id integer not null , > primary key (commit_log_id, port_id) > ); > > Looking at the current data, there's about 140 affected ports, but I > haven't broken that with respect to ranges, which is what the above > will do. > > I'll either do that, or add the vuxml_id column to the > commit_log_ports table, but I'd rather keep it separate. Time will > tell. Seems like it'll work to me :-) Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Wed Aug 25 12:05:21 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 12:21:59 2004 Subject: (was Re: database tables for VuXML) In-Reply-To: <F42627C5-F516-11D8-8CAA-00039312D914@fillmore-labs.com> References: <F42627C5-F516-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <A96CC28A-F6C9-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 23, 2004, at 10:13 AM, Oliver Eikemeier wrote: > Jacques A. Vidrinew wrote: > [...] >> Yeah, stuff like `i386'. I don't see it (or <category>) as useful >> for the FreeBSD Project today, either: these are for use by other >> projects. In the case of <architecture>, I'm not yet sure that it >> will make it through pre-1.2. I'm still working with the project to >> understand their needs. > > FWIIW, I would find a <system> tag that enables the sharing of entries > across operating systems far more useful. How would you use it? Will you maintain vulnerability information for several projects in a single VuXML file? If so, I think the reasonable extension would be to specify a metadata attribute on the existing <package> and <system> tags. You would not likely want the `cross-product' effect. Then one could easily generate individual VuXML documents from a single document, simply selecting on this attribute and including only relevent entries. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Wed Aug 25 12:18:06 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 12:32:23 2004 Subject: database tables for VuXML In-Reply-To: <4129CCC9.11697.7F32B4B4@localhost> References: <4129070D.28233.7C2E4023@localhost> <4129CCC9.11697.7F32B4B4@localhost> Message-ID: <717D5B12-F6CB-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 23, 2004, at 9:54 AM, Dan Langille wrote: > The package names? I think you are referring to PORTVERSION and > PORTREVISION. For example, 1.34.6 is PORTVERSION, and 2 is the > PORTREVISION. This information is generated from the following: > > make -V PORTREVISION -V PORTVERSION > > Is that what you are referring to? You accidently answered my question :-) I guess you have set those whenever the commit mail came in ... so the view of past package versions is from past executions of `make -V'. I wish I could `cd /usr/ports/some/app && make package-names' to get a list of all package names this port ever built. or some such. > I welcome example icons! > I was thinking to link people to the internal FreshPorts pages, and > from there to the official vuxml.freebsd.org pages if they want > additional information. Sounds great. I imagine you probably generate the pages from your database, but if you need any help with rendering directly from the XML, lemme know. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From dan at langille.org Wed Aug 25 13:02:45 2004 From: dan at langille.org (Dan Langille) Date: Wed Aug 25 13:10:19 2004 Subject: database tables for VuXML In-Reply-To: <717D5B12-F6CB-11D8-9236-000A95BC6FAE@FreeBSD.org> References: <4129CCC9.11697.7F32B4B4@localhost> Message-ID: <412CB824.3652.6159279@localhost> On 25 Aug 2004 at 14:17, Jacques Vidrine wrote: > > On Aug 23, 2004, at 9:54 AM, Dan Langille wrote: > > The package names? I think you are referring to PORTVERSION and > > PORTREVISION. For example, 1.34.6 is PORTVERSION, and 2 is the > > PORTREVISION. This information is generated from the following: > > > > make -V PORTREVISION -V PORTVERSION > > > > Is that what you are referring to? > > You accidently answered my question :-) I guess you have set those > whenever the commit mail came in ... so the view of past package > versions is from past executions of `make -V'. Correct. What you see under Commit History are values obtained from make -V at the time of that commit. > I wish I could `cd /usr/ports/some/app && make package-names' to get a > list of all package names this port ever built. or some such. > > > I welcome example icons! > > I was thinking to link people to the internal FreshPorts pages, and > > from there to the official vuxml.freebsd.org pages if they want > > additional information. > > Sounds great. I imagine you probably generate the pages from your > database, but if you need any help with rendering directly from the > XML, lemme know. Thanks. Most pages on FreshPorts have content obtained from the database. See http://www.freshports.org/how-big-is-it.php for page stats (about 250,000 pages now). -- Dan Langille : http://www.langille.org/ From dan at langille.org Wed Aug 25 13:12:25 2004 From: dan at langille.org (Dan Langille) Date: Wed Aug 25 13:20:40 2004 Subject: database tables for VuXML In-Reply-To: <CDBB0430-F6C8-11D8-9236-000A95BC6FAE@FreeBSD.org> References: <412B7D29.31547.1472F80@localhost> Message-ID: <412CBA68.15997.61E6A50@localhost> On 25 Aug 2004 at 13:58, Jacques Vidrine wrote: > > FreshPorts stores PORTVERSION and PORTREVISION as separate text > > fields. I take it that the ranges equations should compare only > > PORTVERSION and ignore PORTREVISION. > > One can think a version number as a tuple as follows (most significant > first): (PORTEPOCH, PORTVERSION, PORTREVISION). Comparisons are done > on these tuples. Oh, PORTEPOCH is first? Why? Shouldn't it be last? Here is the value forumula from bsd.port.mk PKGNAME= ${PKGNAMEPREFIX}${PORTNAME}${PKGNAMESUFFIX}- ${PORTVERSION:C/[-_,]/./g}${_SUF1}${_SUF2} where _SUF2= ,${PORTEPOCH} > > My plan is to populate empty the vuxml_* tables each time there is a > > commit to the data file. This keeps the vuxml system totally > > separate from FreshPorts. > > > > Each row under Commit History (e.g. > > http://beta.freshports.org/sysutils/bacula/) relates to a row from > > the commit_log_ports table. That looks something like this: > > > > create table commit_log_ports > > ( > > commit_log_id integer not null, > > port_id integer not null, > > needs_refresh smallint not null, > > port_version text , > > port_revision text , > > primary key (commit_log_id, port_id) > > ); > > You'll need a `port_epoch' here as well. So these get created as > commits "come in"? Yes, for each commit, a row is added to the commit_log table. For each port in a commit, a row is added to the commit_log_ports table. In addition to PORTEPOCH, I think I need to add: NO_LATEST_LINK NO_PACKAGE I may also want to add PKGNAME -- Dan Langille : http://www.langille.org/ From cswiger at mac.com Wed Aug 25 13:38:08 2004 From: cswiger at mac.com (Charles Swiger) Date: Wed Aug 25 13:40:16 2004 Subject: database tables for VuXML In-Reply-To: <412CBA68.15997.61E6A50@localhost> References: <412B7D29.31547.1472F80@localhost> <412CBA68.15997.61E6A50@localhost> Message-ID: <AA43EF1A-F6D6-11D8-8247-003065ABFD92@mac.com> On Aug 25, 2004, at 4:12 PM, Dan Langille wrote: > On 25 Aug 2004 at 13:58, Jacques Vidrine wrote: >> One can think a version number as a tuple as follows (most significant >> first): (PORTEPOCH, PORTVERSION, PORTREVISION). Comparisons are done >> on these tuples. > > Oh, PORTEPOCH is first? Why? Shouldn't it be last? Here is the > value forumula from bsd.port.mk > > PKGNAME= ${PKGNAMEPREFIX}${PORTNAME}${PKGNAMESUFFIX}- > ${PORTVERSION:C/[-_,]/./g}${_SUF1}${_SUF2} > > where _SUF2= ,${PORTEPOCH} PORTEPOCH is listed last in the package name, but it is more important than the other values. The reason PORTEPOCH exists is to fix PORTVERSION+PORTREVISION not producing a monotonously increasing sequence, right? So, no matter what PORTVERSION is, PORTEPOCH=1 means greater than PORTEPOCH of zero (ie, unset, the default value, whatever). PORTEPOCH=2 with any PORTVERSION is greater than any PORTVERSION value with PORTEPOCH=1, etc. -- -Chuck From eikemeier at fillmore-labs.com Wed Aug 25 16:01:23 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Wed Aug 25 16:06:57 2004 Subject: portaudit wishlist In-Reply-To: <CB00FE41-F6C1-11D8-9236-000A95BC6FAE@FreeBSD.org> Message-ID: <B3E048EE-F6EA-11D8-91E7-00039312D914@fillmore-labs.com> Jacques Vidrine wrote: > On Aug 23, 2004, at 10:21 AM, Oliver Eikemeier wrote: >> Thinking a little about it, I believe this should be discussed in a >> place where portaudit users are present, either ports@ or security. >> freebsd-vuxml@ has too few subscribers to get an useful picture of >> what features desired by users are. > > Certainly if you are looking for feedback on portaudit features, > another list might be more appropriate. This list is for discussing > VuXML format/contents/and so on. So I'd expect for example that you'd > know "I need to implement feature X in portaudit" and make a proposal > here on what might be needed in VuXML to make feature X possible for > you to implement. Uhm, yes. The point is that I want to classify vulnerabilities, so that users can filter the information they get to a certain extend. I don't think the classification should go into an extra file, since many vulnerability databases have classifications. I know that such a feature has a high degree of subjectivity, but I think it is useful when some vulnerabilities just can "stand out", for example 616cf823-f48b-11d8-9837-000c41e2cdad is pretty insignificant. I think I said all this before. I'm not exactly sure what this list is good for, since I keep discussing this stuff with you only, so I want to move this to a place where we can get some more feedback, like "yes, this will be a useful feature" or "no, we don't need that" from someone else besides you and me. Discussing this with people interested in the VuXML format seems like a waste of time, I want to discuss this with people _using_ the data, and the format just has to serve their needs. -Oliver From dan at langille.org Wed Aug 25 21:47:10 2004 From: dan at langille.org (Dan Langille) Date: Wed Aug 25 21:47:11 2004 Subject: icon for VuXML? Message-ID: <412D3303.5833.195280A@localhost> Can anyone recommend/design an icon for VuXML? Something about 13x13 to 21x21 pixals would be good. And something which doesn't look like any of the existing FreshPorts icons such as those at http://www.freshports.org/faq.php Cheers -- Dan Langille : http://www.langille.org/ From dan at langille.org Thu Aug 26 14:24:55 2004 From: dan at langille.org (Dan Langille) Date: Thu Aug 26 14:39:22 2004 Subject: icon for VuXML? In-Reply-To: <412D3303.5833.195280A@localhost> Message-ID: <412E1CE6.21976.526B0F3@localhost> On 26 Aug 2004 at 0:46, Dan Langille wrote: > Can anyone recommend/design an icon for VuXML? Something about 13x13 > to 21x21 pixals would be good. And something which doesn't look like > any of the existing FreshPorts icons such as those at > http://www.freshports.org/faq.php Here is a mock up: http://beta.freshports.org/tmp/freshports.orgftptnftpd.html The goal is to have the little skull beside any package which is affected. -- Dan Langille : http://www.langille.org/ From dan at langille.org Thu Aug 26 15:59:06 2004 From: dan at langille.org (Dan Langille) Date: Thu Aug 26 15:59:08 2004 Subject: database tables for VuXML In-Reply-To: <412B7D29.31547.1472F80@localhost> References: <20040822192915.GA17478@madman.celabo.org> Message-ID: <412E32F9.17168.57CEB1D@localhost> I am now finalizing my tables. I have some comments below and some questions about your tables. On 24 Aug 2004 at 17:38, Dan Langille wrote: > On 22 Aug 2004 at 14:29, Jacques A. Vidrine wrote: > > > [Added freebsd-vuxml@] > > > > On Sat, Aug 21, 2004 at 07:11:21PM -0400, Dan Langille wrote: > > > Hi folks, > > > > > > I'm in the early stage of incorporating VuXML information into > > > FreshPorts. > > > > Alright! That'll be cool. > > > > > I have a preliminary database table design which appears > > > below. If you think I've missed any essential information out of > > > this design, please let me know. > > > > > > First: has anyone already worked on creating database tables for the > > > VuXML data? > > > > I have an old prototype application which uses SQLite. FWIW, here is > > the schema I used then. (the formatting is a little funny because this > > was embedded in some other code) > > > > CREATE TABLE vuxml ( > > vid VARCHAR PRIMARY KEY UNIQUE > > , topic VARCHAR > > , description VARCHAR > > , discovery DATE > > , entry DATE > > , modified DATE > > -- computed max(entry, modified) for convenience in > > -- other expressions > > , last DATE > > ); I'm not going to have a last column. And I am using TEXT, not DATE, as the values are not always dates. I recall reading someone suggesting allowing unknown. If the date is unknown, I suggest leaving this field empty. If the suggestion above is accepted, I will reconsider my DATE versus TEXT decision as I could then use a NULL value. > This is pretty much identical to what I have. > > > > > CREATE TABLE refs ( > > vid VARCHAR > > , type VARCHAR > > , text VARCHAR > > ); What is this type? I fear I have asked this already. I have added the table but without the type field. > Ahh, yes, that repeating group. I'm not storing that. I should be > > > CREATE TABLE affected ( > > vid VARCHAR > > , type VARCHAR > > ); > > What is this type field? I didn't follow what the affected table > contains. Ahh, type is either package or system. e.g. <package> <name>tnftpd</name> <range><lt>20040810</lt></range> </package> <package> <name>lukemftpd</name> <range><ge>0</ge></range> </package> <system> <name>FreeBSD</name> <range><ge>4.7</ge></range> </system> > > CREATE TABLE names ( > > affected INTEGER > > , name VARCHAR > > ); This I now understand. It's a list of the affected packages, or systems. For packages, it appears to be PORTNAME. Is that true? I see now why you have names and affect separate: <package> <name>gaim</name> <name>ja-gaim</name> <range><lt>0.81_1</lt></range> </package> A given package may have more than one name (e.g. slave ports). > >From your example below, the names table would have entries for mutt > and ja-mutt. I think I'll do that as well. See the end of my > message for how I plan to relate VuXML entries to specific ports. > > > CREATE TABLE ranges ( > > affected INTEGER > > , lowop VARCHAR > > , low VARCHAR > > , glow0 INTEGER > > , glow1 INTEGER > > , glow2 INTEGER > > , glow3 INTEGER > > , highop VARCHAR > > , high VARCHAR > > , ghigh0 INTEGER > > , ghigh1 INTEGER > > , ghigh2 INTEGER > > , ghigh3 INTEGER > > ); > > > > The `ranges' table perhaps looks erm interesting. In this case, > > I encoded each package version number into four 32-bit integers. > > This allowed one to lookup an affected package completely in SQL. > > However, the encoding I used may now be incorrect in some ways, since > > pkg_install et al were recently changed to treat version numbers > > differently. Each package has a list of 1 or more ranges which are affected. The ranges affect all names listed in the package. e.g.: <package> <name>samba</name> <range><ge>3</ge><lt>3.0.5,1</lt></range> <range><lt>2.2.10</lt></range> </package> Thanks. This is coming together and I'm getting closer to having tables. -- Dan Langille : http://www.langille.org/ From dan at langille.org Thu Aug 26 18:23:50 2004 From: dan at langille.org (Dan Langille) Date: Thu Aug 26 18:23:51 2004 Subject: FreshPorts - VuXML tables - diagram Message-ID: <412E54E5.5382.6016DEA@localhost> Here's the diagram of the tables I'm considering. This should duplicate the entire contents of the VuXML table. http://beta.freshports.org/tmp/FreshPorts-VuXML-tables.jpg The commit_log_ports_vuxml table relates each entry in the vuxml table to one or more commits and to specific ports within that commit. The actual PORTREVISION, PORTVERSION, PORTEPOCH etc, are stored in another table (commit_log_ports), which is not shown. Why another table just for vuxml? It keep the vuxml separate from the FreshPorts data. The table I would put this information it is the commit_log_ports table (which has about 138,718 rows). I expect the commit_log_ports_vuxml table to have about 1000 rows (i.e. significantly less rows). I'd consider merging the data if the number of vuxml rows started to approach 30% of commit_log_ports... but I don't see that happening. cheers -- Dan Langille : http://www.langille.org/ From dan at langille.org Fri Aug 27 01:49:25 2004 From: dan at langille.org (Dan Langille) Date: Fri Aug 27 02:24:38 2004 Subject: tnftpd -- remotely exploitable vulnerability Message-ID: <20040827044722.G74931@xeon.unixathome.org> For vid="c4b025bb-f05d-11d8-9837-000c41e2cdad", should the following: <package> <name>FreeBSD</name> <range><ge>4.7</ge></range> </package> actually be change to system from package? i.e. <system> <name>FreeBSD</name> <range><ge>4.7</ge></range> </system> -- Dan Langille - http://www.langille.org/ From dan at langille.org Fri Aug 27 04:07:09 2004 From: dan at langille.org (Dan Langille) Date: Fri Aug 27 04:50:10 2004 Subject: FreshPorts - VuXML tables - DDL + data Message-ID: <20040827051637.T78365@xeon.unixathome.org> Here is the DDL I created this morning. Note that some slight changes have been made relative to the diagram I posted last night. The diagram is now out of date. The rest of this message contains a sample vuxml entry followed by the DDL to create the tables and populate it with a single VuXML entry. It also shows how I mark which packages are affected by a given vid (i.e. how I populate the commit_log_ports_vuxml table). I have created the code to display VuXML information, but at present, the only data loaded is for http://beta.freshports.org/ftp/tnftpd/ NOTE: I'm assuming there can be multiple VuXML entries for a given version of a package/port. However, there will be only one VuXML icon shown per row of Commit History. Look at the last commit on the page for a commit which is affected by two VuXML entries (this is achieved with a fake VuXML entry). freshports.org=# select id, vid, topic from vuxml; id | vid | topic ----+--------------------------------------+---------------------------------------------- 1 | c4b025bb-f05d-11d8-9837-000c41e2cdad | tnftpd -- remotely exploitable vulnerability (1 row) freshports.org=# select * from vuxml_affected; id | vuxml_id | type ----+----------+--------- 1 | 1 | package 2 | 1 | package 3 | 1 | system This vid affects two packages and relates to one system. freshports.org=# select * from vuxml_names; id | vport_versionport_versionuxml_affected_id | name ----+-------------------+----------- 1 | 1 | tnftpd 2 | 2 | lukemftpd 3 | 3 | FreeBSD (3 rows) The packages and system are listed above. freshports.org=# select * from vuxml_ranges; id | vuxml_name_id | range_operator_start | range_operator_end | range_version_start | range_version_end ----+---------------+----------------------+--------------------+---------------------+------------------- 7 | 1 | | | < | 20040810 8 | 2 | >= | 0 | | 9 | 3 | >= | 4.7 | | (3 rows) These are the ranges related to each vuxml_affected row. freshports.org=# select * from vuxml_references; vuxml_id | reference | type ----------+-------------------------------------------------------------------------------+------- 1 | http://cvsweb.netbsd.org/bsdweb.cgi/src/libexec/ftpd/ftpd.c#rev1.158 | url 1 | ftp://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2004-009.txt.asc | url 1 | http://lists.netsys.com/pipermail/full-disclosure/2004-August/025418.html | mlist (3 rows) freshports.org=# Given the above information, here is one way to populate the commit_log_ports_vuxml table. freshports.org=# insert into commit_log_ports_vuxml (commit_log_id, port_id, vuxml_id) select commit_log_id, port_id, 1 as vuxml_id from commit_log_ports where port_id = (select id from ports_active where name = 'tnftpd') and port_version < '20040810'; INSERT 0 9 DDL follows ### create table vuxml ( id serial not null, vid text not null, topic text not null, description text not null, date_discovery text , date_entry text , date_modified text , status char(1) not null, primary key (id) ); create table commit_log_ports_vuxml ( id serial not null, commit_log_id integer not null, port_id integer not null, vuxml_id integer not null, primary key (id) ); create table vuxml_ranges ( id serial not null, vuxml_name_id integer not null, range_operator_start text , range_operator_end text , range_version_start text , range_version_end text , primary key (id) ); create table vuxml_references ( vuxml_id integer not null, reference text not null, type text not null, primary key (vuxml_id, reference) ); create table vuxml_affected ( id serial not null, vuxml_id integer not null, type text not null, primary key (id) ); create table vuxml_names ( id serial not null, vuxml_affected_id integer not null, name text not null, primary key (id) ); alter table commit_log_ports_vuxml add foreign key (vuxml_id) references vuxml (id) on update cascade on delete cascade; alter table commit_log_ports_vuxml add foreign key (port_id) references ports (id) on update cascade on delete cascade; alter table commit_log_ports_vuxml add foreign key (commit_log_id) references commit_log (id) on update cascade on delete cascade; alter table vuxml_ranges add foreign key (vuxml_name_id) references vuxml_names (id) on update cascade on delete cascade; alter table vuxml_references add foreign key (vuxml_id) references vuxml (id) on update cascade on delete cascade; alter table vuxml_affected add foreign key (vuxml_id) references vuxml (id) on update cascade on delete cascade; alter table vuxml_names add foreign key (vuxml_affected_id) references vuxml_affected (id) on update cascade on delete cascade; ### Data follows: delete from vuxml where id = 1; insert into vuxml (id, vid, topic, description, date_discovery, date_entry, date_modified, status) values (1, 'c4b025bb-f05d-11d8-9837-000c41e2cdad', 'tnftpd -- remotely exploitable vulnerability', ' lukemftpd(8) is an enhanced BSD FTP server produced within the NetBSD project. The sources for lukemftpd are shipped with some versions of FreeBSD, however it is not built or installed by default. The build system option WANT_LUKEMFTPD must be set to build and install lukemftpd. [NOTE: An exception is FreeBSD 4.7-RELEASE, wherein lukemftpd was installed, but not enabled, by default.] Przemyslaw Frasunek discovered several vulnerabilities in lukemftpd arising from races in the out-of-band signal handling code used to implement the ABOR command. As a result of these races, the internal state of the FTP server may be manipulated in unexpected ways. A remote attacker may be able to cause FTP commands to be executed with the privileges of the running lukemftpd process. This may be a low-privilege ^Ftp\' user if the -r\' command line option is specified, or it may be superuser privileges if -r\' is *not* specified. ', '2004-08-17', '2004-08-17', NULL, 'A'); insert into vuxml_references (vuxml_id, reference, type) values (1, 'http://cvsweb.netbsd.org/bsdweb.cgi/src/libexec/ftpd/ftpd.c#rev1.158', 'url'); insert into vuxml_references (vuxml_id, reference, type) values (1, 'ftp://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2004-009.txt.asc', 'url'); insert into vuxml_references (vuxml_id, reference, type) values (1, 'http://lists.netsys.com/pipermail/full-disclosure/2004-August/025418.html', 'mlist'); insert into vuxml_affected (id, vuxml_id, type) values (1, 1, 'package'); insert into vuxml_affected (id, vuxml_id, type) values (2, 1, 'package'); insert into vuxml_affected (id, vuxml_id, type) values (3, 1, 'system'); insert into vuxml_names (id, vuxml_affected_id, name) values (1, 1, 'tnftpd'); insert into vuxml_names (id, vuxml_affected_id, name) values (2, 2, 'lukemftpd'); insert into vuxml_names (id, vuxml_affected_id, name) values (3, 3, 'FreeBSD'); insert into vuxml_ranges (vuxml_name_id, range_operator_start, range_operator_end, range_version_start, range_version_end) values (1, NULL, NULL, '<', '20040810'); insert into vuxml_ranges (vuxml_name_id, range_operator_start, range_operator_end, range_version_start, range_version_end) values (2, '>=', '0', NULL, NULL); insert into vuxml_ranges (vuxml_name_id, range_operator_start, range_operator_end, range_version_start, range_version_end) values (3, '>=', '4.7', NULL, NULL); -- Dan Langille - http://www.langille.org/ From nectar at FreeBSD.org Fri Aug 27 06:43:39 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Fri Aug 27 07:01:23 2004 Subject: icon for VuXML? In-Reply-To: <412E1CE6.21976.526B0F3@localhost> References: <412D3303.5833.195280A@localhost> <412E1CE6.21976.526B0F3@localhost> Message-ID: <20040827134318.GC54027@madman.celabo.org> On Thu, Aug 26, 2004 at 05:24:54PM -0400, Dan Langille wrote: > Here is a mock up: > > http://beta.freshports.org/tmp/freshports.orgftptnftpd.html > > The goal is to have the little skull beside any package which is > affected. Seems like a good approach to me. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Fri Aug 27 07:04:26 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Fri Aug 27 07:15:30 2004 Subject: database tables for VuXML In-Reply-To: <412E32F9.17168.57CEB1D@localhost> References: <20040822192915.GA17478@madman.celabo.org> <412E32F9.17168.57CEB1D@localhost> Message-ID: <20040827140404.GE54027@madman.celabo.org> On Thu, Aug 26, 2004 at 06:59:05PM -0400, Dan Langille wrote: > I am now finalizing my tables. I have some comments below and some > questions about your tables. [...] > I'm not going to have a last column. And I am using TEXT, not DATE, > as the values are not always dates. I recall reading someone > suggesting allowing unknown. If the date is unknown, I suggest > leaving this field empty. Actually, with SQLite, all columns are really strings even if one specifies other SQL types :-) Oliver requested that `unknown' be a valid value. The only other valid format is YYYY-MM-DD. > If the suggestion above is accepted, I will reconsider my DATE versus > TEXT decision as I could then use a NULL value. Yes, you should expect that `unknown' <discovery> dates are generated by Oliver or his perl script. They should all be transient values: even though such entries will technically be acceptable, they are morally bugs. :-) > > > > > > CREATE TABLE refs ( > > > vid VARCHAR > > > , type VARCHAR > > > , text VARCHAR > > > ); > > What is this type? I fear I have asked this already. I have added > the table but without the type field. This is a different type than the other :-) References are (type, text) tuples, where the type can be "url", "cvename", "uscertta", "mlist", and so forth. The meaning of the text of course depends upon the type. The list of types is constrained by the VuXML DTD, but VuXML processing tools should handle any unrecognized type generically. > > > CREATE TABLE affected ( > > > vid VARCHAR > > > , type VARCHAR > > > ); > > > > What is this type field? I didn't follow what the affected table > > contains. > > Ahh, type is either package or system. > > e.g. > > <package> > <name>tnftpd</name> > <range><lt>20040810</lt></range> > </package> > <package> > <name>lukemftpd</name> > <range><ge>0</ge></range> > </package> > <system> > <name>FreeBSD</name> > <range><ge>4.7</ge></range> > </system> Yes, the type column in *this* table is "package" or "system". > > > CREATE TABLE names ( > > > affected INTEGER > > > , name VARCHAR > > > ); > > This I now understand. It's a list of the affected packages, or > systems. For packages, it appears to be PORTNAME. Is that true? Almost. In FreeBSD ports collection terms, the <name> element contains the PKGNAME minus the version, i.e. ${PKGNAMEPREFIX}${PORTNAME}${PKGNAMESUFFIX}. > I see now why you have names and affect separate: > > <package> > <name>gaim</name> > <name>ja-gaim</name> > <range><lt>0.81_1</lt></range> > </package> > > A given package may have more than one name (e.g. slave ports). Right. Another example is compile time options, e.g. "vim", "vim-lite". [...] > Each package has a list of 1 or more ranges which are affected. The > ranges affect all names listed in the package. > > e.g.: > > <package> > <name>samba</name> > <range><ge>3</ge><lt>3.0.5,1</lt></range> > <range><lt>2.2.10</lt></range> > </package> You got it. > Thanks. This is coming together and I'm getting closer to having > tables. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Fri Aug 27 07:14:45 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Fri Aug 27 07:17:25 2004 Subject: FreshPorts - VuXML tables - DDL + data In-Reply-To: <20040827051637.T78365@xeon.unixathome.org> References: <20040827051637.T78365@xeon.unixathome.org> Message-ID: <20040827141424.GF54027@madman.celabo.org> On Fri, Aug 27, 2004 at 07:07:07AM -0400, Dan Langille wrote: > Here is the DDL I created this morning. Note that some slight changes > have been made relative to the diagram I posted last night. The diagram > is now out of date. > > The rest of this message contains a sample vuxml entry followed by the DDL > to create the tables and populate it with a single VuXML entry. It also > shows how I mark which packages are affected by a given vid (i.e. how I > populate the commit_log_ports_vuxml table). > > I have created the code to display VuXML information, but at present, the > only data loaded is for http://beta.freshports.org/ftp/tnftpd/ Good work! > NOTE: I'm assuming there can be multiple VuXML entries for a given version > of a package/port. However, there will be only one VuXML icon shown per > row of Commit History. Look at the last commit on the page for a commit > which is affected by two VuXML entries (this is achieved with a fake VuXML > entry). Yep, that's a correct assumption. > freshports.org=# select id, vid, topic from vuxml; > id | vid | topic > ----+--------------------------------------+---------------------------------------------- > 1 | c4b025bb-f05d-11d8-9837-000c41e2cdad | tnftpd -- remotely exploitable vulnerability > (1 row) [...snip...] This is all looking pretty good to me. When the beta is up, I'll be happy to keep an eye on the results for anything unusual. By the way-- a bit off-topic from the DBA stuff-- when processing the FreeBSD VuXML document, you may want to insert a validation step between fetching it and loading it into the database. This will catch problems like bad commits or truncated transfers-- cases in which you probably do not want to use the results for updating the database. See ports/security/vuxml/files/validate.sh for a simple example of using xmllint or onsgmls for validation. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From dan at langille.org Fri Aug 27 16:50:50 2004 From: dan at langille.org (Dan Langille) Date: Fri Aug 27 16:50:52 2004 Subject: FreshPorts - VuXML tables - diagram In-Reply-To: <412E54E5.5382.6016DEA@localhost> Message-ID: <412F9098.32178.AD2B8CF@localhost> On 26 Aug 2004 at 21:23, Dan Langille wrote: > Here's the diagram of the tables I'm considering. This should > duplicate the entire contents of the VuXML table. > > http://beta.freshports.org/tmp/FreshPorts-VuXML-tables.jpg This diagram has been updated to reflect recent discussions. -- Dan Langille : http://www.langille.org/ From will at csociety.org Fri Aug 27 20:55:30 2004 From: will at csociety.org (Will Andrews) Date: Sat Aug 28 05:15:33 2004 Subject: database tables for VuXML In-Reply-To: <717D5B12-F6CB-11D8-9236-000A95BC6FAE@FreeBSD.org> References: <4129070D.28233.7C2E4023@localhost> <4129CCC9.11697.7F32B4B4@localhost> <717D5B12-F6CB-11D8-9236-000A95BC6FAE@FreeBSD.org> Message-ID: <20040828035529.GP3862@sirius.firepipe.net> On Wed, Aug 25, 2004 at 02:17:45PM -0500, Jacques Vidrine wrote: > You accidently answered my question :-) I guess you have set those > whenever the commit mail came in ... so the view of past package > versions is from past executions of `make -V'. > > I wish I could `cd /usr/ports/some/app && make package-names' to get a > list of all package names this port ever built. or some such. That's really hard to implement, because ports are highly interdependent. To do this properly, you would need to checkout the entire ports tree n times, where n is the number of times PKGNAME has changed in the history of the ports tree. Plus, they can vary depending on options. Suffice it to say, getting all this information would be astronomically difficult. :-) Regards, -- wca -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-vuxml/attachments/20040827/33f4529e/attachment.bin From dan at langille.org Sun Aug 29 14:44:08 2004 From: dan at langille.org (Dan Langille) Date: Sun Aug 29 14:44:10 2004 Subject: parsing vuln.xml with XML::Node Message-ID: <20040829173317.U9281@xeon.unixathome.org> Hi folks, I've run into a problem parsing the vuln.xlm file. I'm using the perl XML::Node module. The issue is the body field. This field contains XHTML tags (for example, , , <blockquote>). I have been unable to extract the contents of <body>...</body>. The only solution I've found is to explicitly specify tags such as , , and <blockquote>. That is by no means an ideal solution. There must be something I'm missing. It has been suggested I use XML::Parser instead (XML::Node is based upon XML::Parser). See a working example at http://beta.freshports.org/tmp/testing.tgz it runs like this: perl load_vuxml_into_db.pl vuln.xml If someone can figure out how I can do this, it will be appreciated. Thanks. -- Dan Langille - http://www.langille.org/ From dan at langille.org Mon Aug 30 10:39:55 2004 From: dan at langille.org (Dan Langille) Date: Mon Aug 30 10:43:06 2004 Subject: vuln.xml is not XML Message-ID: <20040830133416.X35009@xeon.unixathome.org> I refer to my previous message regarding the difficulties in parsing vuln.xml. I have since learned that any markup (e.g. ) should be be in a CDATA section. See http://www.w3.org/TR/REC-xml/ and look at section 2.7. CDATA sections begin with the string "<![CDATA[" and end with the string "]]>":] I propose that markup be enclosed with a CDATA section. -- Dan Langille - http://www.langille.org/ From nectar at celabo.org Mon Aug 30 17:15:24 2004 From: nectar at celabo.org (Jacques Vidrine) Date: Mon Aug 30 17:17:35 2004 Subject: vuln.xml *is* XML (was Re: vuln.xml is not XML) In-Reply-To: <20040830133416.X35009@xeon.unixathome.org> References: <20040830133416.X35009@xeon.unixathome.org> Message-ID: <CDBCA21A-FAE2-11D8-A99F-000A95BC6FAE@celabo.org> On Aug 30, 2004, at 12:39 PM, Dan Langille wrote: > I refer to my previous message regarding the difficulties in parsing > vuln.xml. I have since learned that any markup (e.g. ) should be > be in > a CDATA section. > > See http://www.w3.org/TR/REC-xml/ and look at section 2.7. > > CDATA sections begin with the string "<![CDATA[" and end with the > string "]]>":] > > I propose that markup be enclosed with a CDATA section. No this is absolutely wrong :-) The XHTML is embedded with VuXML... the whole document is one XML document. Some elements are in the VuXML namespace, while others are in the XHTML namespace. Markup cannot exist in a CDATA section--- if it is in a CDATA section, it is *not* markup but *text content*. I saw your earlier message about XML::Node, but since I am not familiar with that (or XML::Parser), I did not understand what problem you were having. Could you try to describe it differently? Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From kuriyama at imgsrc.co.jp Mon Aug 30 17:23:21 2004 From: kuriyama at imgsrc.co.jp (Jun Kuriyama) Date: Mon Aug 30 17:26:18 2004 Subject: vuln.xml *is* XML (was Re: vuln.xml is not XML) In-Reply-To: <CDBCA21A-FAE2-11D8-A99F-000A95BC6FAE@celabo.org> References: <20040830133416.X35009@xeon.unixathome.org> <CDBCA21A-FAE2-11D8-A99F-000A95BC6FAE@celabo.org> Message-ID: <7mk6vg2m15.wl@black.imgsrc.co.jp> At Mon, 30 Aug 2004 19:15:02 -0500, Jacques Vidrine wrote: > > I refer to my previous message regarding the difficulties in parsing > > vuln.xml. I have since learned that any markup (e.g. ) should be > > be in > > a CDATA section. > > > > See http://www.w3.org/TR/REC-xml/ and look at section 2.7. > > > > CDATA sections begin with the string "<![CDATA[" and end with the > > string "]]>":] > > > > I propose that markup be enclosed with a CDATA section. > > No this is absolutely wrong :-) The XHTML is embedded with VuXML... > the whole document is one XML document. Some elements are in the VuXML > namespace, while others are in the XHTML namespace. Markup cannot > exist in a CDATA section--- if it is in a CDATA section, it is *not* > markup but *text content*. Both are correct. In good old XML world, we should use CDATA section to quote external markup. On the other hand, VuXML lives in XML + Namespace world (see related recommendations). > I saw your earlier message about XML::Node, but since I am not familiar > with that (or XML::Parser), I did not understand what problem you were > having. Could you try to describe it differently? I'm not sure XML::Parser can handle namespace correctly. If it cannot do such, parser will confuse when it reads markups with namespace. -- Jun Kuriyama <kuriyama@imgsrc.co.jp> // IMG SRC, Inc. <kuriyama@FreeBSD.org> // FreeBSD Project From dan at langille.org Mon Aug 30 17:34:19 2004 From: dan at langille.org (Dan Langille) Date: Mon Aug 30 17:35:31 2004 Subject: vuln.xml *is* XML (was Re: vuln.xml is not XML) In-Reply-To: <7mk6vg2m15.wl@black.imgsrc.co.jp> References: <20040830133416.X35009@xeon.unixathome.org> <7mk6vg2m15.wl@black.imgsrc.co.jp> Message-ID: <20040830203241.V35009@xeon.unixathome.org> On Tue, 31 Aug 2004, Jun Kuriyama wrote: > At Mon, 30 Aug 2004 19:15:02 -0500, > Jacques Vidrine wrote: > > > I refer to my previous message regarding the difficulties in parsing > > > vuln.xml. I have since learned that any markup (e.g. ) should be > > > be in > > > a CDATA section. > > > > > > See http://www.w3.org/TR/REC-xml/ and look at section 2.7. > > > > > > CDATA sections begin with the string "<![CDATA[" and end with the > > > string "]]>":] > > > > > > I propose that markup be enclosed with a CDATA section. > > > > No this is absolutely wrong :-) The XHTML is embedded with VuXML... > > the whole document is one XML document. Some elements are in the VuXML > > namespace, while others are in the XHTML namespace. Markup cannot > > exist in a CDATA section--- if it is in a CDATA section, it is *not* > > markup but *text content*. > > Both are correct. In good old XML world, we should use CDATA section > to quote external markup. On the other hand, VuXML lives in XML + > Namespace world (see related recommendations). > > > I saw your earlier message about XML::Node, but since I am not familiar > > with that (or XML::Parser), I did not understand what problem you were > > having. Could you try to describe it differently? > > I'm not sure XML::Parser can handle namespace correctly. If it cannot > do such, parser will confuse when it reads markups with namespace. With CDATA, it works, without, it fails and I have to treat every , and <blockquote> as a node, not markup. -- Dan Langille - http://www.langille.org/ From nectar at FreeBSD.org Mon Aug 30 18:25:41 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Mon Aug 30 18:25:45 2004 Subject: vuln.xml *is* XML (was Re: vuln.xml is not XML) In-Reply-To: <7mk6vg2m15.wl@black.imgsrc.co.jp> References: <20040830133416.X35009@xeon.unixathome.org> <CDBCA21A-FAE2-11D8-A99F-000A95BC6FAE@celabo.org> <7mk6vg2m15.wl@black.imgsrc.co.jp> Message-ID: <9E499E76-FAEC-11D8-84D2-000A95BC6FAE@FreeBSD.org> On Aug 30, 2004, at 7:23 PM, Jun Kuriyama wrote: > Both are correct. In good old XML world, we should use CDATA section > to quote external markup. On the other hand, VuXML lives in XML + > Namespace world (see related recommendations). If you want to quote external markup as *text*, then sure: CDATA is one way of doing that (character and entity references are another). In this case, it is just *text*, not markup--- it looks like markup but it isn't as far as XML processors are concerned. But if you want to do something with that markup (e.g. validation, XSLT) then you really must use real XML and namespaces. I guess you are probably bringing this up from the perspective of DocBook, but it just happens that DocBook--- and some other XML applications such as XML-RPC and RSS--- was born before namespaces and has not adopted support (yet). So we're left with the CDATA workaround that we had to use with SGML. This should never be done in new XML applications. This is finally being addressed in some versions of DocBook (e.g. DocBook 4.3 + SVG). >> I saw your earlier message about XML::Node, but since I am not >> familiar >> with that (or XML::Parser), I did not understand what problem you were >> having. Could you try to describe it differently? > > I'm not sure XML::Parser can handle namespace correctly. If it cannot > do such, parser will confuse when it reads markups with namespace. I don't believe that is correct. Tools that do not grok namespaces will just not see the namespaces. They will still parse the content just fine. Since we use default namespace declarations by convention in vuln.xml, it is particularly un-obtrusive: a parser will just see "xmlns" attribute nodes, but otherwise continue just fine. Basically, a namespace-aware processor will see events like these: start element (http://www.vuxml.org/app/vuxml-1/, description) attributes [] start element (http://www.w3.org/1999/xhtml, body) attributes [] start element (http://www.w3.org/1999/xhtml, blockquote) attributes [(cite, "http://...")] ... end element (http://www.w3.org/1999/xhtml, blockquote) end element (http://www.w3.org/1999/xhtml, body) end element (http://www.vuxml.org/app/vuxml-1/, description) while an old XML processor with no support for namespaces will see events like these: start element description attributes [] start element body attributes [(xmlns, "http://www.w3.org/1999/xhtml")] start element blockquote attributes [(cite, "http://...")] ... end element blockquote end element body end element description You can even ignore the namespaces if you like. You just need to "remember" when you are processing stuff inside a <description> element versus not. AFAIK, XML::Node is based on XML::Parser which is based on expat. expat supports namespaces perfectly well, so it is surprising if the Perl modules built on top of it do not. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Mon Aug 30 18:30:31 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Mon Aug 30 18:30:33 2004 Subject: vuln.xml *is* XML (was Re: vuln.xml is not XML) In-Reply-To: <20040830203241.V35009@xeon.unixathome.org> References: <20040830133416.X35009@xeon.unixathome.org> <20040830203241.V35009@xeon.unixathome.org> Message-ID: <4C792F15-FAED-11D8-84D2-000A95BC6FAE@FreeBSD.org> On Aug 30, 2004, at 7:34 PM, Dan Langille wrote: > With CDATA, it works, With CDATA, it breaks: some markup is incorrectly treated as text content. > without, it fails and I have to treat every , > and <blockquote> as a node, not markup. without, it works perfectly. That is exactly how it is supposed to work, Dan :-) I guess I understand now what you are trying to accomplish. If you want to stuff processed XML (i.e. an XML nodeset) into a database, you must first convert the nodeset into a character stream in the usual fashion. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org

From nectar at FreeBSD.org Tue Aug 17 10:58:58 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Tue Aug 17 10:59:02 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <56FC3488-F075-11D8-924A-00039312D914@fillmore-labs.com> References: <20040817122453.05edaaea@localhost> <56FC3488-F075-11D8-924A-00039312D914@fillmore-labs.com> Message-ID: <20040817175847.GC43426@madman.celabo.org> [Moving to freebsd-vuxml ... oh how I wish Bcc worked so that people on the other list knew where this went :-) ] On Tue, Aug 17, 2004 at 07:46:16PM +0200, Oliver Eikemeier wrote: > When you can live with the dummy text produced by my perl script > ("Please contact the FreeBSD Security Team for more information.") and > we can make the `discovered' entry optional, fine with me. I can write > a `make entry' perl script that parses a form an generates a template > entry, send-pr like. FWIW, this sounds fine by me, except about the part. I see your point about it though... it may be dangerous to have a bogus value (like the date of entry), because it may not get corrected later. But I don't want it optional, so that it is not forgotten. Perhaps we need the possiblity of marking something explicitly for such occassions ... In the mean time, could the date of entry be used? And perhaps a comment could be a workaround for now, something like 2004-08-17 Ugly, I know, but the current format wasn't made for works-in-progress. Maybe we can make some options for that... > >In place of arguing, start forging some code to check the base > >system against the security listings in vuln.xml. > > portaudit could easily do that. The only thing useful here would be to > use __FreeBSD_versions, so we can check -STABLE and -CURRENT too. Or can > I map the version numbers somehow? I added __FreeBSD_versions in the > last entry (multiple CVS vulnerabilities), but they are commented out > since I don't know what the right syntax is. By way of example, I've been using FreeBSD 4.7-RELEASE-p1 == 4.7_1. I'm not entirely satisfied and I am open to suggestions. This part has been ill-specified. :-( Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From trhodes at FreeBSD.org Tue Aug 17 11:06:14 2004 From: trhodes at FreeBSD.org (Tom Rhodes) Date: Tue Aug 17 11:06:16 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817175847.GC43426@madman.celabo.org> References: <20040817122453.05edaaea@localhost> <56FC3488-F075-11D8-924A-00039312D914@fillmore-labs.com> <20040817175847.GC43426@madman.celabo.org> Message-ID: <20040817140521.1d0f252d@localhost> On Tue, 17 Aug 2004 12:58:47 -0500 "Jacques A. Vidrine" wrote: > [Moving to freebsd-vuxml ... oh how I wish Bcc worked so that people on > the other list knew where this went :-) ] NOTE: I am not subscribed to this list yet! I'm working on that right now! > > On Tue, Aug 17, 2004 at 07:46:16PM +0200, Oliver Eikemeier wrote: > > When you can live with the dummy text produced by my perl script > > ("Please contact the FreeBSD Security Team for more information.") and > > we can make the `discovered' entry optional, fine with me. I can write > > a `make entry' perl script that parses a form an generates a template > > entry, send-pr like. > > FWIW, this sounds fine by me, except about the part. > I see your point about it though... it may be dangerous to have a > bogus value (like the date of entry), because it may not get corrected > later. But I don't want it optional, so that it is not forgotten. > Perhaps we need the possiblity of marking something explicitly > for such occassions ... > > In the mean time, could the date of entry be used? And perhaps a > comment could be a workaround for now, something like > > 2004-08-17 > > Ugly, I know, but the current format wasn't made for > works-in-progress. Maybe we can make some options for that... How about N/A or "Unknown"? That shows that it needs corrected and that there is no problem. > > > >In place of arguing, start forging some code to check the base > > >system against the security listings in vuln.xml. > > > > portaudit could easily do that. The only thing useful here would be to > > use __FreeBSD_versions, so we can check -STABLE and -CURRENT too. Or can > > I map the version numbers somehow? I added __FreeBSD_versions in the > > last entry (multiple CVS vulnerabilities), but they are commented out > > since I don't know what the right syntax is. > > By way of example, I've been using FreeBSD 4.7-RELEASE-p1 == 4.7_1. I'm > not entirely satisfied and I am open to suggestions. This part has been > ill-specified. :-( Why not ident(1) on the specific files? A quick sh script could do this using variables parsed from VuXML entries. -- Tom Rhodes From eikemeier at fillmore-labs.com Tue Aug 17 11:25:13 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Tue Aug 17 11:25:15 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817175847.GC43426@madman.celabo.org> Message-ID: <0569BE5A-F07B-11D8-924A-00039312D914@fillmore-labs.com> Jacques A. Vidrine wrote: > [Moving to freebsd-vuxml ... oh how I wish Bcc worked so that people on > the other list knew where this went :-) ] > > On Tue, Aug 17, 2004 at 07:46:16PM +0200, Oliver Eikemeier wrote: >> When you can live with the dummy text produced by my perl script >> ("Please contact the FreeBSD Security Team for more information.") and >> we can make the `discovered' entry optional, fine with me. I can write >> a `make entry' perl script that parses a form an generates a template >> entry, send-pr like. > > FWIW, this sounds fine by me, except about the part. > I see your point about it though... it may be dangerous to have a > bogus value (like the date of entry), because it may not get corrected > later. But I don't want it optional, so that it is not forgotten. > Perhaps we need the possiblity of marking something explicitly > for such occassions ... > > In the mean time, could the date of entry be used? And perhaps a > comment could be a workaround for now, something like > > 2004-08-17 > > Ugly, I know, but the current format wasn't made for > works-in-progress. Maybe we can make some options for that... epoch 0? 1970-01-01? Or the date vuxml was announced? This would be easier to find than XXX, especially in a rendered version. Or just leave the entry empty. Any constant will do, it could be easily rendered to `unknown'. I find a non-constant value (date of entry) a bad choice it is more difficult to test against (and could be correct). >>> In place of arguing, start forging some code to check the base >>> system against the security listings in vuln.xml. >> >> portaudit could easily do that. The only thing useful here would be to >> use __FreeBSD_versions, so we can check -STABLE and -CURRENT too. Or >> can >> I map the version numbers somehow? I added __FreeBSD_versions in the >> last entry (multiple CVS vulnerabilities), but they are commented out >> since I don't know what the right syntax is. > > By way of example, I've been using FreeBSD 4.7-RELEASE-p1 == 4.7_1. I'm > not entirely satisfied and I am open to suggestions. This part has been > ill-specified. :-( Ehm, __FreeBSD_version? What's bad with that? Documented in the Porters Handbook, and to find out. -Oliver From nectar at FreeBSD.org Tue Aug 17 11:25:22 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Tue Aug 17 11:25:24 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817140521.1d0f252d@localhost> References: <20040817122453.05edaaea@localhost> <56FC3488-F075-11D8-924A-00039312D914@fillmore-labs.com> <20040817140521.1d0f252d@localhost> Message-ID: <20040817182512.GA46244@madman.celabo.org> On Tue, Aug 17, 2004 at 02:05:21PM -0400, Tom Rhodes wrote: > On Tue, 17 Aug 2004 12:58:47 -0500 > "Jacques A. Vidrine" wrote: > > > [Moving to freebsd-vuxml ... oh how I wish Bcc worked so that people on > > the other list knew where this went :-) ] > > NOTE: I am not subscribed to this list yet! I'm working on that > right now! > > > > > On Tue, Aug 17, 2004 at 07:46:16PM +0200, Oliver Eikemeier wrote: > > > When you can live with the dummy text produced by my perl script > > > ("Please contact the FreeBSD Security Team for more information.") and > > > we can make the `discovered' entry optional, fine with me. I can write > > > a `make entry' perl script that parses a form an generates a template > > > entry, send-pr like. > > > > FWIW, this sounds fine by me, except about the part. > > I see your point about it though... it may be dangerous to have a > > bogus value (like the date of entry), because it may not get corrected > > later. But I don't want it optional, so that it is not forgotten. > > Perhaps we need the possiblity of marking something explicitly > > for such occassions ... > > > > In the mean time, could the date of entry be used? And perhaps a > > comment could be a workaround for now, something like > > > > 2004-08-17 > > > > Ugly, I know, but the current format wasn't made for > > works-in-progress. Maybe we can make some options for that... > > How about N/A or "Unknown"? That shows that it needs corrected > and that there is no problem. Thank you, I'm braindead lately. We don't need another element or anything, we can just use a fixed string instead of a date string. I prefer unspecified but the others (in lower case) might be OK, also. But let's just pick one. I'll have to check if this breaks anything existing, but I feel it will be easy to accomodate. > > By way of example, I've been using FreeBSD 4.7-RELEASE-p1 == 4.7_1. I'm > > not entirely satisfied and I am open to suggestions. This part has been > > ill-specified. :-( > > Why not ident(1) on the specific files? A quick sh script could > do this using variables parsed from VuXML entries. Aaaaaahh! /me runs away Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Tue Aug 17 11:27:30 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Tue Aug 17 11:27:32 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <0569BE5A-F07B-11D8-924A-00039312D914@fillmore-labs.com> References: <20040817175847.GC43426@madman.celabo.org> <0569BE5A-F07B-11D8-924A-00039312D914@fillmore-labs.com> Message-ID: <20040817182719.GB46244@madman.celabo.org> On Tue, Aug 17, 2004 at 08:26:56PM +0200, Oliver Eikemeier wrote: > epoch 0? 1970-01-01? Or the date vuxml was announced? This would be > easier to find than XXX, especially in a rendered version. Or just leave > the entry empty. > > Any constant will do, it could be easily rendered to `unknown'. I find a > non-constant value (date of entry) a bad choice it is more difficult to > test against (and could be correct). Yes, you are right, we just need a constant string like 'unknown' or 'unspecified'. > >By way of example, I've been using FreeBSD 4.7-RELEASE-p1 == 4.7_1. I'm > >not entirely satisfied and I am open to suggestions. This part has been > >ill-specified. :-( > > Ehm, __FreeBSD_version? What's bad with that? Documented in the Porters > Handbook, and to find out. __FreeBSD_version is for developers, not users. Users need to see actual release numbers. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From trhodes at FreeBSD.org Tue Aug 17 11:34:13 2004 From: trhodes at FreeBSD.org (Tom Rhodes) Date: Tue Aug 17 11:34:16 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817182719.GB46244@madman.celabo.org> References: <20040817175847.GC43426@madman.celabo.org> <0569BE5A-F07B-11D8-924A-00039312D914@fillmore-labs.com> <20040817182719.GB46244@madman.celabo.org> Message-ID: <20040817143322.56d0b19f@localhost> On Tue, 17 Aug 2004 13:27:19 -0500 "Jacques A. Vidrine" wrote: > On Tue, Aug 17, 2004 at 08:26:56PM +0200, Oliver Eikemeier wrote: > > epoch 0? 1970-01-01? Or the date vuxml was announced? This would be > > easier to find than XXX, especially in a rendered version. Or just leave > > the entry empty. > > > > Any constant will do, it could be easily rendered to `unknown'. I find a > > non-constant value (date of entry) a bad choice it is more difficult to > > test against (and could be correct). > > Yes, you are right, we just need a constant string like 'unknown' or > 'unspecified'. > > > >By way of example, I've been using FreeBSD 4.7-RELEASE-p1 == 4.7_1. I'm > > >not entirely satisfied and I am open to suggestions. This part has been > > >ill-specified. :-( > > > > Ehm, __FreeBSD_version? What's bad with that? Documented in the Porters > > Handbook, and to find out. > > __FreeBSD_version is for developers, not users. Users need to see > actual release numbers. > > Cheers, Do we bump __FreeBSD_version for security patches though? This always drove me nuts. As I said, a simple ident(1) and then a quick compare would work. I do something similar in an upgrade script: # Define two functions here: system_mysql and port_mysql and assign # them a task. system_mysql() { system=`ls /var/db/pkg | grep 'mysql-server' | sed 's/mysql-server-//'` } port_mysql() { port=`cat /usr/ports/databases/mysql323-server/Makefile | grep 'PORTVERSION=' |\ sed 's/PORTVERSION=//' | awk '{ print $1 }'` } #if [ "$port" == "$system" ]; #then /usr/bin/printf "MySQL Server is up to date.\n" >> $log; #elif [ "$port" != "$system" ]; #then /usr/bin/mysqldump --opt pittgoth > /root/pittgoth.sql; #if [ $? -eq 0 ] && [ -s /root/pittgoth.sql ]; #then /bin/chmod 777 /var/db/mysql && /bin/rm -rf /var/db/mysql; # else /usr/bin/printf \ # "An error occured while backing up the database.\n" >> $log && #/usr/bin/printf "This command has failed and will exit.\n" \ #>> $log && exit #fi #else /usr/bin/printf "An unknown error occured during the database upgrade.\n" >> \ #$log; #fi Retarded, perhaps, but it can work. :) -- Tom Rhodes From trhodes at FreeBSD.org Tue Aug 17 11:36:03 2004 From: trhodes at FreeBSD.org (Tom Rhodes) Date: Tue Aug 17 11:36:05 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817182512.GA46244@madman.celabo.org> References: <20040817122453.05edaaea@localhost> <56FC3488-F075-11D8-924A-00039312D914@fillmore-labs.com> <20040817175847.GC43426@madman.celabo.org> <20040817140521.1d0f252d@localhost> <20040817182512.GA46244@madman.celabo.org> Message-ID: <20040817143636.59bcabe0@localhost> On Tue, 17 Aug 2004 13:25:12 -0500 "Jacques A. Vidrine" wrote: [SNIP] > Thank you, I'm braindead lately. We don't need another element or > anything, we can just use a fixed string instead of a date string. I > prefer > > unspecified > > but the others (in lower case) might be OK, also. But let's just pick > one. > > I'll have to check if this breaks anything existing, but I feel it will > be easy to accomodate. I like unknown over unspecified. Unspecified makes me think that a discovery date was never released by the developers while unknown just means that we don't know or can't prove it. -- Tom Rhodes From nectar at FreeBSD.org Tue Aug 17 11:37:52 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Tue Aug 17 11:37:54 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817143322.56d0b19f@localhost> References: <20040817175847.GC43426@madman.celabo.org> <0569BE5A-F07B-11D8-924A-00039312D914@fillmore-labs.com> <20040817143322.56d0b19f@localhost> Message-ID: <20040817183741.GD46244@madman.celabo.org> On Tue, Aug 17, 2004 at 02:33:22PM -0400, Tom Rhodes wrote: > Do we bump __FreeBSD_version for security patches though? Definitely not. -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From eikemeier at fillmore-labs.com Tue Aug 17 12:16:48 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Tue Aug 17 12:16:50 2004 Subject: portaudit wishlist Message-ID: <3AF421B2-F082-11D8-924A-00039312D914@fillmore-labs.com> Ok, things that I think would be really useful (incomplete list): - csh-style braces. When this is not the right syntax, this could be done with ja-bugzilla or ja-kr-cups but we have many slave ports which just differ in prefixes/suffixes, and it would be easy to expand them when reading the file. Yes, portaudit does linear searches. Besides, this will greatly diminish the size of the database. I'm even willing to sacrifice glob patterns `*' and `?' for that, although they can be quite convenient sometimes. - 1.* notation as the `smallest 1.x version possible'. 1.a is not the smallest, besides it is not completely transparent why .a is chosen in the range. When the `*' is the problem, this could be easily changed to a random character, or even a (greater equal range) tag (ok, the name is silly), but I want to have some standard way like >= 1.* < 2.* to match all 1.x and nothing else. No, I don't think >= 1.a < 2.a is good here. - make `discovery' optional. It's a nice-to-have, but sometimes hard to find out, and dummy entries like entry = discovery do not help anyone. (ok, superseeded by another thread). - make `description' optional. It is in the way of `quick' entries which should be researched later. Of course it is acceptable to fill it with a dummy value, but in this case it shouldn't be present IMHO and the dummy value should be provided by the rendering code. Or will an empty tag do? - make a `severity' field available. Of course it might be inaccurate, and software might want to ignore it and provide it's own data. Yet it is useful when you only have time for a quick glance (notify me immediately of severe vulnerabilities, all others should only appear in fridays report). It is a valuable guidance for the users, although I'm aware it is very error-prone. - add a classification into remote/local exploitable - add a `fixed' field that lists a version where the vulnerability is fixed. This could be used for a recommendation message, like "upgrade to version xxx" or "no upgrade is available, please deinstall the port or proceed with caution". This could also realized as an alternate tag. - Also we should add tags for the most popular references. Speaking of references, I would prefer something like CVS Multiple Vulnerabilities, which means they canbe rendered with a meaningful line (but most not, so is legal too). Ok, too many threads now. I have too look into this a little closer. -Oliver From eikemeier at fillmore-labs.com Tue Aug 17 12:19:38 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Tue Aug 17 12:19:40 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817182719.GB46244@madman.celabo.org> Message-ID: Jacques A. Vidrine wrote: > __FreeBSD_version is for developers, not users. Users need to see > actual release numbers. Users need tools that interpret xml files correctly. The file must not be user-readable. -Oliver From eikemeier at fillmore-labs.com Tue Aug 17 12:30:20 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Tue Aug 17 12:31:17 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817185332.2B91D1800A@sirius.firepipe.net> Message-ID: <1F055B5E-F084-11D8-924A-00039312D914@fillmore-labs.com> Pete Fritchman wrote: > Perhaps you could use CVS revision IDs (with 'ident'). For example, > > /usr/bin/passwd: > $FreeBSD: src/usr.bin/passwd/passwd.c,v 1.16.2.1 2001/03/12 > 10:48:08 assar Exp $ > $FreeBSD: src/usr.sbin/pwd_mkdb/pw_scan.c,v 1.14.2.2 2004/02/22 > 11:28:06 charnier Exp $ > $FreeBSD: src/usr.sbin/vipw/pw_util.c,v 1.17.2.4 2002/09/04 > 15:28:10 des Exp $ > $FreeBSD: src/libexec/ypxfr/ypxfr_misc.c,v 1.9.2.2 2002/02/15 > 00:46:54 des Exp $ > $FreeBSD: src/include/rpcsvc/yp.x,v 1.12 1999/08/27 23:45:12 peter > Exp $ > $FreeBSD: src/include/rpcsvc/yppasswd.x,v 1.6 1999/08/27 23:45:12 > peter Exp $ > $FreeBSD: src/usr.sbin/rpc.yppasswdd/yppasswd_private.x,v 1.6 > 1999/08/28 01:19:41 peter Exp $ > $FreeBSD: src/usr.sbin/rpc.yppasswdd/yppasswd_private.x,v 1.6 > 1999/08/28 01:19:41 peter Exp $ > > If a security bug was fixed in passwd.c 1.16.3.1, you could point out > that > I'm vulnerable. Most of the security advisories include the revision > that > things were fixed in, so this shouldn't be too hard. Jacques doens't seem to like this: "Aaaaaahh!". I don't really care ident(1) is fine for me, and it seems like this is the only reliable indication. OTOH you'll need a couple of references (file, list of FreeBSD versions). Doable, so when no other ideas pop up we should do this. -Oliver From trhodes at FreeBSD.org Tue Aug 17 12:36:04 2004 From: trhodes at FreeBSD.org (Tom Rhodes) Date: Tue Aug 17 12:36:07 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <1F055B5E-F084-11D8-924A-00039312D914@fillmore-labs.com> References: <20040817185332.2B91D1800A@sirius.firepipe.net> <1F055B5E-F084-11D8-924A-00039312D914@fillmore-labs.com> Message-ID: <20040817153510.6ccfbd8b@localhost> On Tue, 17 Aug 2004 21:32:05 +0200 Oliver Eikemeier wrote: > Pete Fritchman wrote: > > > Perhaps you could use CVS revision IDs (with 'ident'). For example, > > > > /usr/bin/passwd: > > $FreeBSD: src/usr.bin/passwd/passwd.c,v 1.16.2.1 2001/03/12 > > 10:48:08 assar Exp $ > > $FreeBSD: src/usr.sbin/pwd_mkdb/pw_scan.c,v 1.14.2.2 2004/02/22 > > 11:28:06 charnier Exp $ > > $FreeBSD: src/usr.sbin/vipw/pw_util.c,v 1.17.2.4 2002/09/04 > > 15:28:10 des Exp $ > > $FreeBSD: src/libexec/ypxfr/ypxfr_misc.c,v 1.9.2.2 2002/02/15 > > 00:46:54 des Exp $ > > $FreeBSD: src/include/rpcsvc/yp.x,v 1.12 1999/08/27 23:45:12 peter > > Exp $ > > $FreeBSD: src/include/rpcsvc/yppasswd.x,v 1.6 1999/08/27 23:45:12 > > peter Exp $ > > $FreeBSD: src/usr.sbin/rpc.yppasswdd/yppasswd_private.x,v 1.6 > > 1999/08/28 01:19:41 peter Exp $ > > $FreeBSD: src/usr.sbin/rpc.yppasswdd/yppasswd_private.x,v 1.6 > > 1999/08/28 01:19:41 peter Exp $ > > > > If a security bug was fixed in passwd.c 1.16.3.1, you could point out > > that > > I'm vulnerable. Most of the security advisories include the revision > > that > > things were fixed in, so this shouldn't be too hard. > > Jacques doens't seem to like this: "Aaaaaahh!". I don't really care > ident(1) is fine for me, and it seems like this is the only reliable > indication. OTOH you'll need a couple of references (file, list of > FreeBSD versions). Doable, so when no other ideas pop up we should do > this. Yea, I already mentioned this. We could also stat the UPDATING file for the entry? Perhaps some kind of string could be checked with grep or something. -- Tom Rhodes From eikemeier at fillmore-labs.com Tue Aug 17 12:47:31 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Tue Aug 17 12:47:34 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817153510.6ccfbd8b@localhost> Message-ID: <853F2EE0-F086-11D8-A951-00039312D914@fillmore-labs.com> Tom Rhodes wrote: >> Jacques doens't seem to like this: "Aaaaaahh!". I don't really care >> ident(1) is fine for me, and it seems like this is the only reliable >> indication. OTOH you'll need a couple of references (file, list of >> FreeBSD versions). Doable, so when no other ideas pop up we should do >> this. > > Yea, I already mentioned this. We could also stat the UPDATING > file for the entry? Perhaps some kind of string could be checked > with grep or something. UPDATING needs /src, which a) might not be available, and b) may be more recent the the installed software. -Oliver From ceri at submonkey.net Tue Aug 17 12:04:03 2004 From: ceri at submonkey.net (Ceri Davies) Date: Tue Aug 17 12:58:50 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817175847.GC43426@madman.celabo.org> References: <20040817122453.05edaaea@localhost> <56FC3488-F075-11D8-924A-00039312D914@fillmore-labs.com> <20040817175847.GC43426@madman.celabo.org> Message-ID: <20040817190400.GM5433@submonkey.net> On Tue, Aug 17, 2004 at 12:58:47PM -0500, Jacques A. Vidrine wrote: > [Moving to freebsd-vuxml ... oh how I wish Bcc worked so that people on > the other list knew where this went :-) ] > > On Tue, Aug 17, 2004 at 07:46:16PM +0200, Oliver Eikemeier wrote: > > When you can live with the dummy text produced by my perl script > > ("Please contact the FreeBSD Security Team for more information.") and > > we can make the `discovered' entry optional, fine with me. I can write > > a `make entry' perl script that parses a form an generates a template > > entry, send-pr like. > > FWIW, this sounds fine by me, except about the part. > I see your point about it though... it may be dangerous to have a > bogus value (like the date of entry), because it may not get corrected > later. But I don't want it optional, so that it is not forgotten. > Perhaps we need the possiblity of marking something explicitly > for such occassions ... > > In the mean time, could the date of entry be used? And perhaps a > comment could be a workaround for now, something like > > 2004-08-17 Disclaimer: I've come from the other list and am not familiar with the issues here, but this sounds like something that attributes were intended to cover. Something like: 2004-08-17 Adjust values of state depending on what the two options really are. This has the benefit of being backwards compatible, assuming that the consumers are XML parsers. Ceri -- It is not tinfoil, it is my new skin. I am a robot. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-vuxml/attachments/20040817/12a120a2/attachment.bin From trhodes at FreeBSD.org Tue Aug 17 13:39:17 2004 From: trhodes at FreeBSD.org (Tom Rhodes) Date: Tue Aug 17 13:39:19 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <853F2EE0-F086-11D8-A951-00039312D914@fillmore-labs.com> References: <20040817153510.6ccfbd8b@localhost> <853F2EE0-F086-11D8-A951-00039312D914@fillmore-labs.com> Message-ID: <20040817163818.1c307c06@localhost> On Tue, 17 Aug 2004 21:49:15 +0200 Oliver Eikemeier wrote: > Tom Rhodes wrote: > > >> Jacques doens't seem to like this: "Aaaaaahh!". I don't really care > >> ident(1) is fine for me, and it seems like this is the only reliable > >> indication. OTOH you'll need a couple of references (file, list of > >> FreeBSD versions). Doable, so when no other ideas pop up we should do > >> this. > > > > Yea, I already mentioned this. We could also stat the UPDATING > > file for the entry? Perhaps some kind of string could be checked > > with grep or something. > > UPDATING needs /src, which a) might not be available, and b) may be more > recent the the installed software. > -Oliver BLAH! The date from uname compaired to date of advisory. Shit, that only works for kernel. Hmmm -- Tom Rhodes From eikemeier at fillmore-labs.com Tue Aug 17 13:56:40 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Tue Aug 17 13:56:42 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817163818.1c307c06@localhost> Message-ID: <2EA0165A-F090-11D8-A951-00039312D914@fillmore-labs.com> Tom Rhodes wrote: > BLAH! The date from uname compaired to date of advisory. Shit, > that only works for kernel. Hmmm Conditionalizing on kernel version would be fine for me, but isn't this the date when the kernel was compiled? The alternative seems to be to bump __FreeBSD_version for security fixes, but this won't work for patches. What is the problem with ident(1)? I seems like it was invented for stuff like this. -Oliver From eikemeier at fillmore-labs.com Tue Aug 17 13:59:30 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Tue Aug 17 13:59:32 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <2EA0165A-F090-11D8-A951-00039312D914@fillmore-labs.com> Message-ID: <93C7E544-F090-11D8-A951-00039312D914@fillmore-labs.com> Oliver Eikemeier wrote: > Tom Rhodes wrote: > >> BLAH! The date from uname compaired to date of advisory. Shit, >> that only works for kernel. Hmmm > > Conditionalizing on kernel version would be fine for me, but isn't this > the date when the kernel was compiled? The alternative seems to be to > bump __FreeBSD_version for security fixes, but this won't work for > patches. What is the problem with ident(1)? I seems like it was > invented for stuff like this. OTOH `ident /usr/bin/cvs' won't help with the recent vulnerability :( From kuriyama at imgsrc.co.jp Thu Aug 19 19:26:44 2004 From: kuriyama at imgsrc.co.jp (Jun Kuriyama) Date: Thu Aug 19 19:26:47 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817175847.GC43426@madman.celabo.org> References: <20040817122453.05edaaea@localhost> <56FC3488-F075-11D8-924A-00039312D914@fillmore-labs.com> <20040817175847.GC43426@madman.celabo.org> Message-ID: <7mn00qa6ji.wl@black.imgsrc.co.jp> At Tue, 17 Aug 2004 12:58:47 -0500, Jacques A. Vidrine wrote: > In the mean time, could the date of entry be used? And perhaps a > comment could be a workaround for now, something like > > 2004-08-17 > > Ugly, I know, but the current format wasn't made for > works-in-progress. Maybe we can make some options for that... This is just an idea, but if you want markup some elements as not yet determined, that elements should be supplied with attribute (such as state="TBD") to indicate the current situation of element body text. -- Jun Kuriyama // IMG SRC, Inc. // FreeBSD Project From kuriyama at imgsrc.co.jp Thu Aug 19 19:31:55 2004 From: kuriyama at imgsrc.co.jp (Jun Kuriyama) Date: Thu Aug 19 19:31:57 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <7mn00qa6ji.wl@black.imgsrc.co.jp> References: <20040817122453.05edaaea@localhost> <56FC3488-F075-11D8-924A-00039312D914@fillmore-labs.com> <20040817175847.GC43426@madman.celabo.org> <7mn00qa6ji.wl@black.imgsrc.co.jp> Message-ID: <7mllgaa6au.wl@black.imgsrc.co.jp> At Fri, 20 Aug 2004 11:26:41 +0900, kuriyama wrote: > This is just an idea, but if you want markup some elements as not yet > determined, that elements should be supplied with attribute (such as > state="TBD") to indicate the current situation of element body text. Sorry, Ceri already proposed what I want to say. Grin. :-( -- Jun Kuriyama // IMG SRC, Inc. // FreeBSD Project From nectar at FreeBSD.org Sun Aug 22 12:29:37 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Sun Aug 22 12:29:38 2004 Subject: database tables for VuXML In-Reply-To: <41279E59.13631.76AD2AB9@localhost> References: <41279E59.13631.76AD2AB9@localhost> Message-ID: <20040822192915.GA17478@madman.celabo.org> [Added freebsd-vuxml@] On Sat, Aug 21, 2004 at 07:11:21PM -0400, Dan Langille wrote: > Hi folks, > > I'm in the early stage of incorporating VuXML information into > FreshPorts. Alright! That'll be cool. > I have a preliminary database table design which appears > below. If you think I've missed any essential information out of > this design, please let me know. > > First: has anyone already worked on creating database tables for the > VuXML data? I have an old prototype application which uses SQLite. FWIW, here is the schema I used then. (the formatting is a little funny because this was embedded in some other code) CREATE TABLE vuxml ( vid VARCHAR PRIMARY KEY UNIQUE , topic VARCHAR , description VARCHAR , discovery DATE , entry DATE , modified DATE -- computed max(entry, modified) for convenience in -- other expressions , last DATE ); CREATE TABLE refs ( vid VARCHAR , type VARCHAR , text VARCHAR ); CREATE TABLE affected ( vid VARCHAR , type VARCHAR ); CREATE TABLE names ( affected INTEGER , name VARCHAR ); CREATE TABLE ranges ( affected INTEGER , lowop VARCHAR , low VARCHAR , glow0 INTEGER , glow1 INTEGER , glow2 INTEGER , glow3 INTEGER , highop VARCHAR , high VARCHAR , ghigh0 INTEGER , ghigh1 INTEGER , ghigh2 INTEGER , ghigh3 INTEGER ); The `ranges' table perhaps looks erm interesting. In this case, I encoded each package version number into four 32-bit integers. This allowed one to lookup an affected package completely in SQL. However, the encoding I used may now be incorrect in some ways, since pkg_install et al were recently changed to treat version numbers differently. Also, SQLite could not really optimize the complex SQL statement that resulted. So I think it is interesting for further study, but I doubt I'd do it that way again unless I had a very particular need to use pure SQL or the database was huge. > create table vuxml > ( > id serial not null, > -- internal FreshPorts ID > vid text not null, > -- the vuln vid. e.g. c4b025bb-f05d-11d8-9837-000c41e2cdad > topic text not null, > description text not null, > date_discovery date , > date_entry date , > date_modified date , > status date not null, > primary key (id) > ); > > create table vuxml_port_xref > ( > id serial not null, > vuxml_id integer not null, > port_id integer not null, > primary key (id) > ); > > create table vuxml_port_xref_range > ( > id serial not null, > vuxml_port_xref_id text , > range_operator_start text , > range_operator_end text , > range_version_start text , > range_version_end text , > primary key (id) > ); > > If you consider the output from: vxquery -t vuxml ~/VuXML/vuln.xml > tnftpd, the entry in vuxml_port_xref_range for tnftpd might be: > > (1, 'lt', 20040810, NULL, NULL) Seem reasonable. For what its worth, storing the range operators as `>', `=>', etc lets you get something human readable as the output of a SELECT, e.g. SELECT range_version_start, range_operator_start, port_name, range_operator_end, range_version_end FROM ... WHERE ...; ('20040810', '<', 'tnftpd', '', '') I found that handy for manual inspection of the database. You may notice from my example that I used separate `name', `ranges', and `affected' tables. I found this easier when filling the database, since the / elements are kind of a cross-product operator. For example, when processing something like mutt ja-mutt 1.41.4.2 I first created an `affected' entry to contain the other references. Then, as I hit each `name' I just associated it with the `affected' entry, and as I hit each `range' I did the same. (Otherwise one needs to construct a few lists, and then make all entries once the end tag is seen.) I think this is also easier to modify if we get new children of , e.g. the and tags that are currently in the VuXML pre-1.2 DTD. Have fun, and please let me know if I can assist! Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Sun Aug 22 12:40:47 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Sun Aug 22 12:41:48 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <1F055B5E-F084-11D8-924A-00039312D914@fillmore-labs.com> References: <20040817185332.2B91D1800A@sirius.firepipe.net> <1F055B5E-F084-11D8-924A-00039312D914@fillmore-labs.com> Message-ID: <20040822194025.GB17478@madman.celabo.org> On Tue, Aug 17, 2004 at 09:32:05PM +0200, Oliver Eikemeier wrote: > Jacques doens't seem to like this: "Aaaaaahh!". :-) > I don't really care > ident(1) is fine for me, and it seems like this is the only reliable > indication. OTOH you'll need a couple of references (file, list of > FreeBSD versions). Doable, so when no other ideas pop up we should do > this. I don't think ident information is all that useful, and I *know* that it is a PITA to maintain. I instituted the current practice of including revision numbers in FreeBSD Security Advisories. I thought it was a good idea at the time :-) But in fact, it is one of the most time-consuming parts of producing a FreeBSD SA. From the correspondance I've received over the last few years, I'm quite certain that no one really uses them. Those who *do* use them are users who are quite capable of accurately determining the revisions on their own. In addition, the revisions are often quite misleading. Some examples are: * Some revisions just aren't important from the perspective of the security fix. src/UPDATING is an obvious case :-) but there are less obvious cases, such as result from a merge. * Often the revision numbers are not available in the resulting binary. * Patches are most often produced *without* the revision number changes, since these changes tend to get in the way of tools such as patch(1). Thus, patched systems have "old" revision numbers. The only practical way to specify affected versions of the system is with a patch level as we've done for years. e.g. 4.8-RELEASE-p9 is unaffected, all those before are not. This is analogous to the situation with ports... we use the package version number, not the revision numbers of the Makefile and associated ports skeleton files, nor the revision numbers of the original source files or anything silly like that. We use the administratively maintained package number with PORTEPOCH, PORTREVISION and such. re@ and so@ actually had a long (unfortunately incomplete) thread about this kind of thing. We'd like to be able to decouple the system version number from the kernel build, but there are lots of details. I guess we should revisit this soon, as this becomes IMHO more important with the advent of errata branches. (I expect more patch releases.) Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From eikemeier at fillmore-labs.com Sun Aug 22 13:40:49 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Sun Aug 22 13:40:51 2004 Subject: determining vulnerable FreeBSD system components [Was: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml] In-Reply-To: <20040822194025.GB17478@madman.celabo.org> Message-ID: <8D9F2B2C-F47B-11D8-8CAA-00039312D914@fillmore-labs.com> Jacques A. Vidrine wrote: > I don't think ident information is all that useful, and I *know* > that it is a PITA to maintain. Jup, I realized that the cvs binary doesn't contain enough information to be identifiable with ident(1). > [...] > The only practical way to specify affected versions of the system > is with a patch level as we've done for years. e.g. 4.8-RELEASE-p9 > is unaffected, all those before are not. This is analogous to the > situation with ports... we use the package version number, not the > revision numbers of the Makefile and associated ports skeleton files, > nor the revision numbers of the original source files or anything > silly like that. We use the administratively maintained package > number with PORTEPOCH, PORTREVISION and such. Yup. We should use __FreeBSD_version for -STABLE and -CURRENT, since this is easy determinable. I now -CURRENT is not supported, but it would be useful nevertheless. I don't know how to handle release branches though. Especially when only the affected binary is patched, without rebooting the system (and possibly bumping __FreeBSD_version). Maybe we should invent some kind of global registry where the (security) patches applied are recorded. -Oliver From nectar at FreeBSD.org Sun Aug 22 13:47:07 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Sun Aug 22 13:47:11 2004 Subject: portaudit wishlist In-Reply-To: <3AF421B2-F082-11D8-924A-00039312D914@fillmore-labs.com> References: <3AF421B2-F082-11D8-924A-00039312D914@fillmore-labs.com> Message-ID: <20040822204644.GC17478@madman.celabo.org> On Tue, Aug 17, 2004 at 09:18:33PM +0200, Oliver Eikemeier wrote: > Ok, things that I think would be really useful (incomplete list): > > - csh-style braces. When this is not the right syntax, this could be > done with > ja-bugzilla > or > ja-kr-cups > > but we have many slave ports which just differ in prefixes/suffixes, and > it would be easy to expand them when reading the file. > > Yes, portaudit does linear searches. Besides, this will greatly diminish > the size of the database. > > I'm even willing to sacrifice glob patterns `*' and `?' for that, > although they can be quite convenient sometimes. I deliberately avoided including any "mini syntax" within VuXML. Each such syntax puts an additional, perhaps insurmountable, burden on processing tools. For example, including such things makes it practically impossible to use XSLT to translate VuXML into other XML formats such as XHTML, or to build accurate XQuery/XPath expressions. A less severe case is that of a package audit tool that uses a database backend (SQL, db4, whatever) for lookups: it must first expand these csh-style constructs before entering them into the database. These things are maybe not deal killers (well, the XSLT issue is very close), but the real problem is that this simply adds complexity to the format with no benefit. The size savings is insignificant, and it is highly arguable that it is preferable to edit or read even in extreme cases such as php4 php4-cgi php4-cli php4-dtc php4-horde php4-nms mod_php4-twig 4.3.8 versus php4 php4-{cgi,cli,dtc,horde,nms} mod_php4-twig 4.3.8 > - 1.* notation as the `smallest 1.x version possible'. 1.a is not the > smallest, besides it is not completely transparent why .a is chosen in > the range. When the `*' is the problem, this could be easily changed to > a random character, or even a (greater equal range) tag (ok, > the name is silly), but I want to have some standard way like >= 1.* < > 2.* to match all 1.x and nothing else. No, I don't think >= 1.a < 2.a is > good here. I understand your motivation here, but I want to be careful about adding extraneous notation. Honestly, I do see the *convenience* of using e.g. `2.*'. It requires less thinking :-) As you well know I have made the mistake of specifying `2' when `2.a' or similar was needed, and I think that if `2.*' would have been "available" I probably would have used it instead. I assume this goes double for anyone less familiar with VuXML, and those who are maybe "copying" from another entry. I'm a little nervous about people seeing generated documentation with `*' and thinking it means something other than it means. But, as we discussed previously, I really cannot think of a better character. Maybe I'm overly concerned. So basically I think we should introduce this in VuXML 1.2. I'd like to hear some comments from others about it, especially from the point of view of the user reading content generated from VuXML. > - make `discovery' optional. It's a nice-to-have, but sometimes hard to > find out, and dummy entries like entry = discovery do not help anyone. > (ok, superseeded by another thread). > > - make `description' optional. It is in the way of `quick' entries which > should be researched later. Of course it is acceptable to fill it with a > dummy value, but in this case it shouldn't be present IMHO and the dummy > value should be provided by the rendering code. Or will an empty tag do? Let's cover both of these in the other thread, because I think they are kinda related. > - make a `severity' field available. Of course it might be inaccurate, > and software might want to ignore it and provide it's own data. Yet it > is useful when you only have time for a quick glance (notify me > immediately of severe vulnerabilities, all others should only appear in > fridays report). It is a valuable guidance for the users, although I'm > aware it is very error-prone. This is a policy issue, and does not belong in the FreeBSD VuXML document. I think adjunct documents/database for that purpose are great. If some were available and well-maintained, I would for example want to provide a sidebar on www.vuxml.org for each vulnerability showing the ratings. We've already discussed this in the thread which includes the message http://lists.freebsd.org/pipermail/cvs-ports/2004-March/031704.html. I'm not sure it is useful to go over the same ground again. I think it is likely you will say that the Security Officer should assign some severity. To pre-respond :-) I'll say that the security team's perspective is that either (a) you understand the security issue, in which case you can decide whether it is a risk for you to run the software or not; or (b) you don't understand the security issue, in which case you should not run the software. That said, I wouldn't rule out one day publishing an adjunct document with some coarse-grained ratings. But I wouldn't claim that it was the final word on severity on risk. In any case, I'm not prepared to do that now, and neither are most of my colleagues doing this kind of security work for other operating systems/ distributions/ what have you. In fact, I think it is far more likely that we will wait until a certain well-known organization completes their risk assessment system, and use that as an adjunct. > - add a classification into remote/local exploitable On the one hand, I feel this should be handled in the narrative description and that it is otherwise just another facet of the severity rating. On the other hand, I can imagine someone deciding that it was OK e.g. to install any ports that did not have a *remotely* exploitable vulnerability. My instinct tells me this too should be adjunct, at least for now. Why would we include remotely/locally, but not authenticated/unauthenticated, application privileges/system privileges, user privileges/superuser privileges, or other such aspects? If you have a server with a vulnerability that lets someone who has a local account to get some other access, would you record that as local or remote? Yes, I think it is misleading to apply such tags which a user might take as an absolute judgement when in fact they just need to read the description. > - add a `fixed' field that lists a version where the vulnerability is > fixed. This could be used for a recommendation message, like "upgrade to > version xxx" or "no upgrade is available, please deinstall the port or > proceed with caution". > This could also realized as an alternate tag. I guess I don't understand this request. That is the purpose of the element and children. > - Also we should add tags for the most popular references. VuXML 1.2 will include OSVDB (and almost any other you care to suggest right now). In addition, the element will be parametized so that new reference types can be added `between' VuXML DTD updates. VuXML tools are expected to render "unknown" reference types in the obvious fashion, e.g. old tools might render http://archive.site/... in a references table as mlist http://archive.site/... With parameterization, adding a FooDB element while remaining valid is accomplished with an internal DTD subset in the usual fashion, e.g. ] > Speaking of > references, I would prefer something like CVS Multiple > Vulnerabilities, which means they canbe rendered with a meaningful > line (but most not, so is legal too). I have two problems with this suggestion, both of which may be fixable: (a) Important stuff should be element content, and not attributes. [1] (b) When rendering a Bugtraq bug ID, the bug ID *is* the meaningful thing and must be displayed in a "meaningful line". The title or description of the bug ID is meta-data. A modification of your proposal that wouldn't have these problems would be to add a descriptive attribute that can be used on any child of , analogous to the XHTML "alt" attribute found on the element e.g. 10499 Such descriptions could be rendered inline, as footnotes, or as popups (e.g. XHTML hover). However, I really don't see this providing much value, especially considering that the vast majority of such references will not be filled in. Your example might be particularly poor. Assuming that we're talking about an entry with a like "cvs multiple vulnerabilities", what the heck else would we point at in ? :-) I'm sure you can find a few good examples, but I imagine those will be few and stretched. The whole point of VuXML is to give enough information but not too much :-) ``Too much'' is anything that is not likely to be supplied in the vast majority of cases. > Ok, too many threads now. I have too look into this a little closer. Thanks, Oliver! Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org [1] The element uses attribute data for what might be considered "important stuff" (msgid, the message ID). However, this is to keep within a "meta content model" for , namely the value model so that VuXML tools can be dumb and just output or search (type, value) tuples. If it weren't for this, I probably would have specified to have two children, and . On the other hand, the msgid attribute is a kind of meta-information, so maybe it is not so bad :-) From eikemeier at fillmore-labs.com Sun Aug 22 13:53:26 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Sun Aug 22 13:53:45 2004 Subject: database tables for VuXML In-Reply-To: <20040822192915.GA17478@madman.celabo.org> Message-ID: <50BEFF1D-F47D-11D8-8CAA-00039312D914@fillmore-labs.com> Jacques A. Vidrine wrote: > The `ranges' table perhaps looks erm interesting. In this case, > I encoded each package version number into four 32-bit integers. > This allowed one to lookup an affected package completely in SQL. > However, the encoding I used may now be incorrect in some ways, since > pkg_install et al were recently changed to treat version numbers > differently. I can just donate some code that converts version numbers into arbitrary precision floting point numbers between zero and one. An alternate (and maybe easier to handle) form are hex strings which can simply compared with strcmp. >> description text not null, FWIIW, I would prefer to have a null description that is replaced by some dummy text when rendering. >> date_discovery date , That same here: don't assume a valid date (or map non-existing dates to 1970-1-1). The reason is that I want to have a form of `quick' entries, which are not fully reviewed. > I think this is also easier to > modify if we get new children of , e.g. the and > tags that are currently in the VuXML pre-1.2 DTD. Ehm, is the DTD somewhere available form review? And how about the csh-style braces / tags suggested in ? -Oliver From nectar at FreeBSD.org Sun Aug 22 13:54:51 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Sun Aug 22 13:55:55 2004 Subject: determining vulnerable FreeBSD system components [Was: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml] In-Reply-To: <8D9F2B2C-F47B-11D8-8CAA-00039312D914@fillmore-labs.com> References: <20040822194025.GB17478@madman.celabo.org> <8D9F2B2C-F47B-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <20040822205430.GD17478@madman.celabo.org> On Sun, Aug 22, 2004 at 10:40:50PM +0200, Oliver Eikemeier wrote: > Yup. We should use __FreeBSD_version for -STABLE and -CURRENT, since > this is easy determinable. __FreeBSD_version is not and should not be bumped for security updates. It is strictly for source (and perhaps in some cases, binary) code compatibility, and security updates do not (should not) impact code compatibility. > I now -CURRENT is not supported, but it would > be useful nevertheless. I don't know how to handle release branches > though. Especially when only the affected binary is patched, without > rebooting the system (and possibly bumping __FreeBSD_version). Maybe we > should invent some kind of global registry where the (security) patches > applied are recorded. Yeah, that has also come up before. Perhaps we should pick it up again. Also, this kinda relates to Julian's desire to have the advisories in the source tree, so that when you checked out say RELENG_4_10, you would get all the advisories that affected 4.10 (and ONLY those advisories). That could of course work for -STABLE and -CURRENT as well, but IIRC there were some objections due to the realities of how we manage the source tree. For example, I would not like to need to have N different advisories for N different branches (i.e. branching the advisory in CVS), but re@ has reasons they do not want to allow the sliding of tags within src/. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Sun Aug 22 14:32:57 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Sun Aug 22 14:34:22 2004 Subject: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml In-Reply-To: <20040817175847.GC43426@madman.celabo.org> References: <20040817122453.05edaaea@localhost> <56FC3488-F075-11D8-924A-00039312D914@fillmore-labs.com> <20040817175847.GC43426@madman.celabo.org> Message-ID: <20040822213232.GE17478@madman.celabo.org> On Tue, Aug 17, 2004 at 12:58:47PM -0500, Jacques A. Vidrine wrote: > [Moving to freebsd-vuxml ... oh how I wish Bcc worked so that people on > the other list knew where this went :-) ] > > On Tue, Aug 17, 2004 at 07:46:16PM +0200, Oliver Eikemeier wrote: > > When you can live with the dummy text produced by my perl script > > ("Please contact the FreeBSD Security Team for more information.") and > > we can make the `discovered' entry optional, fine with me. I can write > > a `make entry' perl script that parses a form an generates a template > > entry, send-pr like. > > FWIW, this sounds fine by me, except about the part. > I see your point about it though... it may be dangerous to have a > bogus value (like the date of entry), because it may not get corrected > later. But I don't want it optional, so that it is not forgotten. > Perhaps we need the possiblity of marking something explicitly > for such occassions ... OK, so this has been in the back of my mind for the past few days, and I feel pretty strongly about requiring certain portions of the VuXML entry. During the development of the DTD, it was basically decided that in order to be useful, each entry *must* provide the following information: (I'm repeating some of what is in the DTD in English prose here :-) - A "one-liner" - What is . (If nothing is affected, it shouldn't be included.) - A brief or even incredibly rich of the problem, including details specific to the particular packaging system or vendor. Quotes of other security advisories are fine. - At least one entry in . It is highly recommended that a CVE name be included, but this is not always possible. There should certainly at least be a public email or source file to which to point. - The date the issue was first disclosed (this was possibly mis-named ). - The date of this issue into the document So in this thread and another, Oliver has requested that and be made optional. I understand that this is due to a desire to be able to make "quick" entries. But I have to wonder, how does this really help? I feel very strongly that a must be required. If one cannot provide even a quote from some other source, then one has not properly researched the issue. It *is* possible, of course, to specify a description like

Description not yet available.

or even

and still have a valid VuXML document, but this is certainly not within the spirit of even "quick" documentation. So, as an editor, I wouldn't prohibit such entries, just frown on them :-) I mean, if one has the single reference required, then one certainly has something to quote. I feel less strongly about the element (as mentioned in my earlier message quoted above). But still, after reflection I do not think that it should be optional. I routinely set this to be the earliest public notice that I've found when looking for references. I have never felt that it was difficult to decide. In my case, I have to be a little more careful because I don't want to include a date earlier than any public reference (even if I was included in private discussion many weeks earlier). But most people don't have to deal with that issue. Finally, if an earlier reference eventually turns up, the date can be modified, no big deal. However, I must admit that I have some doubt the value of the date in any case. What I'd really like to hear are some arguments for keeping it or getting rid of it! I think it is useful information of itself to many reading VuXML content, and that combined with it provides a good metric about our response time. But I could be overestimating the value of it, and if it somehow puts people off to need to provide this information, then maybe it loses. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Sun Aug 22 14:33:25 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Sun Aug 22 14:35:37 2004 Subject: database tables for VuXML In-Reply-To: <50BEFF1D-F47D-11D8-8CAA-00039312D914@fillmore-labs.com> References: <20040822192915.GA17478@madman.celabo.org> <50BEFF1D-F47D-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <20040822213304.GF17478@madman.celabo.org> On Sun, Aug 22, 2004 at 10:53:26PM +0200, Oliver Eikemeier wrote: > Jacques A. Vidrine wrote: > > >The `ranges' table perhaps looks erm interesting. In this case, > >I encoded each package version number into four 32-bit integers. > >This allowed one to lookup an affected package completely in SQL. > >However, the encoding I used may now be incorrect in some ways, since > >pkg_install et al were recently changed to treat version numbers > >differently. > > I can just donate some code that converts version numbers into arbitrary > precision floting point numbers between zero and one. Sure, that's great if it covers all cases! Floating point is limited to something like 64 bits, and I found version numbers that required almost 96 bits in the encoding I used (which I believe to be pretty tight). > An alternate (and > maybe easier to handle) form are hex strings which can simply compared > with strcmp. Definitely easier to handle (assuming everything is padded correctly). I found using integer comparisons was quite a bit faster, but this could be a micro-optimization. Heck, I'm not convinced that the whole exercise of encoding version strings is not a micro-optimization :-) but it is fun and interesting. > >> description text not null, > > FWIIW, I would prefer to have a null description that is replaced by > some dummy text when rendering. > > >> date_discovery date , > > That same here: don't assume a valid date (or map non-existing dates to > 1970-1-1). The reason is that I want to have a form of `quick' entries, > which are not fully reviewed. The current version of VuXML does not allow these to be null, and I it is probably good to reflect those constraints in the database. > >I think this is also easier to > >modify if we get new children of , e.g. the and > > tags that are currently in the VuXML pre-1.2 DTD. > > Ehm, is the DTD somewhere available form review? And how about the > csh-style braces / tags suggested in > > ? http://www.vuxml.org/dtd/vuxml-1/vuxml-pre12.dtd http://www.vuxml.org/dtd/vuxml-1/vuxml-model-pre12.mod The is directly from the previous thread here, while was recently added due to a conversation from last week that I thought was here ... but I don't see it in the archives, so I'm clearly mistaken about that. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From eikemeier at fillmore-labs.com Sun Aug 22 14:39:48 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Sun Aug 22 14:40:57 2004 Subject: portaudit wishlist In-Reply-To: <20040822204644.GC17478@madman.celabo.org> Message-ID: Jacques A. Vidrine wrote: > On Tue, Aug 17, 2004 at 09:18:33PM +0200, Oliver Eikemeier wrote: >> Ok, things that I think would be really useful (incomplete list): >> >> - csh-style braces. When this is not the right syntax, this could be >> done with >> ja-bugzilla >> or >> ja-kr-cups >> >> but we have many slave ports which just differ in prefixes/suffixes, >> and >> it would be easy to expand them when reading the file. >> >> Yes, portaudit does linear searches. Besides, this will greatly >> diminish >> the size of the database. >> >> I'm even willing to sacrifice glob patterns `*' and `?' for that, >> although they can be quite convenient sometimes. > > I deliberately avoided including any "mini syntax" within VuXML. > Each such syntax puts an additional, perhaps insurmountable, burden > on processing tools. For example, including such things makes it > practically impossible to use XSLT to translate VuXML into other XML > formats such as XHTML, or to build accurate XQuery/XPath expressions. > A less severe case is that of a package audit tool that uses a > database backend (SQL, db4, whatever) for lookups: it must first > expand these csh-style constructs before entering them into the > database. > > These things are maybe not deal killers (well, the XSLT issue is very > close), but the real problem is that this simply adds complexity to the > format with no benefit. The size savings is insignificant, and it is > highly arguable that it is preferable to edit or read even in extreme > cases such as > > > php4 > php4-cgi > php4-cli > php4-dtc > php4-horde > php4-nms > mod_php4-twig > 4.3.8 > > > versus > > > php4 > php4-{cgi,cli,dtc,horde,nms} > mod_php4-twig > 4.3.8 > It should be trivial to deal with the tag in XSLT, the same should be possible with , and for entering them into databases you have to render the stuff anyway. Readability of the XML code is a non-issue, since it is designed to be machine-readable, not human-readable. portaudit is designed to be lightweight and work without a database, so it does linear searches on all systems. I might change that, but that's the way things currently are, so what's the problem here? >> - 1.* notation as the `smallest 1.x version possible'. 1.a is not the >> smallest, besides it is not completely transparent why .a is chosen in >> the range. When the `*' is the problem, this could be easily changed to >> a random character, or even a (greater equal range) tag >> (ok, >> the name is silly), but I want to have some standard way like >= 1.* < >> 2.* to match all 1.x and nothing else. No, I don't think >= 1.a < 2.a >> is >> good here. > > I understand your motivation here, but I want to be careful about > adding extraneous notation. Honestly, I do see the *convenience* of > using e.g. `2.*'. It requires less thinking :-) As you well know I > have made the mistake of specifying `2' when `2.a' or similar was > needed, and I think that if `2.*' would have been "available" I > probably would have used it instead. I assume this goes double for > anyone less familiar with VuXML, and those who are maybe "copying" from > another entry. > > I'm a little nervous about people seeing generated documentation with > `*' and thinking it means something other than it means. But, as we > discussed previously, I really cannot think of a better character. > Maybe I'm overly concerned. As stated before: The vuxml entry doesn't have to use `*', nor do you have to use it in the rendered version. It is perfectly legal to render this as `>= 2.x' with a cursive x. We should just have _something_. If you prefer, we can even use an instead of and or something similar that clarifies truncation is used. Anyway, the way things currently are works for me, and avoids bugs. > So basically I think we should introduce this in VuXML 1.2. I'd like > to hear some comments from others about it, especially from the point > of view of the user reading content generated from VuXML. Uhm, you don't need vuxml 1.2 for that, this is perfectly legal in vuxml since 1.0. >> - make a `severity' field available. Of course it might be inaccurate, >> and software might want to ignore it and provide it's own data. Yet it >> is useful when you only have time for a quick glance (notify me >> immediately of severe vulnerabilities, all others should only appear in >> fridays report). It is a valuable guidance for the users, although I'm >> aware it is very error-prone. > > This is a policy issue, and does not belong in the FreeBSD VuXML > document. I think adjunct documents/database for that purpose are > great. If some were available and well-maintained, I would for example > want to provide a sidebar on www.vuxml.org for each vulnerability > showing the ratings. > > We've already discussed this in the thread which includes the message > http://lists.freebsd.org/pipermail/cvs-ports/2004-March/031704.html. > I'm not sure it is useful to go over the same ground again. > > I think it is likely you will say that the Security Officer should > assign some severity. To pre-respond :-) I'll say that the security > team's perspective is that either (a) you understand the security > issue, in which case you can decide whether it is a risk for you to > run the software or not; or (b) you don't understand the security > issue, in which case you should not run the software. > > That said, I wouldn't rule out one day publishing an adjunct document > with some coarse-grained ratings. But I wouldn't claim that it was > the final word on severity on risk. In any case, I'm not prepared to > do that now, and neither are most of my colleagues doing this kind of > security work for other operating systems/ distributions/ what have > you. In fact, I think it is far more likely that we will wait until > a certain well-known organization completes their risk assessment > system, and use that as an adjunct. I can't really follow your rationale here. You claim that because it can't be done perfectly, it shouldn't be done at all. I would find it incredibly useful to get some categorization into `dangerous', `important' and `minor', even when it's wrong sometimes. As discussed before: You usually have a pretty good idea whether a vulnerability is severe or not, you just don't want to tell anybody. I consider this valuable to give users a chance to customize the notification scheme of portaudit, and why shouldn't this information find its place in the vuxml database? Make this tag optional, so you don't have to fill in anything when you on't feel like it. >> - add a classification into remote/local exploitable > > On the one hand, I feel this should be handled in the narrative > description and that it is otherwise just another facet of the > severity rating. On the other hand, I can imagine someone deciding > that it was OK e.g. to install any ports that did not have a > *remotely* exploitable vulnerability. > > My instinct tells me this too should be adjunct, at least > for now. Why would we include remotely/locally, but not > authenticated/unauthenticated, application privileges/system > privileges, user privileges/superuser privileges, or other such > aspects? If you have a server with a vulnerability that lets someone > who has a local account to get some other access, would you record > that as local or remote? umh, local of course? > Yes, I think it is misleading to apply such tags which a user might > take as an absolute judgement when in fact they just need to read the > description. Not everyone has the time to review every description. Besides, the description might be as wrong or misleading as the tags mentioned. If you say "users have to understand the system fully or they shouldn't run the software" you basically state "FreeBSD is only for experts". I'm just trying to make some often asked questions machine readable. For example when I run portaudit on a server with no users, I might decide to care for local exploitable vulnerabilities only ever friday, while I have to handle remote exploitable vulnerabilities immediately. This system is not perfect, but usable. You give users basically no way to filter the information, which would be a valuable feature. One one hand you state users have to be knowledgeable to run a system, one the other you claim they might take tags `as an absolute judgement'. In this case reading the (possibly wrong) description might not improve anything. >> - add a `fixed' field that lists a version where the vulnerability is >> fixed. This could be used for a recommendation message, like "upgrade >> to >> version xxx" or "no upgrade is available, please deinstall the port or >> proceed with caution". >> This could also realized as an alternate tag. > > I guess I don't understand this request. That is the purpose of the > element and children. There is no information whether there is an update available (and since which date the vulnerability is fixed), or if I simply have to deinstall the software and use a different product. >> Speaking of >> references, I would prefer something like CVS Multiple >> Vulnerabilities, which means they canbe rendered with a >> meaningful >> line (but most not, so is legal too). > > I have two problems with this suggestion, both of which may be fixable: > > (a) Important stuff should be element content, and not attributes. [1] > (b) When rendering a Bugtraq bug ID, the bug ID *is* the meaningful > thing and must be displayed in a "meaningful line". The title or > description of the bug ID is meta-data. > > A modification of your proposal that wouldn't have these problems > would be to add a descriptive attribute that can be used on any child of > , analogous to the XHTML "alt" attribute found on the > element e.g. > > 10499 > > Such descriptions could be rendered inline, as footnotes, or as popups > (e.g. XHTML hover). Ok, since we have to be backwards compatible, let's do it that way. > However, I really don't see this providing much value, especially > considering that the vast majority of such references will not be > filled in. Your example might be particularly poor. Assuming > that we're talking about an entry with a like "cvs > multiple vulnerabilities", what the heck else would we point at in > ? :-) I'm sure you can find a few good examples, but I > imagine those will be few and stretched. xfdb does it that way, and I like it. It's especially useful for mailing list posts, to see whether they contain an advisory or an exploit, for entries that cover multiple vulnerabilities (to distinguish the CVE references) and to filter out those `Updated packages fix multiple vulnerabilities' references for other operating systems. > The whole point of VuXML is to give enough information but not too > much :-) ``Too much'' is anything that is not likely to be supplied in > the vast majority of cases. This could be automated by the tool that makes entries, or even by a tool that adds missing descriptions, so it is likely to be supplied. -Oliver From eikemeier at fillmore-labs.com Sun Aug 22 14:56:42 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Sun Aug 22 14:57:38 2004 Subject: making optional [Was: cvs commit: ports/security/portaudit-db/database portaudit.txt portaudit.xlist portaudit.xml] In-Reply-To: <20040822213232.GE17478@madman.celabo.org> Message-ID: <272AEBD2-F486-11D8-8CAA-00039312D914@fillmore-labs.com> Jacques A. Vidrine wrote: > On Tue, Aug 17, 2004 at 12:58:47PM -0500, Jacques A. Vidrine wrote: >> [Moving to freebsd-vuxml ... oh how I wish Bcc worked so that people on >> the other list knew where this went :-) ] >> >> On Tue, Aug 17, 2004 at 07:46:16PM +0200, Oliver Eikemeier wrote: >>> When you can live with the dummy text produced by my perl script >>> ("Please contact the FreeBSD Security Team for more information.") and >>> we can make the `discovered' entry optional, fine with me. I can write >>> a `make entry' perl script that parses a form an generates a template >>> entry, send-pr like. >> >> FWIW, this sounds fine by me, except about the part. >> I see your point about it though... it may be dangerous to have a >> bogus value (like the date of entry), because it may not get corrected >> later. But I don't want it optional, so that it is not forgotten. >> Perhaps we need the possiblity of marking something explicitly >> for such occassions ... > > OK, so this has been in the back of my mind for the past few days, and > I feel pretty strongly about requiring certain portions of the VuXML > entry. During the development of the DTD, it was basically decided > that in order to be useful, each entry *must* provide the following > information: > > (I'm repeating some of what is in the DTD in English prose here :-) > > - A "one-liner" > - What is . (If nothing is affected, it shouldn't be > included.) > - A brief or even incredibly rich of the problem, > including details specific to the particular packaging system or > vendor. Quotes of other security advisories are fine. > - At least one entry in . It is highly recommended that > a CVE name be included, but this is not always possible. There > should certainly at least be a public email or source file to > which to point. > - The date the issue was first disclosed (this was possibly > mis-named ). > - The date of this issue into the document > > So in this thread and another, Oliver has requested that > and be made optional. I understand that this is due to > a desire to be able to make "quick" entries. But I have to wonder, > how does this really help? I feel very strongly that a > must be required. If one cannot provide even a quote from some other > source, then one has not properly researched the issue. It *is* > possible, of course, to specify a description like > > > >

Description not yet available.

> > > > or even > > >

> > > and still have a valid VuXML document, but this is certainly not > within the spirit of even "quick" documentation. So, as an editor, I > wouldn't prohibit such entries, just frown on them :-) I mean, if one > has the single reference required, then one certainly has something to > quote. 60 (in words: sixty) entries in portaudit have the description `Please contact the FreeBSD Security Team for more information'. There are references, so when you care to add a quote, feel free, in fact this might be a job for the security team. You can frown on them as often as you like, the question is whether you just want to have an optional entry as an easy to spot sign that an editor is needed, or if you prefer to search for

and similar constructs. > I feel less strongly about the element (as mentioned in > my earlier message quoted above). But still, after reflection I do > not think that it should be optional. I routinely set this to be the > earliest public notice that I've found when looking for references. I > have never felt that it was difficult to decide. In my case, I have > to be a little more careful because I don't want to include a date > earlier than any public reference (even if I was included in private > discussion many weeks earlier). But most people don't have to deal > with that issue. Finally, if an earlier reference eventually turns > up, the date can be modified, no big deal. > > However, I must admit that I have some doubt the value of the > date in any case. What I'd really like to hear are some > arguments for keeping it or getting rid of it! I think it is useful > information of itself to many reading VuXML content, and that combined > with it provides a good metric about our response time. But I > could be overestimating the value of it, and if it somehow puts people > off to need to provide this information, then maybe it loses. Oviously we have a different opinion what is useful here. I expect most users to be simple consumers, not security researchers. They need information about the serverity of a vulnerability, and maybe remote/local exploitability, whoever cares about the discovery date could check the references. Often I find the discovery date entertaining, but not useful. -Oliver From eikemeier at fillmore-labs.com Sun Aug 22 15:09:23 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Sun Aug 22 15:09:26 2004 Subject: database tables for VuXML In-Reply-To: <20040822213304.GF17478@madman.celabo.org> Message-ID: Jacques A. Vidrine wrote: > On Sun, Aug 22, 2004 at 10:53:26PM +0200, Oliver Eikemeier wrote: >> Jacques A. Vidrine wrote: >> >>> The `ranges' table perhaps looks erm interesting. In this case, >>> I encoded each package version number into four 32-bit integers. >>> This allowed one to lookup an affected package completely in SQL. >>> However, the encoding I used may now be incorrect in some ways, since >>> pkg_install et al were recently changed to treat version numbers >>> differently. >> >> I can just donate some code that converts version numbers into >> arbitrary >> precision floting point numbers between zero and one. > > Sure, that's great if it covers all cases! Floating point is limited > to something like 64 bits, and I found version numbers that required > almost 96 bits in the encoding I used (which I believe to be pretty > tight). > >> An alternate (and >> maybe easier to handle) form are hex strings which can simply compared >> with strcmp. > > Definitely easier to handle (assuming everything is padded correctly). > I found using integer comparisons was quite a bit faster, but this > could be a micro-optimization. Heck, I'm not convinced that the whole > exercise of encoding version strings is not a micro-optimization :-) > but it is fun and interesting. Ok, I check if my prototypes still work. I'm not sure about the value of encoding version string (especially since you have to reencode everything again in case we make some additions), but if you have applications for it I won't be in the way. >>>> description text not null, >> >> FWIIW, I would prefer to have a null description that is replaced by >> some dummy text when rendering. >> >>>> date_discovery date , >> >> That same here: don't assume a valid date (or map non-existing dates to >> 1970-1-1). The reason is that I want to have a form of `quick' entries, >> which are not fully reviewed. > > The current version of VuXML does not allow these to be null, and I it > is probably good to reflect those constraints in the database. Which means you possibly have to change the database description when upgrading. >>> I think this is also easier to >>> modify if we get new children of , e.g. the and >>> tags that are currently in the VuXML pre-1.2 DTD. >> >> Ehm, is the DTD somewhere available form review? And how about the >> csh-style braces / tags suggested in >> >> ? > > http://www.vuxml.org/dtd/vuxml-1/vuxml-pre12.dtd > http://www.vuxml.org/dtd/vuxml-1/vuxml-model-pre12.mod > > The is directly from the previous thread here, while > was recently added due to a conversation from last week > that I thought was here ... but I don't see it in the archives, so I'm > clearly mistaken about that. should be stuff like i386? I'm not sure that there are many applications for that. Usually the description is sufficient for those purposes. Or did you have something different in mind? -Olver From dan at langille.org Sun Aug 22 17:50:22 2004 From: dan at langille.org (Dan Langille) Date: Mon Aug 23 05:26:34 2004 Subject: database tables for VuXML In-Reply-To: <20040822192915.GA17478@madman.celabo.org> References: <41279E59.13631.76AD2AB9@localhost> Message-ID: <4129070D.28233.7C2E4023@localhost> On 22 Aug 2004 at 14:29, Jacques A. Vidrine wrote: > [Added freebsd-vuxml@] > > On Sat, Aug 21, 2004 at 07:11:21PM -0400, Dan Langille wrote: > > Hi folks, > > > > I'm in the early stage of incorporating VuXML information into > > FreshPorts. > > Alright! That'll be cool. A thought which came to me overnight: does anyone have suggestions as to how existing FreshPorts pages should be modified to include VuXML messages? Please feel free to take an existing FreshPorts page, do a file-save-as, and alter the HTML to suit. Then give us the URL where we can find your version. My idea: Take http://www.freshports.org/sysutils/bacula/ as an example. I think adding a special icon to the left hand colum under Commit History. Alternatives welcome. I've not had time to read the replies to my original message yet. -- Dan Langille : http://www.langille.org/ From nectar at FreeBSD.org Mon Aug 23 07:10:13 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Mon Aug 23 07:11:10 2004 Subject: portaudit wishlist In-Reply-To: References: <20040822204644.GC17478@madman.celabo.org> Message-ID: <20040823140949.GL27355@madman.celabo.org> On Sun, Aug 22, 2004 at 11:39:47PM +0200, Oliver Eikemeier wrote: > It should be trivial to deal with the tag in XSLT, the same > should be possible with , and for entering them into databases > you have to render the stuff anyway. One of your concerns was bloat, and I think using / like this would be just as much bloat. The only redeeming quality of the csh-style syntax was brevity (but as I said, I don't think that's a sufficient reason to use it). > Readability of the XML code is a > non-issue, since it is designed to be machine-readable, not > human-readable. No, that's really not correct. XML is not necessarily user-readable, but it may certainly be human-readable. Or do you think that all our doc committers that work on the web site or hand book and so forth work in "write-only" mode? (rhetorical question) > portaudit is designed to be lightweight and work without > a database, so it does linear searches on all systems. I might change > that, but that's the way things currently are, so what's the problem > here? I don't know, what *is* the problem here? What problem are you trying to solve with this new syntax? It doesn't seem to be readability (the csh syntax and / elements are both worse for readability). You mentioned space, but I don't believe that's an issue, and you seem to not either. > As stated before: The vuxml entry doesn't have to use `*', nor do you > have to use it in the rendered version. It is perfectly legal to render > this as `>= 2.x' with a cursive x. Oh, that's a nice idea. > We should just have _something_. If > you prefer, we can even use an instead of and > or something similar that clarifies truncation is used. Anyway, the way > things currently are works for me, and avoids bugs. Yeah, I prefer your original proposal. > >So basically I think we should introduce this in VuXML 1.2. I'd like > >to hear some comments from others about it, especially from the point > >of view of the user reading content generated from VuXML. > > Uhm, you don't need vuxml 1.2 for that, this is perfectly legal in vuxml > since 1.0. No, it is not. You can of course have a *valid* VuXML document with `1.*' or even `@#$@% mary had a little lamb' as content for the child elements, but that doesn't make it a legal or proper VuXML document. The DTD is only part of the specification. The other part is described in the DTD comments. > >>- make a `severity' field available. Of course it might be inaccurate, > >>and software might want to ignore it and provide it's own data. Yet it > >>is useful when you only have time for a quick glance (notify me > >>immediately of severe vulnerabilities, all others should only appear in > >>fridays report). It is a valuable guidance for the users, although I'm > >>aware it is very error-prone. > > > >This is a policy issue, and does not belong in the FreeBSD VuXML > >document. I think adjunct documents/database for that purpose are > >great. If some were available and well-maintained, I would for example > >want to provide a sidebar on www.vuxml.org for each vulnerability > >showing the ratings. > > > >We've already discussed this in the thread which includes the message > >http://lists.freebsd.org/pipermail/cvs-ports/2004-March/031704.html. > >I'm not sure it is useful to go over the same ground again. > > > >I think it is likely you will say that the Security Officer should > >assign some severity. To pre-respond :-) I'll say that the security > >team's perspective is that either (a) you understand the security > >issue, in which case you can decide whether it is a risk for you to > >run the software or not; or (b) you don't understand the security > >issue, in which case you should not run the software. > > > >That said, I wouldn't rule out one day publishing an adjunct document > >with some coarse-grained ratings. But I wouldn't claim that it was > >the final word on severity on risk. In any case, I'm not prepared to > >do that now, and neither are most of my colleagues doing this kind of > >security work for other operating systems/ distributions/ what have > >you. In fact, I think it is far more likely that we will wait until > >a certain well-known organization completes their risk assessment > >system, and use that as an adjunct. > > I can't really follow your rationale here. You claim that because it > can't be done perfectly, it shouldn't be done at all. I don't think you read and understood what I wrote. Let me try to be clearer: the security team does not currently assign severity to security advisories or VuXML entries, and has no plans to do so. Rather, we believe that all of these security notices should be acted upon. > I would find it > incredibly useful to get some categorization into `dangerous', > `important' and `minor', even when it's wrong sometimes. As discussed > before: You usually have a pretty good idea whether a vulnerability is > severe or not, you just don't want to tell anybody. > > I consider this valuable to give users a chance to customize the > notification scheme of portaudit, and why shouldn't this information > find its place in the vuxml database? Make this tag optional, so you > don't have to fill in anything when you on't feel like it. I think that any severity rating system should be seperate from the VuXML documentation to allow for multiple such systems and policies, particularly as practices in this area evolve. > >>- add a classification into remote/local exploitable > > > >On the one hand, I feel this should be handled in the narrative > >description and that it is otherwise just another facet of the > >severity rating. On the other hand, I can imagine someone deciding > >that it was OK e.g. to install any ports that did not have a > >*remotely* exploitable vulnerability. > > > >My instinct tells me this too should be adjunct, at least > >for now. Why would we include remotely/locally, but not > >authenticated/unauthenticated, application privileges/system > >privileges, user privileges/superuser privileges, or other such > >aspects? If you have a server with a vulnerability that lets someone > >who has a local account to get some other access, would you record > >that as local or remote? > > umh, local of course? Funny you should say `of course'. I would have classified it as remote. I picked this example from some correspondence involving several colleagues, and guess what: the answers were mixed. Which is `of course' my point. > >Yes, I think it is misleading to apply such tags which a user might > >take as an absolute judgement when in fact they just need to read the > >description. > > Not everyone has the time to review every description. Besides, the > description might be as wrong or misleading as the tags mentioned. If > you say "users have to understand the system fully or they shouldn't run > the software" you basically state "FreeBSD is only for experts". I'm > just trying to make some often asked questions machine readable. For > example when I run portaudit on a server with no users, I might decide > to care for local exploitable vulnerabilities only ever friday, while I > have to handle remote exploitable vulnerabilities immediately. This > system is not perfect, but usable. You give users basically no way to > filter the information, which would be a valuable feature. One one hand > you state users have to be knowledgeable to run a system, one the other > you claim they might take tags `as an absolute judgement'. In this case > reading the (possibly wrong) description might not improve anything. Your ``reasoning'' makes me dizzy. Look Oliver, knock yourself out: come up with your own severity rating scheme and implement it. Stop bugging the security team to do it, I've already explained that we will not at this time. > >>- add a `fixed' field that lists a version where the vulnerability is > >>fixed. This could be used for a recommendation message, like "upgrade > >>to > >>version xxx" or "no upgrade is available, please deinstall the port or > >>proceed with caution". > >>This could also realized as an alternate tag. > > > >I guess I don't understand this request. That is the purpose of the > > element and children. > > There is no information whether there is an update available (and since > which date the vulnerability is fixed), or if I simply have to deinstall > the software and use a different product. Maybe you can describe this a bit more fully? (BTW, Can you make a separate thread for each of your proposals?) I felt that the affected ranges were sufficient, but if there is a better way let's hear it. [... snip: adding `description's to each reference ...] > xfdb does it that way, and I like it. It's especially useful for mailing > list posts, to see whether they contain an advisory or an exploit, for > entries that cover multiple vulnerabilities (to distinguish the CVE > references) and to filter out those `Updated packages fix multiple > vulnerabilities' references for other operating systems. > > >The whole point of VuXML is to give enough information but not too > >much :-) ``Too much'' is anything that is not likely to be supplied in > >the vast majority of cases. > > This could be automated by the tool that makes entries, or even by a > tool that adds missing descriptions, so it is likely to be supplied. Well, I'm not sure I'm convinced. But let's try an example. CAN-2004-0691 CAN-2004-0692 CAN-2004-0693 http://www.trolltech.com/developer/changes/changes-3.3.3.html http://scary.beasts.org/security/CESA-2004-004.txt Is this the kind of thing you mean? I wonder if it will really be used much *shrug* (by authors or users). I'd like to know what you mean by `a tool that adds missing descriptions'. Are you thinking of following the references and snarfing the or similar? That might be neat. But, it would create two different meanings for this attribute: (a) editorial description, i.e. to disambiguate multiple similar references to CVE names, source code, whatever; (b) the referenced item's description, e.g. <title> for HTML documents, Subject: for email. I'm not sure I like that. As you seem to point out, (b) could be largely automated, so I'm not sure it is worth storing it in the document. Then when rendering a document, how will a reader know that what she sees is (a) or (b)? One more thing: I suggested `description' as the attribute name, but I have second thoughts. I believe it might be better to pick something that is not also an element name, and perhaps `description' is a little too much typing. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Mon Aug 23 07:18:25 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Mon Aug 23 07:19:33 2004 Subject: making <description> optional In-Reply-To: <272AEBD2-F486-11D8-8CAA-00039312D914@fillmore-labs.com> References: <20040822213232.GE17478@madman.celabo.org> <272AEBD2-F486-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <20040823141803.GN27355@madman.celabo.org> On Sun, Aug 22, 2004 at 11:56:42PM +0200, Oliver Eikemeier wrote: > Jacques A. Vidrine wrote: > 60 (in words: sixty) entries in portaudit have the description `Please > contact the FreeBSD Security Team for more information'. There are > references, so when you care to add a quote, feel free, in fact this > might be a job for the security team. You can frown on them as often as > you like, the question is whether you just want to have an optional > <description> entry as an easy to spot sign that an editor is needed, or > if you prefer to search for and similar constructs. I'm not sure what you are talking about. I don't see any such entries in VuXML ... but you said `portaudit' so maybe you are talking about your personal database? > >However, I must admit that I have some doubt the value of the > ><discovery> date in any case. What I'd really like to hear are some > >arguments for keeping it or getting rid of it! I think it is useful > >information of itself to many reading VuXML content, and that combined > >with <entry> it provides a good metric about our response time. But I > >could be overestimating the value of it, and if it somehow puts people > >off to need to provide this information, then maybe it loses. > > Oviously we have a different opinion what is useful here. I expect most > users to be simple consumers, not security researchers. They need > information about the serverity of a vulnerability, and maybe > remote/local exploitability, whoever cares about the discovery date > could check the references. Often I find the discovery date > entertaining, but not useful. So I'll take that as a vote for not keeping it (<discovery>). Such a change (dropping required content) would need to take place in a `major' update e.g. VuXML 2.0. We'll revisit it then, maybe someone else will add some opinions before then. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Mon Aug 23 07:22:48 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Mon Aug 23 07:23:36 2004 Subject: database tables for VuXML In-Reply-To: <EC0A88B8-F487-11D8-8CAA-00039312D914@fillmore-labs.com> References: <20040822213304.GF17478@madman.celabo.org> <EC0A88B8-F487-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <20040823142227.GO27355@madman.celabo.org> On Mon, Aug 23, 2004 at 12:09:22AM +0200, Oliver Eikemeier wrote: > >The current version of VuXML does not allow these to be null, and I it > >is probably good to reflect those constraints in the database. > > Which means you possibly have to change the database description when > upgrading. I guess you have some point. You must like Tcl :-) No required elements will be dropped in any VuXML 1.x, but if/when there is a VuXML 2.0 there will likely be some changes to be made in applications (duh). > <architecture> should be stuff like i386? I'm not sure that there are > many applications for that. Usually the description is sufficient for > those purposes. Or did you have something different in mind? Yeah, stuff like `i386'. I don't see it (or <category>) as useful for the FreeBSD Project today, either: these are for use by other projects. In the case of <architecture>, I'm not yet sure that it will make it through pre-1.2. I'm still working with the project to understand their needs. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Mon Aug 23 07:29:31 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Mon Aug 23 07:29:33 2004 Subject: database tables for VuXML In-Reply-To: <4129070D.28233.7C2E4023@localhost> References: <41279E59.13631.76AD2AB9@localhost> <4129070D.28233.7C2E4023@localhost> Message-ID: <20040823142910.GP27355@madman.celabo.org> On Sun, Aug 22, 2004 at 08:50:21PM -0400, Dan Langille wrote: > A thought which came to me overnight: does anyone have suggestions as > to how existing FreshPorts pages should be modified to include VuXML > messages? Please feel free to take an existing FreshPorts page, do a > file-save-as, and alter the HTML to suit. Then give us the URL where > we can find your version. > > My idea: Take http://www.freshports.org/sysutils/bacula/ as an > example. I think adding a special icon to the left hand colum under > Commit History. Alternatives welcome. First something off-topic: Dan, how do you generate the package names that are seen in that left-hand column? I've often thought it would be handy to see the "history" of package names, and that seems to be exactly what you have there. Personally, I think a special icon with a link to vuxml.freebsd.org/<id>.html (or internally rendered page) and an ALT="<topic>" would be just swell! It might also be handy to have a bright warning in the top, main section if the HEAD is currently affected... similar to how you handle FORBIDDEN, BROKEN, IGNORE, etc ports. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From dan at langille.org Mon Aug 23 07:54:03 2004 From: dan at langille.org (Dan Langille) Date: Mon Aug 23 07:54:06 2004 Subject: database tables for VuXML In-Reply-To: <20040823142910.GP27355@madman.celabo.org> References: <4129070D.28233.7C2E4023@localhost> Message-ID: <4129CCC9.11697.7F32B4B4@localhost> On 23 Aug 2004 at 9:29, Jacques A. Vidrine wrote: > On Sun, Aug 22, 2004 at 08:50:21PM -0400, Dan Langille wrote: > > A thought which came to me overnight: does anyone have suggestions as > > to how existing FreshPorts pages should be modified to include VuXML > > messages? Please feel free to take an existing FreshPorts page, do a > > file-save-as, and alter the HTML to suit. Then give us the URL where > > we can find your version. > > > > My idea: Take http://www.freshports.org/sysutils/bacula/ as an > > example. I think adding a special icon to the left hand colum under > > Commit History. Alternatives welcome. > > First something off-topic: Dan, how do you generate the package names > that are seen in that left-hand column? I've often thought it would > be handy to see the "history" of package names, and that seems to be > exactly what you have there. The package names? I think you are referring to PORTVERSION and PORTREVISION. For example, 1.34.6 is PORTVERSION, and 2 is the PORTREVISION. This information is generated from the following: make -V PORTREVISION -V PORTVERSION Is that what you are referring to? > Personally, I think a special icon with a link to > vuxml.freebsd.org/<id>.html (or internally rendered page) and an > ALT="<topic>" would be just swell! It might also be handy to have > a bright warning in the top, main section if the HEAD is currently > affected... similar to how you handle FORBIDDEN, BROKEN, IGNORE, etc > ports. I welcome example icons! I was thinking to link people to the internal FreshPorts pages, and from there to the official vuxml.freebsd.org pages if they want additional information. -- Dan Langille : http://www.langille.org/ From eikemeier at fillmore-labs.com Mon Aug 23 08:03:22 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Mon Aug 23 08:03:25 2004 Subject: portaudit wishlist In-Reply-To: <20040823140949.GL27355@madman.celabo.org> Message-ID: <941610FA-F515-11D8-8CAA-00039312D914@fillmore-labs.com> Jacques A. Vidrine wrote: > On Sun, Aug 22, 2004 at 11:39:47PM +0200, Oliver Eikemeier wrote: >> It should be trivial to deal with the <alternate> tag in XSLT, the same >> should be possible with <optional>, and for entering them into >> databases >> you have to render the stuff anyway. > > One of your concerns was bloat, and I think using > <alternate>/<optional> like this would be just as much bloat. The > only redeeming quality of the csh-style syntax was brevity (but as I > said, I don't think that's a sufficient reason to use it). Bloat as in `linear search'. It's easy to render <alternate> to csh-style braces. >> Readability of the XML code is a >> non-issue, since it is designed to be machine-readable, not >> human-readable. > > No, that's really not correct. XML is not necessarily user-readable, > but it may certainly be human-readable. Or do you think that all our > doc committers that work on the web site or hand book and so forth > work in "write-only" mode? (rhetorical question) Users should have tools. I may write a `make entry' when we settled on a format suitable for our purposes. >> portaudit is designed to be lightweight and work without >> a database, so it does linear searches on all systems. I might change >> that, but that's the way things currently are, so what's the problem >> here? > > I don't know, what *is* the problem here? What problem are you trying > to solve with this new syntax? It doesn't seem to be readability > (the csh syntax and <alternate>/<optional> elements are both worse > for readability). You mentioned space, but I don't believe that's an > issue, and you seem to not either. I don't care about the size of the vuxml file, but about the rendered version. I'm fine with openldap-{client,server}, but you had concerns that this isn't unsable for xml tools, so I suggested an alternative in xml. >> [...] >>> So basically I think we should introduce this in VuXML 1.2. I'd like >>> to hear some comments from others about it, especially from the point >>> of view of the user reading content generated from VuXML. >> >> Uhm, you don't need vuxml 1.2 for that, this is perfectly legal in >> vuxml >> since 1.0. > > No, it is not. You can of course have a *valid* VuXML document with > `1.*' or even `@#$@% mary had a little lamb' as content for the <range> > child elements, but that doesn't make it a legal or proper VuXML > document. The DTD is only part of the specification. The other part is > described in the DTD comments. I can only find `The version ranges given must not overlap.' Other pointers? I still think this is a proper vuxml range specification. How a platform defines (and sorts) version numbers isn't handled in the vuxml specification. >>>> [...] >> I can't really follow your rationale here. You claim that because it >> can't be done perfectly, it shouldn't be done at all. > > I don't think you read and understood what I wrote. Let me try to > be clearer: the security team does not currently assign severity to > security advisories or VuXML entries, and has no plans to do so. > Rather, we believe that all of these security notices should be acted > upon. Theory and practice. There are seven unfixed vulnerabilities on freefall: ssh <login>@freefall.freebsd.org pkg_info | cut -f 1 -d ' ' | xargs portaudit It is useful to have the ability to mark vulnerabilities as `minor', like for example CAN-2004-0777. I did make an entry because we can't currently rank them. And insisting that this is not useful for `the security team' supports the theory that portaudit and vuxml need different databases. >> I would find it >> incredibly useful to get some categorization into `dangerous', >> `important' and `minor', even when it's wrong sometimes. As discussed >> before: You usually have a pretty good idea whether a vulnerability is >> severe or not, you just don't want to tell anybody. >> >> I consider this valuable to give users a chance to customize the >> notification scheme of portaudit, and why shouldn't this information >> find its place in the vuxml database? Make this tag optional, so you >> don't have to fill in anything when you on't feel like it. > > I think that any severity rating system should be seperate from the > VuXML documentation to allow for multiple such systems and policies, > particularly as practices in this area evolve. I don't think so. Either we have a common view on vulnerabilities, or we don't. I guess we should make vuxml the official database format, portaudit the official tool and be set. What are `multiple systems and policies' in this context? a severity rating is advisory only, and could be used or ignored however the system thinks is fit. What's the point when FreshPorts has an other severity level that portaudit? Or should it use the portaudit severity ranking? I can't understand your problems here. >>>> - add a classification into remote/local exploitable >>> >>> On the one hand, I feel this should be handled in the narrative >>> description and that it is otherwise just another facet of the >>> severity rating. On the other hand, I can imagine someone deciding >>> that it was OK e.g. to install any ports that did not have a >>> *remotely* exploitable vulnerability. >>> >>> My instinct tells me this too should be adjunct, at least >>> for now. Why would we include remotely/locally, but not >>> authenticated/unauthenticated, application privileges/system >>> privileges, user privileges/superuser privileges, or other such >>> aspects? If you have a server with a vulnerability that lets someone >>> who has a local account to get some other access, would you record >>> that as local or remote? >> >> umh, local of course? > > Funny you should say `of course'. I would have classified it as > remote. I picked this example from some correspondence involving > several colleagues, and guess what: the answers were mixed. Which is > `of course' my point. You have border cases in every system. Just denying to add useful features because some entries could be erroneous isn't helpful. Users could use this information, even when it's not 100% correct. Even the <range> entries in vuxml are not 100% correct, so what? >>> Yes, I think it is misleading to apply such tags which a user might >>> take as an absolute judgement when in fact they just need to read the >>> description. >> >> Not everyone has the time to review every description. Besides, the >> description might be as wrong or misleading as the tags mentioned. If >> you say "users have to understand the system fully or they shouldn't >> run >> the software" you basically state "FreeBSD is only for experts". I'm >> just trying to make some often asked questions machine readable. For >> example when I run portaudit on a server with no users, I might decide >> to care for local exploitable vulnerabilities only ever friday, while I >> have to handle remote exploitable vulnerabilities immediately. This >> system is not perfect, but usable. You give users basically no way to >> filter the information, which would be a valuable feature. One one hand >> you state users have to be knowledgeable to run a system, one the other >> you claim they might take tags `as an absolute judgement'. In this case >> reading the (possibly wrong) description might not improve anything. > > Your ``reasoning'' makes me dizzy. > > Look Oliver, knock yourself out: come up with your own severity rating > scheme and implement it. Stop bugging the security team to do it, > I've already explained that we will not at this time. Ok, back to my own database specification then? We have just a different view on our user base, and I think you fail to address some needs. Not everybody is a purist here, some `just want to have the job done', even when this means to err once or twice. >>>> - add a `fixed' field that lists a version where the vulnerability is >>>> fixed. This could be used for a recommendation message, like "upgrade >>>> to >>>> version xxx" or "no upgrade is available, please deinstall the port >>>> or >>>> proceed with caution". >>>> This could also realized as an alternate <lt> tag. >>> >>> I guess I don't understand this request. That is the purpose of the >>> <affected> element and children. >> >> There is no information whether there is an update available (and since >> which date the vulnerability is fixed), or if I simply have to >> deinstall >> the software and use a different product. > > Maybe you can describe this a bit more fully? (BTW, Can you make a > separate thread for each of your proposals?) I felt that the affected > ranges were sufficient, but if there is a better way let's hear it. There is no way to deduce from the VuXML document alone whether a fixed version exists (like in port<1.5: is there a port-1.5 in the tree or not?). So basically the user has to do the research whether s/he could just upgrade or has to deinstall the port completely (or live with the vulnerability). > [... snip: adding `description's to each reference ...] >> xfdb does it that way, and I like it. It's especially useful for >> mailing >> list posts, to see whether they contain an advisory or an exploit, for >> entries that cover multiple vulnerabilities (to distinguish the CVE >> references) and to filter out those `Updated packages fix multiple >> vulnerabilities' references for other operating systems. >> >>> The whole point of VuXML is to give enough information but not too >>> much :-) ``Too much'' is anything that is not likely to be supplied >>> in >>> the vast majority of cases. >> >> This could be automated by the tool that makes entries, or even by a >> tool that adds missing descriptions, so it is likely to be supplied. > > Well, I'm not sure I'm convinced. But let's try an example. > > <references> > <cvename description="BMP heap overflow">CAN-2004-0691</cvename> > <cvename description="XPM crash">CAN-2004-0692</cvename> > <cvename description="GIF crash">CAN-2004-0693</cvename> > > <url>http://www.trolltech.com/developer/changes/changes-3.3.3.html</url> > <url>http://scary.beasts.org/security/CESA-2004-004.txt</url> > </references> > > Is this the kind of thing you mean? More or less. Since these CVE references have no title yet, they would be shown as simple URLs. But you could have <url description="Trolltech - Changes 3.3.3"> http://www.trolltech.com/developer/changes/changes-3.3.3.html</url>. > I wonder if it will really be > used much *shrug* (by authors or users). I'd like to know what you > mean by `a tool that adds missing descriptions'. Are you thinking > of following the references and snarfing the <title> or similar? Jup, with a possibility to edit or deny it when it is too meaningless. > That might be neat. But, it would create two different meanings > for this attribute: (a) editorial description, i.e. to disambiguate > multiple similar references to CVE names, source code, whatever; (b) > the referenced item's description, e.g. <title> for HTML documents, > Subject: for email. I'm not sure I like that. As you seem to point > out, (b) could be largely automated, so I'm not sure it is worth > storing it in the document. Then when rendering a document, how will > a reader know that what she sees is (a) or (b)? Probably (b) or the url is fine here. (a) is too much work in the general case. > One more thing: I suggested `description' as the attribute name, but I > have second thoughts. I believe it might be better to pick something > that is not also an element name, and perhaps `description' is a > little too much typing. Whatever you like. Since entries should be done by tools, I don't care how much typing is necessary. -Oliver From eikemeier at fillmore-labs.com Mon Aug 23 08:10:08 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Mon Aug 23 08:10:10 2004 Subject: making <description> optional In-Reply-To: <20040823141803.GN27355@madman.celabo.org> Message-ID: <86914F26-F516-11D8-8CAA-00039312D914@fillmore-labs.com> Jacques A. Vidrine wrote: > On Sun, Aug 22, 2004 at 11:56:42PM +0200, Oliver Eikemeier wrote: >> Jacques A. Vidrine wrote: >> 60 (in words: sixty) entries in portaudit have the description `Please >> contact the FreeBSD Security Team for more information'. There are >> references, so when you care to add a quote, feel free, in fact this >> might be a job for the security team. You can frown on them as often as >> you like, the question is whether you just want to have an optional >> <description> entry as an easy to spot sign that an editor is needed, >> or >> if you prefer to search for and similar constructs. > > I'm not sure what you are talking about. I don't see any such entries > in VuXML ... but you said `portaudit' so maybe you are talking about > your personal database? I have a supplementary databases that are as much `personal' as vuxml is. The portaudit text database has been announced and documented as mentioned in a previous discussion. Anyway, I think making the entry optional would be the best solution, but if you prefer a placeholder, we can keep `Please contact the FreeBSD Security Team for more information'. >>> However, I must admit that I have some doubt the value of the >>> <discovery> date in any case. What I'd really like to hear are some >>> arguments for keeping it or getting rid of it! I think it is useful >>> information of itself to many reading VuXML content, and that combined >>> with <entry> it provides a good metric about our response time. But I >>> could be overestimating the value of it, and if it somehow puts people >>> off to need to provide this information, then maybe it loses. >> >> Oviously we have a different opinion what is useful here. I expect most >> users to be simple consumers, not security researchers. They need >> information about the serverity of a vulnerability, and maybe >> remote/local exploitability, whoever cares about the discovery date >> could check the references. Often I find the discovery date >> entertaining, but not useful. > > So I'll take that as a vote for not keeping it (<discovery>). Such > a change (dropping required content) would need to take place in a > `major' update e.g. VuXML 2.0. We'll revisit it then, maybe someone > else will add some opinions before then. Whatever you like. Simply using dummy values is fine with me. -Oliver From eikemeier at fillmore-labs.com Mon Aug 23 08:13:11 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Mon Aug 23 08:23:05 2004 Subject: database tables for VuXML In-Reply-To: <20040823142227.GO27355@madman.celabo.org> Message-ID: <F42627C5-F516-11D8-8CAA-00039312D914@fillmore-labs.com> Jacques A. Vidrinew wrote: > On Mon, Aug 23, 2004 at 12:09:22AM +0200, Oliver Eikemeier wrote: [...] >> <architecture> should be stuff like i386? I'm not sure that there are >> many applications for that. Usually the description is sufficient for >> those purposes. Or did you have something different in mind? > > Yeah, stuff like `i386'. I don't see it (or <category>) as useful > for the FreeBSD Project today, either: these are for use by other > projects. In the case of <architecture>, I'm not yet sure that it > will make it through pre-1.2. I'm still working with the project to > understand their needs. FWIIW, I would find a <system> tag that enables the sharing of entries across operating systems far more useful. -Oliver From eikemeier at fillmore-labs.com Mon Aug 23 08:21:18 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Mon Aug 23 08:40:05 2004 Subject: portaudit wishlist In-Reply-To: <941610FA-F515-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <15E125E6-F518-11D8-8CAA-00039312D914@fillmore-labs.com> [...] >>>> Yes, I think it is misleading to apply such tags which a user might >>>> take as an absolute judgement when in fact they just need to read the >>>> description. >>> >>> Not everyone has the time to review every description. Besides, the >>> description might be as wrong or misleading as the tags mentioned. If >>> you say "users have to understand the system fully or they shouldn't >>> run >>> the software" you basically state "FreeBSD is only for experts". I'm >>> just trying to make some often asked questions machine readable. For >>> example when I run portaudit on a server with no users, I might decide >>> to care for local exploitable vulnerabilities only ever friday, >>> while I >>> have to handle remote exploitable vulnerabilities immediately. This >>> system is not perfect, but usable. You give users basically no way to >>> filter the information, which would be a valuable feature. One one >>> hand >>> you state users have to be knowledgeable to run a system, one the >>> other >>> you claim they might take tags `as an absolute judgement'. In this >>> case >>> reading the (possibly wrong) description might not improve anything. >> >> Your ``reasoning'' makes me dizzy. >> >> Look Oliver, knock yourself out: come up with your own severity rating >> scheme and implement it. Stop bugging the security team to do it, >> I've already explained that we will not at this time. > > Ok, back to my own database specification then? We have just a > different view on our user base, and I think you fail to address some > needs. Not everybody is a purist here, some `just want to have the job > done', even when this means to err once or twice. Thinking a little about it, I believe this should be discussed in a place where portaudit users are present, either ports@ or security. freebsd-vuxml@ has too few subscribers to get an useful picture of what features desired by users are. -Oliver From eikemeier at fillmore-labs.com Mon Aug 23 10:59:06 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Mon Aug 23 11:00:08 2004 Subject: database tables for VuXML In-Reply-To: <4129CCC9.11697.7F32B4B4@localhost> Message-ID: <218B20FB-F52E-11D8-8CAA-00039312D914@fillmore-labs.com> Dan Langille wrote: >> Personally, I think a special icon with a link to >> vuxml.freebsd.org/<id>.html (or internally rendered page) and an >> ALT="<topic>" would be just swell! It might also be handy to have >> a bright warning in the top, main section if the HEAD is currently >> affected... similar to how you handle FORBIDDEN, BROKEN, IGNORE, etc >> ports. > > I welcome example icons! > > I was thinking to link people to the internal FreshPorts pages, and > from there to the official vuxml.freebsd.org pages if they want > additional information. When you want to render your own pages: portaudit-db does this. -Oliver From dan at langille.org Tue Aug 24 14:38:57 2004 From: dan at langille.org (Dan Langille) Date: Tue Aug 24 14:50:39 2004 Subject: database tables for VuXML In-Reply-To: <20040822192915.GA17478@madman.celabo.org> References: <41279E59.13631.76AD2AB9@localhost> Message-ID: <412B7D29.31547.1472F80@localhost> On 22 Aug 2004 at 14:29, Jacques A. Vidrine wrote: > [Added freebsd-vuxml@] > > On Sat, Aug 21, 2004 at 07:11:21PM -0400, Dan Langille wrote: > > Hi folks, > > > > I'm in the early stage of incorporating VuXML information into > > FreshPorts. > > Alright! That'll be cool. > > > I have a preliminary database table design which appears > > below. If you think I've missed any essential information out of > > this design, please let me know. > > > > First: has anyone already worked on creating database tables for the > > VuXML data? > > I have an old prototype application which uses SQLite. FWIW, here is > the schema I used then. (the formatting is a little funny because this > was embedded in some other code) > > CREATE TABLE vuxml ( > vid VARCHAR PRIMARY KEY UNIQUE > , topic VARCHAR > , description VARCHAR > , discovery DATE > , entry DATE > , modified DATE > -- computed max(entry, modified) for convenience in > -- other expressions > , last DATE > ); This is pretty much identical to what I have. > > CREATE TABLE refs ( > vid VARCHAR > , type VARCHAR > , text VARCHAR > ); Ahh, yes, that repeating group. I'm not storing that. I should be > CREATE TABLE affected ( > vid VARCHAR > , type VARCHAR > ); What is this type field? I didn't follow what the affected table contains. > CREATE TABLE names ( > affected INTEGER > , name VARCHAR > ); >From your example below, the names table would have entries for mutt and ja-mutt. I think I'll do that as well. See the end of my message for how I plan to relate VuXML entries to specific ports. > CREATE TABLE ranges ( > affected INTEGER > , lowop VARCHAR > , low VARCHAR > , glow0 INTEGER > , glow1 INTEGER > , glow2 INTEGER > , glow3 INTEGER > , highop VARCHAR > , high VARCHAR > , ghigh0 INTEGER > , ghigh1 INTEGER > , ghigh2 INTEGER > , ghigh3 INTEGER > ); > > The `ranges' table perhaps looks erm interesting. In this case, > I encoded each package version number into four 32-bit integers. > This allowed one to lookup an affected package completely in SQL. > However, the encoding I used may now be incorrect in some ways, since > pkg_install et al were recently changed to treat version numbers > differently. FreshPorts stores PORTVERSION and PORTREVISION as separate text fields. I take it that the ranges equations should compare only PORTVERSION and ignore PORTREVISION. > Also, SQLite could not really optimize the complex SQL > statement that resulted. So I think it is interesting for further > study, but I doubt I'd do it that way again unless I had a very > particular need to use pure SQL or the database was huge. SQLite is a great little database. I like what I've seen of it (mostly via my work on Bacula). I'll be using PostgreSQL for this. > > > create table vuxml > > ( > > id serial not null, > > -- internal FreshPorts ID > > vid text not null, > > -- the vuln vid. e.g. c4b025bb-f05d-11d8-9837-000c41e2cdad > > topic text not null, > > description text not null, > > date_discovery date , > > date_entry date , > > date_modified date , > > status date not null, > > primary key (id) > > ); > > > > create table vuxml_port_xref > > ( > > id serial not null, > > vuxml_id integer not null, > > port_id integer not null, > > primary key (id) > > ); > > > > create table vuxml_port_xref_range > > ( > > id serial not null, > > vuxml_port_xref_id text , > > range_operator_start text , > > range_operator_end text , > > range_version_start text , > > range_version_end text , > > primary key (id) > > ); > > > > If you consider the output from: vxquery -t vuxml ~/VuXML/vuln.xml > > tnftpd, the entry in vuxml_port_xref_range for tnftpd might be: > > > > (1, 'lt', 20040810, NULL, NULL) > > Seem reasonable. For what its worth, storing the range operators as > `>', `=>', etc lets you get something human readable as the output of a > SELECT, e.g. > > SELECT range_version_start, range_operator_start, port_name, > range_operator_end, range_version_end FROM ... WHERE ...; > > ('20040810', '<', 'tnftpd', '', '') > > I found that handy for manual inspection of the database. Whatever I store, need to be able to run queries upon it. I was thinking of keeping the existing values as that might be easier with perl. Not sure yetp. > You may notice from my example that I used separate `name', `ranges', > and `affected' tables. I found this easier when filling the database, > since the <package>/<system> elements are kind of a cross-product > operator. For example, when processing something like > > <package> > <name>mutt</name> > <name>ja-mutt</name> > <range><ge>1.4</ge><lt>1.4.2</lt></range> > </package> This is what I do with my vuxml_port_xref table. That table relates a vid to a port. > I first created an `affected' entry to contain the other references. > Then, as I hit each `name' I just associated it with the `affected' > entry, and as I hit each `range' I did the same. (Otherwise one > needs to construct a few lists, and then make all entries once > the </package> end tag is seen.) I think this is also easier to > modify if we get new children of <package>, e.g. the <category> and > <architecture> tags that are currently in the VuXML pre-1.2 DTD. My plan is to populate empty the vuxml_* tables each time there is a commit to the data file. This keeps the vuxml system totally separate from FreshPorts. Each row under Commit History (e.g. http://beta.freshports.org/sysutils/bacula/) relates to a row from the commit_log_ports table. That looks something like this: create table commit_log_ports ( commit_log_id integer not null, port_id integer not null, needs_refresh smallint not null, port_version text , port_revision text , primary key (commit_log_id, port_id) ); I'll probably create another table commit_log_ports_vuxml: create table commit_log_ports ( commit_log_id integer not null, port_id integer not null, vuxml_id integer not null , primary key (commit_log_id, port_id) ); Looking at the current data, there's about 140 affected ports, but I haven't broken that with respect to ranges, which is what the above will do. I'll either do that, or add the vuxml_id column to the commit_log_ports table, but I'd rather keep it separate. Time will tell. -- Dan Langille : http://www.langille.org/ From eikemeier at fillmore-labs.com Tue Aug 24 15:35:30 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Tue Aug 24 15:51:18 2004 Subject: database tables for VuXML In-Reply-To: <412B7D29.31547.1472F80@localhost> Message-ID: <EA56FB26-F61D-11D8-91E7-00039312D914@fillmore-labs.com> Dan Langille wrote: > FreshPorts stores PORTVERSION and PORTREVISION as separate text > fields. I take it that the ranges equations should compare only > PORTVERSION and ignore PORTREVISION. Please not. pkg_version -t is the reference here, when you get different results something is wrong (and ignoring PORTREVISION will get you different results). Also keep in mind that PORTEPOCH is an essential component. -Oliver From dan at langille.org Tue Aug 24 17:53:29 2004 From: dan at langille.org (Dan Langille) Date: Tue Aug 24 17:53:31 2004 Subject: database tables for VuXML In-Reply-To: <EA56FB26-F61D-11D8-91E7-00039312D914@fillmore-labs.com> References: <412B7D29.31547.1472F80@localhost> Message-ID: <412BAAC8.10821.1F94E6D@localhost> On 25 Aug 2004 at 0:35, Oliver Eikemeier wrote: > Dan Langille wrote: > > > FreshPorts stores PORTVERSION and PORTREVISION as separate text > > fields. I take it that the ranges equations should compare only > > PORTVERSION and ignore PORTREVISION. > > Please not. pkg_version -t is the reference here, when you get different > results something is wrong I'm confused. FreshPorts obtains information primarily via "make - V". pkg_version works on installed ports only. FreshPorts is not gong to install a port to find out what version we have. Are we talking about the same thing? > (and ignoring PORTREVISION will get you different results). > Also keep in mind that PORTEPOCH is an essential component. That's news to me. Thanks. I see it is used to create PKGNAME. Looking at range data of vuxml, I see now that PORTEPOCH is used. OK. Thanks for the heads up. -- Dan Langille : http://www.langille.org/ From eikemeier at fillmore-labs.com Wed Aug 25 00:41:04 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Wed Aug 25 00:41:06 2004 Subject: database tables for VuXML In-Reply-To: <412BAAC8.10821.1F94E6D@localhost> Message-ID: <1F0717EB-F66A-11D8-91E7-00039312D914@fillmore-labs.com> Dan Langille wrote: > On 25 Aug 2004 at 0:35, Oliver Eikemeier wrote: > >> Dan Langille wrote: >> >>> FreshPorts stores PORTVERSION and PORTREVISION as separate text >>> fields. I take it that the ranges equations should compare only >>> PORTVERSION and ignore PORTREVISION. >> >> Please not. pkg_version -t is the reference here, when you get >> different >> results something is wrong > > I'm confused. FreshPorts obtains information primarily via "make - > V". pkg_version works on installed ports only. FreshPorts is not > gong to install a port to find out what version we have. Are we > talking about the same thing? <http://www.freebsd.org/cgi/man.cgi?query=pkg_version> -Oliver From nectar at FreeBSD.org Wed Aug 25 09:33:56 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 09:39:35 2004 Subject: making <description> optional In-Reply-To: <86914F26-F516-11D8-8CAA-00039312D914@fillmore-labs.com> References: <86914F26-F516-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <825857C8-F6B4-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 23, 2004, at 10:10 AM, Oliver Eikemeier wrote: > I have a supplementary databases that are as much `personal' as vuxml > is. The portaudit text database has been announced and documented as > mentioned in a previous discussion. I'm sorry, I couldn't find that text (`Please contact [...]') anywhere in portaudit.txt or related files. Now I realize that is generated in the pages at ~eik/portaudit. I think we're on the same page now. > Anyway, I think making the entry optional would be the best solution, > but if you prefer a placeholder, we can keep `Please contact the > FreeBSD Security Team for more information'. I prefer an actual description, but OK :-) > Whatever you like. Simply using dummy values is fine with me. For the sake of harmony, let's agree that setting <discovery>unknown</discovery> is acceptable for now. Gimme a few days to check that stuff doesn't break when processing said entry. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Wed Aug 25 11:05:49 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 11:16:03 2004 Subject: portaudit wishlist In-Reply-To: <941610FA-F515-11D8-8CAA-00039312D914@fillmore-labs.com> References: <941610FA-F515-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <56E15C66-F6C1-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 23, 2004, at 10:03 AM, Oliver Eikemeier wrote: > Jacques A. Vidrine wrote: > >> On Sun, Aug 22, 2004 at 11:39:47PM +0200, Oliver Eikemeier wrote: >>> It should be trivial to deal with the <alternate> tag in XSLT, the >>> same >>> should be possible with <optional>, and for entering them into >>> databases >>> you have to render the stuff anyway. >> >> One of your concerns was bloat, and I think using >> <alternate>/<optional> like this would be just as much bloat. The >> only redeeming quality of the csh-style syntax was brevity (but as I >> said, I don't think that's a sufficient reason to use it). > > Bloat as in `linear search'. It's easy to render <alternate> to > csh-style braces. OK, now I understand what you are talking about. I'm sorry, I think you tried to tell me several times already, but I didn't understand. Today there are about 245 <name> elements, and I'd guess that maybe 40 could be replaced with a syntax such as you suggest. However, I don't think a 17% reduction in entries to speed up a linear search is worth the uglification. The first rule of optimization is, well, "don't do it", but one of the other top rules is to optimize by selecting the right algorithm, rather than attempting to speed up an inherently slower one. [...] >>> Uhm, you don't need vuxml 1.2 for that, this is perfectly legal in >>> vuxml >>> since 1.0. >> >> No, it is not. You can of course have a *valid* VuXML document with >> `1.*' or even `@#$@% mary had a little lamb' as content for the >> <range> >> child elements, but that doesn't make it a legal or proper VuXML >> document. The DTD is only part of the specification. The other part >> is >> described in the DTD comments. > > I can only find `The version ranges given must not overlap.' Other > pointers? I still think this is a proper vuxml range specification. > How a platform defines (and sorts) version numbers isn't handled in > the vuxml specification. You are right, this could only be specified per platform. I guess I'm confusing it with VuXML point releases, because changing the syntax also means tools have to adapt. For example, vxquery uses code from pkg_install which does not "understand" `1.*'. This just meants that vxquery needs to be updated. portaudit already groks it, and I don't think it matters for vuxml.org's rendered pages. So, I'll update vxquery and then it should be OK to use. Tangent --- maybe the version comparison code should be available in a library. [...] > It is useful to have the ability to mark vulnerabilities as `minor', > like for example CAN-2004-0777. Hmm, I think that was a mistake. I feel quite certain that a significant number of installations actually have DEBUG_LOGIN set. [...] >> I think that any severity rating system should be seperate from the >> VuXML documentation to allow for multiple such systems and policies, >> particularly as practices in this area evolve. > > I don't think so. Either we have a common view on vulnerabilities, or > we don't. Well, we disagree, Oliver. > I guess we should make vuxml the official database format, portaudit > the official tool and be set. What are `multiple systems and policies' > in this context? A FreeBSD Project mantra is `tools, not policy'. `multiple systems and policies' means multiple tools. > a severity rating is advisory only, and could be used or ignored > however the system thinks is fit. What's the point when FreshPorts has > an other severity level that portaudit? Perhaps FreshPorts (or whatever) has a rating system that better fits what some users need to accomplish. > Or should it use the portaudit severity ranking? I can't understand > your problems here. Conversely, I do not understand yours. :-( >>>>> - add a classification into remote/local exploitable >>>> >>>> On the one hand, I feel this should be handled in the narrative >>>> description and that it is otherwise just another facet of the >>>> severity rating. On the other hand, I can imagine someone deciding >>>> that it was OK e.g. to install any ports that did not have a >>>> *remotely* exploitable vulnerability. >>>> >>>> My instinct tells me this too should be adjunct, at least >>>> for now. Why would we include remotely/locally, but not >>>> authenticated/unauthenticated, application privileges/system >>>> privileges, user privileges/superuser privileges, or other such >>>> aspects? If you have a server with a vulnerability that lets >>>> someone >>>> who has a local account to get some other access, would you record >>>> that as local or remote? >>> >>> umh, local of course? >> >> Funny you should say `of course'. I would have classified it as >> remote. I picked this example from some correspondence involving >> several colleagues, and guess what: the answers were mixed. Which is >> `of course' my point. > > You have border cases in every system. I think it is not a border case, but fundamental. The fact that you think it is a border case and I think it is fundamental is indicative of our lack of agreement in this area. > Just denying to add useful features because some entries could be > erroneous isn't helpful. You think it is useful, I don't. I almost feel it is harmful, in fact :-) You have the freedom to implement those features you think are useful, and I have the freedom to ignore your insistence that I need to do something I consider not useful. > Users could use this information, even when it's not 100% correct. Sure. There is always the possibility of flaws. I suggest treating every issue as severe unless one knows better (for herself). This is called "fail closed", and I think that it is appropriate for a project such as ours. > Even the <range> entries in vuxml are not 100% correct, so what? Such entries should be considered bugs. Or was this a rhetorical question? [...] > Ok, back to my own database specification then? We have just a > different view on our user base, and I think you fail to address some > needs. Not everybody is a purist here, some `just want to have the job > done', even when this means to err once or twice. You already have your own *database* specification, and you should continue to use and evolve it. I think that issues should be *documented* in VuXML. Did you mean something else? [...] >> Maybe you can describe this a bit more fully? (BTW, Can you make a >> separate thread for each of your proposals?) I felt that the affected >> ranges were sufficient, but if there is a better way let's hear it. > > There is no way to deduce from the VuXML document alone whether a > fixed version exists (like in port<1.5: is there a port-1.5 in the > tree or not?). So basically the user has to do the research whether > s/he could just upgrade or has to deinstall the port completely (or > live with the vulnerability). I understand what you are saying, but I was looking for a proposal here. Frankly, what we have seems appropriate. Either a non-vulnerable version --- older or newer --- is available, or it isn't. But of course I haven't seen a good alternative approach. [...] >> Well, I'm not sure I'm convinced. But let's try an example. >> >> <references> >> <cvename description="BMP heap overflow">CAN-2004-0691</cvename> >> <cvename description="XPM crash">CAN-2004-0692</cvename> >> <cvename description="GIF crash">CAN-2004-0693</cvename> >> >> <url>http://www.trolltech.com/developer/changes/changes-3.3.3.html</ >> url> >> <url>http://scary.beasts.org/security/CESA-2004-004.txt</url> >> </references> >> >> Is this the kind of thing you mean? > > More or less. Since these CVE references have no title yet, they would > be shown as simple URLs. For my part, I would want to *always* show the URLs, and the description separately (extra line or column, "hover" pop-up, whatever). > But you could have <url description="Trolltech - Changes 3.3.3"> > http://www.trolltech.com/developer/changes/changes-3.3.3.html</url>. >> I wonder if it will really be >> used much *shrug* (by authors or users). I'd like to know what you >> mean by `a tool that adds missing descriptions'. Are you thinking >> of following the references and snarfing the <title> or similar? > > Jup, with a possibility to edit or deny it when it is too meaningless. > >> That might be neat. But, it would create two different meanings >> for this attribute: (a) editorial description, i.e. to disambiguate >> multiple similar references to CVE names, source code, whatever; (b) >> the referenced item's description, e.g. <title> for HTML documents, >> Subject: for email. I'm not sure I like that. As you seem to point >> out, (b) could be largely automated, so I'm not sure it is worth >> storing it in the document. Then when rendering a document, how will >> a reader know that what she sees is (a) or (b)? > > Probably (b) or the url is fine here. (a) is too much work in the > general case. So I guess what you are proposing is a tool that reads a VuXML document, snarfs all the remote reference titles, and spits out a new VuXML document with the titles inserted? In effect, this is just caching remote reference titles, isn't it? It seems to me (a) is useful to include in VuXML since it can only be provided by authors, while (b) doesn't seem to be very useful except possibly in rendering... and I have a feeling that a large number (maybe majority) of references have titles that are not well suited to being lifted out of context. CVE names are a particularly good example. All of them will say 'CAN-YYYY-XXXX (under review)' or 'CVE-YYYY-XXXX', which is useless. Advisories are often plain text, and so have no title that can be automatically snarfed. Web-based mailing list archives have additional spam at best, and often are completely useless (e.g. ``SecurityFocus BUGTRAQ Mailing List: BugTraq'' or ``Neohapsis Archives - Red Hat Linux - #0003 - [RHSA-2004:158-01] Updated cadaver package fixes security vulnerability in neon''). Yes, upon further reflection, I think automatically filling out the information in (b) is not a great idea. But I think (a) could be really useful in some cases... except that, I think authors probably would not fill it out often. >> One more thing: I suggested `description' as the attribute name, but I >> have second thoughts. I believe it might be better to pick something >> that is not also an element name, and perhaps `description' is a >> little too much typing. > > Whatever you like. I suppose we'll wait for inspiration, then :-) > Since entries should be done by tools, I don't care how much typing is > necessary. Well *in fact* manual editing is how VuXML documents are created/edited, along with billions of other XML and SGML documents, regardless of the availability of tools. This is kind of beside the point, but you seem to like to say this in justification of one thing or another when it is simply wrong. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From trhodes at FreeBSD.org Wed Aug 25 11:05:57 2004 From: trhodes at FreeBSD.org (Tom Rhodes) Date: Wed Aug 25 11:16:04 2004 Subject: making <description> optional In-Reply-To: <825857C8-F6B4-11D8-9236-000A95BC6FAE@FreeBSD.org> References: <86914F26-F516-11D8-8CAA-00039312D914@fillmore-labs.com> <825857C8-F6B4-11D8-9236-000A95BC6FAE@FreeBSD.org> Message-ID: <20040825140459.36c75b42@localhost.pittgoth.com> On Wed, 25 Aug 2004 11:33:34 -0500 Jacques Vidrine <nectar@freebsd.org> wrote: > On Aug 23, 2004, at 10:10 AM, Oliver Eikemeier wrote: > > I have a supplementary databases that are as much `personal' as vuxml > > is. The portaudit text database has been announced and documented as > > mentioned in a previous discussion. > > I'm sorry, I couldn't find that text (`Please contact [...]') anywhere > in portaudit.txt or related files. Now I realize that is generated in > the pages at ~eik/portaudit. I think we're on the same page now. > > > Anyway, I think making the entry optional would be the best solution, > > but if you prefer a placeholder, we can keep `Please contact the > > FreeBSD Security Team for more information'. > > I prefer an actual description, but OK :-) > > > Whatever you like. Simply using dummy values is fine with me. > > For the sake of harmony, let's agree that setting > <discovery>unknown</discovery> is acceptable for now. Gimme a few days > to check that stuff doesn't break when processing said entry. Ok, so we're on our way to closing this entire discussion. I seen Oliver's wishlist but have been too busy to reply. I'll try to get it done in the next few days. Thanks guys! -- Tom Rhodes From nectar at FreeBSD.org Wed Aug 25 11:09:02 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 11:17:40 2004 Subject: portaudit wishlist In-Reply-To: <15E125E6-F518-11D8-8CAA-00039312D914@fillmore-labs.com> References: <15E125E6-F518-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <CB00FE41-F6C1-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 23, 2004, at 10:21 AM, Oliver Eikemeier wrote: > Thinking a little about it, I believe this should be discussed in a > place where portaudit users are present, either ports@ or security. > freebsd-vuxml@ has too few subscribers to get an useful picture of > what features desired by users are. Certainly if you are looking for feedback on portaudit features, another list might be more appropriate. This list is for discussing VuXML format/contents/and so on. So I'd expect for example that you'd know "I need to implement feature X in portaudit" and make a proposal here on what might be needed in VuXML to make feature X possible for you to implement. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Wed Aug 25 11:59:14 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 12:12:10 2004 Subject: database tables for VuXML In-Reply-To: <412B7D29.31547.1472F80@localhost> References: <41279E59.13631.76AD2AB9@localhost> <412B7D29.31547.1472F80@localhost> Message-ID: <CDBB0430-F6C8-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 24, 2004, at 4:38 PM, Dan Langille wrote: [...] > What is this type field? I didn't follow what the affected table > contains. Type is either `package' or `system'. You are probably concerned only with the former. [...] > FreshPorts stores PORTVERSION and PORTREVISION as separate text > fields. I take it that the ranges equations should compare only > PORTVERSION and ignore PORTREVISION. One can think a version number as a tuple as follows (most significant first): (PORTEPOCH, PORTVERSION, PORTREVISION). Comparisons are done on these tuples. > SQLite is a great little database. I like what I've seen of it > (mostly via my work on Bacula). I'll be using PostgreSQL for this. Yeah, SQLite is great for single-user things. I especially like the way it is so easy to add SQL functions from C. > [...] > My plan is to populate empty the vuxml_* tables each time there is a > commit to the data file. This keeps the vuxml system totally > separate from FreshPorts. > > Each row under Commit History (e.g. > http://beta.freshports.org/sysutils/bacula/) relates to a row from > the commit_log_ports table. That looks something like this: > > create table commit_log_ports > ( > commit_log_id integer not null, > port_id integer not null, > needs_refresh smallint not null, > port_version text , > port_revision text , > primary key (commit_log_id, port_id) > ); You'll need a `port_epoch' here as well. So these get created as commits "come in"? > I'll probably create another table commit_log_ports_vuxml: > > create table commit_log_ports > ( > commit_log_id integer not null, > port_id integer not null, > vuxml_id integer not null , > primary key (commit_log_id, port_id) > ); > > Looking at the current data, there's about 140 affected ports, but I > haven't broken that with respect to ranges, which is what the above > will do. > > I'll either do that, or add the vuxml_id column to the > commit_log_ports table, but I'd rather keep it separate. Time will > tell. Seems like it'll work to me :-) Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Wed Aug 25 12:05:21 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 12:21:59 2004 Subject: (was Re: database tables for VuXML) In-Reply-To: <F42627C5-F516-11D8-8CAA-00039312D914@fillmore-labs.com> References: <F42627C5-F516-11D8-8CAA-00039312D914@fillmore-labs.com> Message-ID: <A96CC28A-F6C9-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 23, 2004, at 10:13 AM, Oliver Eikemeier wrote: > Jacques A. Vidrinew wrote: > [...] >> Yeah, stuff like `i386'. I don't see it (or <category>) as useful >> for the FreeBSD Project today, either: these are for use by other >> projects. In the case of <architecture>, I'm not yet sure that it >> will make it through pre-1.2. I'm still working with the project to >> understand their needs. > > FWIIW, I would find a <system> tag that enables the sharing of entries > across operating systems far more useful. How would you use it? Will you maintain vulnerability information for several projects in a single VuXML file? If so, I think the reasonable extension would be to specify a metadata attribute on the existing <package> and <system> tags. You would not likely want the `cross-product' effect. Then one could easily generate individual VuXML documents from a single document, simply selecting on this attribute and including only relevent entries. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Wed Aug 25 12:18:06 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Wed Aug 25 12:32:23 2004 Subject: database tables for VuXML In-Reply-To: <4129CCC9.11697.7F32B4B4@localhost> References: <4129070D.28233.7C2E4023@localhost> <4129CCC9.11697.7F32B4B4@localhost> Message-ID: <717D5B12-F6CB-11D8-9236-000A95BC6FAE@FreeBSD.org> On Aug 23, 2004, at 9:54 AM, Dan Langille wrote: > The package names? I think you are referring to PORTVERSION and > PORTREVISION. For example, 1.34.6 is PORTVERSION, and 2 is the > PORTREVISION. This information is generated from the following: > > make -V PORTREVISION -V PORTVERSION > > Is that what you are referring to? You accidently answered my question :-) I guess you have set those whenever the commit mail came in ... so the view of past package versions is from past executions of `make -V'. I wish I could `cd /usr/ports/some/app && make package-names' to get a list of all package names this port ever built. or some such. > I welcome example icons! > I was thinking to link people to the internal FreshPorts pages, and > from there to the official vuxml.freebsd.org pages if they want > additional information. Sounds great. I imagine you probably generate the pages from your database, but if you need any help with rendering directly from the XML, lemme know. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From dan at langille.org Wed Aug 25 13:02:45 2004 From: dan at langille.org (Dan Langille) Date: Wed Aug 25 13:10:19 2004 Subject: database tables for VuXML In-Reply-To: <717D5B12-F6CB-11D8-9236-000A95BC6FAE@FreeBSD.org> References: <4129CCC9.11697.7F32B4B4@localhost> Message-ID: <412CB824.3652.6159279@localhost> On 25 Aug 2004 at 14:17, Jacques Vidrine wrote: > > On Aug 23, 2004, at 9:54 AM, Dan Langille wrote: > > The package names? I think you are referring to PORTVERSION and > > PORTREVISION. For example, 1.34.6 is PORTVERSION, and 2 is the > > PORTREVISION. This information is generated from the following: > > > > make -V PORTREVISION -V PORTVERSION > > > > Is that what you are referring to? > > You accidently answered my question :-) I guess you have set those > whenever the commit mail came in ... so the view of past package > versions is from past executions of `make -V'. Correct. What you see under Commit History are values obtained from make -V at the time of that commit. > I wish I could `cd /usr/ports/some/app && make package-names' to get a > list of all package names this port ever built. or some such. > > > I welcome example icons! > > I was thinking to link people to the internal FreshPorts pages, and > > from there to the official vuxml.freebsd.org pages if they want > > additional information. > > Sounds great. I imagine you probably generate the pages from your > database, but if you need any help with rendering directly from the > XML, lemme know. Thanks. Most pages on FreshPorts have content obtained from the database. See http://www.freshports.org/how-big-is-it.php for page stats (about 250,000 pages now). -- Dan Langille : http://www.langille.org/ From dan at langille.org Wed Aug 25 13:12:25 2004 From: dan at langille.org (Dan Langille) Date: Wed Aug 25 13:20:40 2004 Subject: database tables for VuXML In-Reply-To: <CDBB0430-F6C8-11D8-9236-000A95BC6FAE@FreeBSD.org> References: <412B7D29.31547.1472F80@localhost> Message-ID: <412CBA68.15997.61E6A50@localhost> On 25 Aug 2004 at 13:58, Jacques Vidrine wrote: > > FreshPorts stores PORTVERSION and PORTREVISION as separate text > > fields. I take it that the ranges equations should compare only > > PORTVERSION and ignore PORTREVISION. > > One can think a version number as a tuple as follows (most significant > first): (PORTEPOCH, PORTVERSION, PORTREVISION). Comparisons are done > on these tuples. Oh, PORTEPOCH is first? Why? Shouldn't it be last? Here is the value forumula from bsd.port.mk PKGNAME= ${PKGNAMEPREFIX}${PORTNAME}${PKGNAMESUFFIX}- ${PORTVERSION:C/[-_,]/./g}${_SUF1}${_SUF2} where _SUF2= ,${PORTEPOCH} > > My plan is to populate empty the vuxml_* tables each time there is a > > commit to the data file. This keeps the vuxml system totally > > separate from FreshPorts. > > > > Each row under Commit History (e.g. > > http://beta.freshports.org/sysutils/bacula/) relates to a row from > > the commit_log_ports table. That looks something like this: > > > > create table commit_log_ports > > ( > > commit_log_id integer not null, > > port_id integer not null, > > needs_refresh smallint not null, > > port_version text , > > port_revision text , > > primary key (commit_log_id, port_id) > > ); > > You'll need a `port_epoch' here as well. So these get created as > commits "come in"? Yes, for each commit, a row is added to the commit_log table. For each port in a commit, a row is added to the commit_log_ports table. In addition to PORTEPOCH, I think I need to add: NO_LATEST_LINK NO_PACKAGE I may also want to add PKGNAME -- Dan Langille : http://www.langille.org/ From cswiger at mac.com Wed Aug 25 13:38:08 2004 From: cswiger at mac.com (Charles Swiger) Date: Wed Aug 25 13:40:16 2004 Subject: database tables for VuXML In-Reply-To: <412CBA68.15997.61E6A50@localhost> References: <412B7D29.31547.1472F80@localhost> <412CBA68.15997.61E6A50@localhost> Message-ID: <AA43EF1A-F6D6-11D8-8247-003065ABFD92@mac.com> On Aug 25, 2004, at 4:12 PM, Dan Langille wrote: > On 25 Aug 2004 at 13:58, Jacques Vidrine wrote: >> One can think a version number as a tuple as follows (most significant >> first): (PORTEPOCH, PORTVERSION, PORTREVISION). Comparisons are done >> on these tuples. > > Oh, PORTEPOCH is first? Why? Shouldn't it be last? Here is the > value forumula from bsd.port.mk > > PKGNAME= ${PKGNAMEPREFIX}${PORTNAME}${PKGNAMESUFFIX}- > ${PORTVERSION:C/[-_,]/./g}${_SUF1}${_SUF2} > > where _SUF2= ,${PORTEPOCH} PORTEPOCH is listed last in the package name, but it is more important than the other values. The reason PORTEPOCH exists is to fix PORTVERSION+PORTREVISION not producing a monotonously increasing sequence, right? So, no matter what PORTVERSION is, PORTEPOCH=1 means greater than PORTEPOCH of zero (ie, unset, the default value, whatever). PORTEPOCH=2 with any PORTVERSION is greater than any PORTVERSION value with PORTEPOCH=1, etc. -- -Chuck From eikemeier at fillmore-labs.com Wed Aug 25 16:01:23 2004 From: eikemeier at fillmore-labs.com (Oliver Eikemeier) Date: Wed Aug 25 16:06:57 2004 Subject: portaudit wishlist In-Reply-To: <CB00FE41-F6C1-11D8-9236-000A95BC6FAE@FreeBSD.org> Message-ID: <B3E048EE-F6EA-11D8-91E7-00039312D914@fillmore-labs.com> Jacques Vidrine wrote: > On Aug 23, 2004, at 10:21 AM, Oliver Eikemeier wrote: >> Thinking a little about it, I believe this should be discussed in a >> place where portaudit users are present, either ports@ or security. >> freebsd-vuxml@ has too few subscribers to get an useful picture of >> what features desired by users are. > > Certainly if you are looking for feedback on portaudit features, > another list might be more appropriate. This list is for discussing > VuXML format/contents/and so on. So I'd expect for example that you'd > know "I need to implement feature X in portaudit" and make a proposal > here on what might be needed in VuXML to make feature X possible for > you to implement. Uhm, yes. The point is that I want to classify vulnerabilities, so that users can filter the information they get to a certain extend. I don't think the classification should go into an extra file, since many vulnerability databases have classifications. I know that such a feature has a high degree of subjectivity, but I think it is useful when some vulnerabilities just can "stand out", for example 616cf823-f48b-11d8-9837-000c41e2cdad is pretty insignificant. I think I said all this before. I'm not exactly sure what this list is good for, since I keep discussing this stuff with you only, so I want to move this to a place where we can get some more feedback, like "yes, this will be a useful feature" or "no, we don't need that" from someone else besides you and me. Discussing this with people interested in the VuXML format seems like a waste of time, I want to discuss this with people _using_ the data, and the format just has to serve their needs. -Oliver From dan at langille.org Wed Aug 25 21:47:10 2004 From: dan at langille.org (Dan Langille) Date: Wed Aug 25 21:47:11 2004 Subject: icon for VuXML? Message-ID: <412D3303.5833.195280A@localhost> Can anyone recommend/design an icon for VuXML? Something about 13x13 to 21x21 pixals would be good. And something which doesn't look like any of the existing FreshPorts icons such as those at http://www.freshports.org/faq.php Cheers -- Dan Langille : http://www.langille.org/ From dan at langille.org Thu Aug 26 14:24:55 2004 From: dan at langille.org (Dan Langille) Date: Thu Aug 26 14:39:22 2004 Subject: icon for VuXML? In-Reply-To: <412D3303.5833.195280A@localhost> Message-ID: <412E1CE6.21976.526B0F3@localhost> On 26 Aug 2004 at 0:46, Dan Langille wrote: > Can anyone recommend/design an icon for VuXML? Something about 13x13 > to 21x21 pixals would be good. And something which doesn't look like > any of the existing FreshPorts icons such as those at > http://www.freshports.org/faq.php Here is a mock up: http://beta.freshports.org/tmp/freshports.orgftptnftpd.html The goal is to have the little skull beside any package which is affected. -- Dan Langille : http://www.langille.org/ From dan at langille.org Thu Aug 26 15:59:06 2004 From: dan at langille.org (Dan Langille) Date: Thu Aug 26 15:59:08 2004 Subject: database tables for VuXML In-Reply-To: <412B7D29.31547.1472F80@localhost> References: <20040822192915.GA17478@madman.celabo.org> Message-ID: <412E32F9.17168.57CEB1D@localhost> I am now finalizing my tables. I have some comments below and some questions about your tables. On 24 Aug 2004 at 17:38, Dan Langille wrote: > On 22 Aug 2004 at 14:29, Jacques A. Vidrine wrote: > > > [Added freebsd-vuxml@] > > > > On Sat, Aug 21, 2004 at 07:11:21PM -0400, Dan Langille wrote: > > > Hi folks, > > > > > > I'm in the early stage of incorporating VuXML information into > > > FreshPorts. > > > > Alright! That'll be cool. > > > > > I have a preliminary database table design which appears > > > below. If you think I've missed any essential information out of > > > this design, please let me know. > > > > > > First: has anyone already worked on creating database tables for the > > > VuXML data? > > > > I have an old prototype application which uses SQLite. FWIW, here is > > the schema I used then. (the formatting is a little funny because this > > was embedded in some other code) > > > > CREATE TABLE vuxml ( > > vid VARCHAR PRIMARY KEY UNIQUE > > , topic VARCHAR > > , description VARCHAR > > , discovery DATE > > , entry DATE > > , modified DATE > > -- computed max(entry, modified) for convenience in > > -- other expressions > > , last DATE > > ); I'm not going to have a last column. And I am using TEXT, not DATE, as the values are not always dates. I recall reading someone suggesting allowing unknown. If the date is unknown, I suggest leaving this field empty. If the suggestion above is accepted, I will reconsider my DATE versus TEXT decision as I could then use a NULL value. > This is pretty much identical to what I have. > > > > > CREATE TABLE refs ( > > vid VARCHAR > > , type VARCHAR > > , text VARCHAR > > ); What is this type? I fear I have asked this already. I have added the table but without the type field. > Ahh, yes, that repeating group. I'm not storing that. I should be > > > CREATE TABLE affected ( > > vid VARCHAR > > , type VARCHAR > > ); > > What is this type field? I didn't follow what the affected table > contains. Ahh, type is either package or system. e.g. <package> <name>tnftpd</name> <range><lt>20040810</lt></range> </package> <package> <name>lukemftpd</name> <range><ge>0</ge></range> </package> <system> <name>FreeBSD</name> <range><ge>4.7</ge></range> </system> > > CREATE TABLE names ( > > affected INTEGER > > , name VARCHAR > > ); This I now understand. It's a list of the affected packages, or systems. For packages, it appears to be PORTNAME. Is that true? I see now why you have names and affect separate: <package> <name>gaim</name> <name>ja-gaim</name> <range><lt>0.81_1</lt></range> </package> A given package may have more than one name (e.g. slave ports). > >From your example below, the names table would have entries for mutt > and ja-mutt. I think I'll do that as well. See the end of my > message for how I plan to relate VuXML entries to specific ports. > > > CREATE TABLE ranges ( > > affected INTEGER > > , lowop VARCHAR > > , low VARCHAR > > , glow0 INTEGER > > , glow1 INTEGER > > , glow2 INTEGER > > , glow3 INTEGER > > , highop VARCHAR > > , high VARCHAR > > , ghigh0 INTEGER > > , ghigh1 INTEGER > > , ghigh2 INTEGER > > , ghigh3 INTEGER > > ); > > > > The `ranges' table perhaps looks erm interesting. In this case, > > I encoded each package version number into four 32-bit integers. > > This allowed one to lookup an affected package completely in SQL. > > However, the encoding I used may now be incorrect in some ways, since > > pkg_install et al were recently changed to treat version numbers > > differently. Each package has a list of 1 or more ranges which are affected. The ranges affect all names listed in the package. e.g.: <package> <name>samba</name> <range><ge>3</ge><lt>3.0.5,1</lt></range> <range><lt>2.2.10</lt></range> </package> Thanks. This is coming together and I'm getting closer to having tables. -- Dan Langille : http://www.langille.org/ From dan at langille.org Thu Aug 26 18:23:50 2004 From: dan at langille.org (Dan Langille) Date: Thu Aug 26 18:23:51 2004 Subject: FreshPorts - VuXML tables - diagram Message-ID: <412E54E5.5382.6016DEA@localhost> Here's the diagram of the tables I'm considering. This should duplicate the entire contents of the VuXML table. http://beta.freshports.org/tmp/FreshPorts-VuXML-tables.jpg The commit_log_ports_vuxml table relates each entry in the vuxml table to one or more commits and to specific ports within that commit. The actual PORTREVISION, PORTVERSION, PORTEPOCH etc, are stored in another table (commit_log_ports), which is not shown. Why another table just for vuxml? It keep the vuxml separate from the FreshPorts data. The table I would put this information it is the commit_log_ports table (which has about 138,718 rows). I expect the commit_log_ports_vuxml table to have about 1000 rows (i.e. significantly less rows). I'd consider merging the data if the number of vuxml rows started to approach 30% of commit_log_ports... but I don't see that happening. cheers -- Dan Langille : http://www.langille.org/ From dan at langille.org Fri Aug 27 01:49:25 2004 From: dan at langille.org (Dan Langille) Date: Fri Aug 27 02:24:38 2004 Subject: tnftpd -- remotely exploitable vulnerability Message-ID: <20040827044722.G74931@xeon.unixathome.org> For vid="c4b025bb-f05d-11d8-9837-000c41e2cdad", should the following: <package> <name>FreeBSD</name> <range><ge>4.7</ge></range> </package> actually be change to system from package? i.e. <system> <name>FreeBSD</name> <range><ge>4.7</ge></range> </system> -- Dan Langille - http://www.langille.org/ From dan at langille.org Fri Aug 27 04:07:09 2004 From: dan at langille.org (Dan Langille) Date: Fri Aug 27 04:50:10 2004 Subject: FreshPorts - VuXML tables - DDL + data Message-ID: <20040827051637.T78365@xeon.unixathome.org> Here is the DDL I created this morning. Note that some slight changes have been made relative to the diagram I posted last night. The diagram is now out of date. The rest of this message contains a sample vuxml entry followed by the DDL to create the tables and populate it with a single VuXML entry. It also shows how I mark which packages are affected by a given vid (i.e. how I populate the commit_log_ports_vuxml table). I have created the code to display VuXML information, but at present, the only data loaded is for http://beta.freshports.org/ftp/tnftpd/ NOTE: I'm assuming there can be multiple VuXML entries for a given version of a package/port. However, there will be only one VuXML icon shown per row of Commit History. Look at the last commit on the page for a commit which is affected by two VuXML entries (this is achieved with a fake VuXML entry). freshports.org=# select id, vid, topic from vuxml; id | vid | topic ----+--------------------------------------+---------------------------------------------- 1 | c4b025bb-f05d-11d8-9837-000c41e2cdad | tnftpd -- remotely exploitable vulnerability (1 row) freshports.org=# select * from vuxml_affected; id | vuxml_id | type ----+----------+--------- 1 | 1 | package 2 | 1 | package 3 | 1 | system This vid affects two packages and relates to one system. freshports.org=# select * from vuxml_names; id | vport_versionport_versionuxml_affected_id | name ----+-------------------+----------- 1 | 1 | tnftpd 2 | 2 | lukemftpd 3 | 3 | FreeBSD (3 rows) The packages and system are listed above. freshports.org=# select * from vuxml_ranges; id | vuxml_name_id | range_operator_start | range_operator_end | range_version_start | range_version_end ----+---------------+----------------------+--------------------+---------------------+------------------- 7 | 1 | | | < | 20040810 8 | 2 | >= | 0 | | 9 | 3 | >= | 4.7 | | (3 rows) These are the ranges related to each vuxml_affected row. freshports.org=# select * from vuxml_references; vuxml_id | reference | type ----------+-------------------------------------------------------------------------------+------- 1 | http://cvsweb.netbsd.org/bsdweb.cgi/src/libexec/ftpd/ftpd.c#rev1.158 | url 1 | ftp://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2004-009.txt.asc | url 1 | http://lists.netsys.com/pipermail/full-disclosure/2004-August/025418.html | mlist (3 rows) freshports.org=# Given the above information, here is one way to populate the commit_log_ports_vuxml table. freshports.org=# insert into commit_log_ports_vuxml (commit_log_id, port_id, vuxml_id) select commit_log_id, port_id, 1 as vuxml_id from commit_log_ports where port_id = (select id from ports_active where name = 'tnftpd') and port_version < '20040810'; INSERT 0 9 DDL follows ### create table vuxml ( id serial not null, vid text not null, topic text not null, description text not null, date_discovery text , date_entry text , date_modified text , status char(1) not null, primary key (id) ); create table commit_log_ports_vuxml ( id serial not null, commit_log_id integer not null, port_id integer not null, vuxml_id integer not null, primary key (id) ); create table vuxml_ranges ( id serial not null, vuxml_name_id integer not null, range_operator_start text , range_operator_end text , range_version_start text , range_version_end text , primary key (id) ); create table vuxml_references ( vuxml_id integer not null, reference text not null, type text not null, primary key (vuxml_id, reference) ); create table vuxml_affected ( id serial not null, vuxml_id integer not null, type text not null, primary key (id) ); create table vuxml_names ( id serial not null, vuxml_affected_id integer not null, name text not null, primary key (id) ); alter table commit_log_ports_vuxml add foreign key (vuxml_id) references vuxml (id) on update cascade on delete cascade; alter table commit_log_ports_vuxml add foreign key (port_id) references ports (id) on update cascade on delete cascade; alter table commit_log_ports_vuxml add foreign key (commit_log_id) references commit_log (id) on update cascade on delete cascade; alter table vuxml_ranges add foreign key (vuxml_name_id) references vuxml_names (id) on update cascade on delete cascade; alter table vuxml_references add foreign key (vuxml_id) references vuxml (id) on update cascade on delete cascade; alter table vuxml_affected add foreign key (vuxml_id) references vuxml (id) on update cascade on delete cascade; alter table vuxml_names add foreign key (vuxml_affected_id) references vuxml_affected (id) on update cascade on delete cascade; ### Data follows: delete from vuxml where id = 1; insert into vuxml (id, vid, topic, description, date_discovery, date_entry, date_modified, status) values (1, 'c4b025bb-f05d-11d8-9837-000c41e2cdad', 'tnftpd -- remotely exploitable vulnerability', ' lukemftpd(8) is an enhanced BSD FTP server produced within the NetBSD project. The sources for lukemftpd are shipped with some versions of FreeBSD, however it is not built or installed by default. The build system option WANT_LUKEMFTPD must be set to build and install lukemftpd. [NOTE: An exception is FreeBSD 4.7-RELEASE, wherein lukemftpd was installed, but not enabled, by default.] Przemyslaw Frasunek discovered several vulnerabilities in lukemftpd arising from races in the out-of-band signal handling code used to implement the ABOR command. As a result of these races, the internal state of the FTP server may be manipulated in unexpected ways. A remote attacker may be able to cause FTP commands to be executed with the privileges of the running lukemftpd process. This may be a low-privilege ^Ftp\' user if the -r\' command line option is specified, or it may be superuser privileges if -r\' is *not* specified. ', '2004-08-17', '2004-08-17', NULL, 'A'); insert into vuxml_references (vuxml_id, reference, type) values (1, 'http://cvsweb.netbsd.org/bsdweb.cgi/src/libexec/ftpd/ftpd.c#rev1.158', 'url'); insert into vuxml_references (vuxml_id, reference, type) values (1, 'ftp://ftp.netbsd.org/pub/NetBSD/security/advisories/NetBSD-SA2004-009.txt.asc', 'url'); insert into vuxml_references (vuxml_id, reference, type) values (1, 'http://lists.netsys.com/pipermail/full-disclosure/2004-August/025418.html', 'mlist'); insert into vuxml_affected (id, vuxml_id, type) values (1, 1, 'package'); insert into vuxml_affected (id, vuxml_id, type) values (2, 1, 'package'); insert into vuxml_affected (id, vuxml_id, type) values (3, 1, 'system'); insert into vuxml_names (id, vuxml_affected_id, name) values (1, 1, 'tnftpd'); insert into vuxml_names (id, vuxml_affected_id, name) values (2, 2, 'lukemftpd'); insert into vuxml_names (id, vuxml_affected_id, name) values (3, 3, 'FreeBSD'); insert into vuxml_ranges (vuxml_name_id, range_operator_start, range_operator_end, range_version_start, range_version_end) values (1, NULL, NULL, '<', '20040810'); insert into vuxml_ranges (vuxml_name_id, range_operator_start, range_operator_end, range_version_start, range_version_end) values (2, '>=', '0', NULL, NULL); insert into vuxml_ranges (vuxml_name_id, range_operator_start, range_operator_end, range_version_start, range_version_end) values (3, '>=', '4.7', NULL, NULL); -- Dan Langille - http://www.langille.org/ From nectar at FreeBSD.org Fri Aug 27 06:43:39 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Fri Aug 27 07:01:23 2004 Subject: icon for VuXML? In-Reply-To: <412E1CE6.21976.526B0F3@localhost> References: <412D3303.5833.195280A@localhost> <412E1CE6.21976.526B0F3@localhost> Message-ID: <20040827134318.GC54027@madman.celabo.org> On Thu, Aug 26, 2004 at 05:24:54PM -0400, Dan Langille wrote: > Here is a mock up: > > http://beta.freshports.org/tmp/freshports.orgftptnftpd.html > > The goal is to have the little skull beside any package which is > affected. Seems like a good approach to me. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Fri Aug 27 07:04:26 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Fri Aug 27 07:15:30 2004 Subject: database tables for VuXML In-Reply-To: <412E32F9.17168.57CEB1D@localhost> References: <20040822192915.GA17478@madman.celabo.org> <412E32F9.17168.57CEB1D@localhost> Message-ID: <20040827140404.GE54027@madman.celabo.org> On Thu, Aug 26, 2004 at 06:59:05PM -0400, Dan Langille wrote: > I am now finalizing my tables. I have some comments below and some > questions about your tables. [...] > I'm not going to have a last column. And I am using TEXT, not DATE, > as the values are not always dates. I recall reading someone > suggesting allowing unknown. If the date is unknown, I suggest > leaving this field empty. Actually, with SQLite, all columns are really strings even if one specifies other SQL types :-) Oliver requested that `unknown' be a valid value. The only other valid format is YYYY-MM-DD. > If the suggestion above is accepted, I will reconsider my DATE versus > TEXT decision as I could then use a NULL value. Yes, you should expect that `unknown' <discovery> dates are generated by Oliver or his perl script. They should all be transient values: even though such entries will technically be acceptable, they are morally bugs. :-) > > > > > > CREATE TABLE refs ( > > > vid VARCHAR > > > , type VARCHAR > > > , text VARCHAR > > > ); > > What is this type? I fear I have asked this already. I have added > the table but without the type field. This is a different type than the other :-) References are (type, text) tuples, where the type can be "url", "cvename", "uscertta", "mlist", and so forth. The meaning of the text of course depends upon the type. The list of types is constrained by the VuXML DTD, but VuXML processing tools should handle any unrecognized type generically. > > > CREATE TABLE affected ( > > > vid VARCHAR > > > , type VARCHAR > > > ); > > > > What is this type field? I didn't follow what the affected table > > contains. > > Ahh, type is either package or system. > > e.g. > > <package> > <name>tnftpd</name> > <range><lt>20040810</lt></range> > </package> > <package> > <name>lukemftpd</name> > <range><ge>0</ge></range> > </package> > <system> > <name>FreeBSD</name> > <range><ge>4.7</ge></range> > </system> Yes, the type column in *this* table is "package" or "system". > > > CREATE TABLE names ( > > > affected INTEGER > > > , name VARCHAR > > > ); > > This I now understand. It's a list of the affected packages, or > systems. For packages, it appears to be PORTNAME. Is that true? Almost. In FreeBSD ports collection terms, the <name> element contains the PKGNAME minus the version, i.e. ${PKGNAMEPREFIX}${PORTNAME}${PKGNAMESUFFIX}. > I see now why you have names and affect separate: > > <package> > <name>gaim</name> > <name>ja-gaim</name> > <range><lt>0.81_1</lt></range> > </package> > > A given package may have more than one name (e.g. slave ports). Right. Another example is compile time options, e.g. "vim", "vim-lite". [...] > Each package has a list of 1 or more ranges which are affected. The > ranges affect all names listed in the package. > > e.g.: > > <package> > <name>samba</name> > <range><ge>3</ge><lt>3.0.5,1</lt></range> > <range><lt>2.2.10</lt></range> > </package> You got it. > Thanks. This is coming together and I'm getting closer to having > tables. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Fri Aug 27 07:14:45 2004 From: nectar at FreeBSD.org (Jacques A. Vidrine) Date: Fri Aug 27 07:17:25 2004 Subject: FreshPorts - VuXML tables - DDL + data In-Reply-To: <20040827051637.T78365@xeon.unixathome.org> References: <20040827051637.T78365@xeon.unixathome.org> Message-ID: <20040827141424.GF54027@madman.celabo.org> On Fri, Aug 27, 2004 at 07:07:07AM -0400, Dan Langille wrote: > Here is the DDL I created this morning. Note that some slight changes > have been made relative to the diagram I posted last night. The diagram > is now out of date. > > The rest of this message contains a sample vuxml entry followed by the DDL > to create the tables and populate it with a single VuXML entry. It also > shows how I mark which packages are affected by a given vid (i.e. how I > populate the commit_log_ports_vuxml table). > > I have created the code to display VuXML information, but at present, the > only data loaded is for http://beta.freshports.org/ftp/tnftpd/ Good work! > NOTE: I'm assuming there can be multiple VuXML entries for a given version > of a package/port. However, there will be only one VuXML icon shown per > row of Commit History. Look at the last commit on the page for a commit > which is affected by two VuXML entries (this is achieved with a fake VuXML > entry). Yep, that's a correct assumption. > freshports.org=# select id, vid, topic from vuxml; > id | vid | topic > ----+--------------------------------------+---------------------------------------------- > 1 | c4b025bb-f05d-11d8-9837-000c41e2cdad | tnftpd -- remotely exploitable vulnerability > (1 row) [...snip...] This is all looking pretty good to me. When the beta is up, I'll be happy to keep an eye on the results for anything unusual. By the way-- a bit off-topic from the DBA stuff-- when processing the FreeBSD VuXML document, you may want to insert a validation step between fetching it and loading it into the database. This will catch problems like bad commits or truncated transfers-- cases in which you probably do not want to use the results for updating the database. See ports/security/vuxml/files/validate.sh for a simple example of using xmllint or onsgmls for validation. Cheers, -- Jacques Vidrine / nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From dan at langille.org Fri Aug 27 16:50:50 2004 From: dan at langille.org (Dan Langille) Date: Fri Aug 27 16:50:52 2004 Subject: FreshPorts - VuXML tables - diagram In-Reply-To: <412E54E5.5382.6016DEA@localhost> Message-ID: <412F9098.32178.AD2B8CF@localhost> On 26 Aug 2004 at 21:23, Dan Langille wrote: > Here's the diagram of the tables I'm considering. This should > duplicate the entire contents of the VuXML table. > > http://beta.freshports.org/tmp/FreshPorts-VuXML-tables.jpg This diagram has been updated to reflect recent discussions. -- Dan Langille : http://www.langille.org/ From will at csociety.org Fri Aug 27 20:55:30 2004 From: will at csociety.org (Will Andrews) Date: Sat Aug 28 05:15:33 2004 Subject: database tables for VuXML In-Reply-To: <717D5B12-F6CB-11D8-9236-000A95BC6FAE@FreeBSD.org> References: <4129070D.28233.7C2E4023@localhost> <4129CCC9.11697.7F32B4B4@localhost> <717D5B12-F6CB-11D8-9236-000A95BC6FAE@FreeBSD.org> Message-ID: <20040828035529.GP3862@sirius.firepipe.net> On Wed, Aug 25, 2004 at 02:17:45PM -0500, Jacques Vidrine wrote: > You accidently answered my question :-) I guess you have set those > whenever the commit mail came in ... so the view of past package > versions is from past executions of `make -V'. > > I wish I could `cd /usr/ports/some/app && make package-names' to get a > list of all package names this port ever built. or some such. That's really hard to implement, because ports are highly interdependent. To do this properly, you would need to checkout the entire ports tree n times, where n is the number of times PKGNAME has changed in the history of the ports tree. Plus, they can vary depending on options. Suffice it to say, getting all this information would be astronomically difficult. :-) Regards, -- wca -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-vuxml/attachments/20040827/33f4529e/attachment.bin From dan at langille.org Sun Aug 29 14:44:08 2004 From: dan at langille.org (Dan Langille) Date: Sun Aug 29 14:44:10 2004 Subject: parsing vuln.xml with XML::Node Message-ID: <20040829173317.U9281@xeon.unixathome.org> Hi folks, I've run into a problem parsing the vuln.xlm file. I'm using the perl XML::Node module. The issue is the body field. This field contains XHTML tags (for example, , , <blockquote>). I have been unable to extract the contents of <body>...</body>. The only solution I've found is to explicitly specify tags such as , , and <blockquote>. That is by no means an ideal solution. There must be something I'm missing. It has been suggested I use XML::Parser instead (XML::Node is based upon XML::Parser). See a working example at http://beta.freshports.org/tmp/testing.tgz it runs like this: perl load_vuxml_into_db.pl vuln.xml If someone can figure out how I can do this, it will be appreciated. Thanks. -- Dan Langille - http://www.langille.org/ From dan at langille.org Mon Aug 30 10:39:55 2004 From: dan at langille.org (Dan Langille) Date: Mon Aug 30 10:43:06 2004 Subject: vuln.xml is not XML Message-ID: <20040830133416.X35009@xeon.unixathome.org> I refer to my previous message regarding the difficulties in parsing vuln.xml. I have since learned that any markup (e.g. ) should be be in a CDATA section. See http://www.w3.org/TR/REC-xml/ and look at section 2.7. CDATA sections begin with the string "<![CDATA[" and end with the string "]]>":] I propose that markup be enclosed with a CDATA section. -- Dan Langille - http://www.langille.org/ From nectar at celabo.org Mon Aug 30 17:15:24 2004 From: nectar at celabo.org (Jacques Vidrine) Date: Mon Aug 30 17:17:35 2004 Subject: vuln.xml *is* XML (was Re: vuln.xml is not XML) In-Reply-To: <20040830133416.X35009@xeon.unixathome.org> References: <20040830133416.X35009@xeon.unixathome.org> Message-ID: <CDBCA21A-FAE2-11D8-A99F-000A95BC6FAE@celabo.org> On Aug 30, 2004, at 12:39 PM, Dan Langille wrote: > I refer to my previous message regarding the difficulties in parsing > vuln.xml. I have since learned that any markup (e.g. ) should be > be in > a CDATA section. > > See http://www.w3.org/TR/REC-xml/ and look at section 2.7. > > CDATA sections begin with the string "<![CDATA[" and end with the > string "]]>":] > > I propose that markup be enclosed with a CDATA section. No this is absolutely wrong :-) The XHTML is embedded with VuXML... the whole document is one XML document. Some elements are in the VuXML namespace, while others are in the XHTML namespace. Markup cannot exist in a CDATA section--- if it is in a CDATA section, it is *not* markup but *text content*. I saw your earlier message about XML::Node, but since I am not familiar with that (or XML::Parser), I did not understand what problem you were having. Could you try to describe it differently? Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From kuriyama at imgsrc.co.jp Mon Aug 30 17:23:21 2004 From: kuriyama at imgsrc.co.jp (Jun Kuriyama) Date: Mon Aug 30 17:26:18 2004 Subject: vuln.xml *is* XML (was Re: vuln.xml is not XML) In-Reply-To: <CDBCA21A-FAE2-11D8-A99F-000A95BC6FAE@celabo.org> References: <20040830133416.X35009@xeon.unixathome.org> <CDBCA21A-FAE2-11D8-A99F-000A95BC6FAE@celabo.org> Message-ID: <7mk6vg2m15.wl@black.imgsrc.co.jp> At Mon, 30 Aug 2004 19:15:02 -0500, Jacques Vidrine wrote: > > I refer to my previous message regarding the difficulties in parsing > > vuln.xml. I have since learned that any markup (e.g. ) should be > > be in > > a CDATA section. > > > > See http://www.w3.org/TR/REC-xml/ and look at section 2.7. > > > > CDATA sections begin with the string "<![CDATA[" and end with the > > string "]]>":] > > > > I propose that markup be enclosed with a CDATA section. > > No this is absolutely wrong :-) The XHTML is embedded with VuXML... > the whole document is one XML document. Some elements are in the VuXML > namespace, while others are in the XHTML namespace. Markup cannot > exist in a CDATA section--- if it is in a CDATA section, it is *not* > markup but *text content*. Both are correct. In good old XML world, we should use CDATA section to quote external markup. On the other hand, VuXML lives in XML + Namespace world (see related recommendations). > I saw your earlier message about XML::Node, but since I am not familiar > with that (or XML::Parser), I did not understand what problem you were > having. Could you try to describe it differently? I'm not sure XML::Parser can handle namespace correctly. If it cannot do such, parser will confuse when it reads markups with namespace. -- Jun Kuriyama <kuriyama@imgsrc.co.jp> // IMG SRC, Inc. <kuriyama@FreeBSD.org> // FreeBSD Project From dan at langille.org Mon Aug 30 17:34:19 2004 From: dan at langille.org (Dan Langille) Date: Mon Aug 30 17:35:31 2004 Subject: vuln.xml *is* XML (was Re: vuln.xml is not XML) In-Reply-To: <7mk6vg2m15.wl@black.imgsrc.co.jp> References: <20040830133416.X35009@xeon.unixathome.org> <7mk6vg2m15.wl@black.imgsrc.co.jp> Message-ID: <20040830203241.V35009@xeon.unixathome.org> On Tue, 31 Aug 2004, Jun Kuriyama wrote: > At Mon, 30 Aug 2004 19:15:02 -0500, > Jacques Vidrine wrote: > > > I refer to my previous message regarding the difficulties in parsing > > > vuln.xml. I have since learned that any markup (e.g. ) should be > > > be in > > > a CDATA section. > > > > > > See http://www.w3.org/TR/REC-xml/ and look at section 2.7. > > > > > > CDATA sections begin with the string "<![CDATA[" and end with the > > > string "]]>":] > > > > > > I propose that markup be enclosed with a CDATA section. > > > > No this is absolutely wrong :-) The XHTML is embedded with VuXML... > > the whole document is one XML document. Some elements are in the VuXML > > namespace, while others are in the XHTML namespace. Markup cannot > > exist in a CDATA section--- if it is in a CDATA section, it is *not* > > markup but *text content*. > > Both are correct. In good old XML world, we should use CDATA section > to quote external markup. On the other hand, VuXML lives in XML + > Namespace world (see related recommendations). > > > I saw your earlier message about XML::Node, but since I am not familiar > > with that (or XML::Parser), I did not understand what problem you were > > having. Could you try to describe it differently? > > I'm not sure XML::Parser can handle namespace correctly. If it cannot > do such, parser will confuse when it reads markups with namespace. With CDATA, it works, without, it fails and I have to treat every , and <blockquote> as a node, not markup. -- Dan Langille - http://www.langille.org/ From nectar at FreeBSD.org Mon Aug 30 18:25:41 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Mon Aug 30 18:25:45 2004 Subject: vuln.xml *is* XML (was Re: vuln.xml is not XML) In-Reply-To: <7mk6vg2m15.wl@black.imgsrc.co.jp> References: <20040830133416.X35009@xeon.unixathome.org> <CDBCA21A-FAE2-11D8-A99F-000A95BC6FAE@celabo.org> <7mk6vg2m15.wl@black.imgsrc.co.jp> Message-ID: <9E499E76-FAEC-11D8-84D2-000A95BC6FAE@FreeBSD.org> On Aug 30, 2004, at 7:23 PM, Jun Kuriyama wrote: > Both are correct. In good old XML world, we should use CDATA section > to quote external markup. On the other hand, VuXML lives in XML + > Namespace world (see related recommendations). If you want to quote external markup as *text*, then sure: CDATA is one way of doing that (character and entity references are another). In this case, it is just *text*, not markup--- it looks like markup but it isn't as far as XML processors are concerned. But if you want to do something with that markup (e.g. validation, XSLT) then you really must use real XML and namespaces. I guess you are probably bringing this up from the perspective of DocBook, but it just happens that DocBook--- and some other XML applications such as XML-RPC and RSS--- was born before namespaces and has not adopted support (yet). So we're left with the CDATA workaround that we had to use with SGML. This should never be done in new XML applications. This is finally being addressed in some versions of DocBook (e.g. DocBook 4.3 + SVG). >> I saw your earlier message about XML::Node, but since I am not >> familiar >> with that (or XML::Parser), I did not understand what problem you were >> having. Could you try to describe it differently? > > I'm not sure XML::Parser can handle namespace correctly. If it cannot > do such, parser will confuse when it reads markups with namespace. I don't believe that is correct. Tools that do not grok namespaces will just not see the namespaces. They will still parse the content just fine. Since we use default namespace declarations by convention in vuln.xml, it is particularly un-obtrusive: a parser will just see "xmlns" attribute nodes, but otherwise continue just fine. Basically, a namespace-aware processor will see events like these: start element (http://www.vuxml.org/app/vuxml-1/, description) attributes [] start element (http://www.w3.org/1999/xhtml, body) attributes [] start element (http://www.w3.org/1999/xhtml, blockquote) attributes [(cite, "http://...")] ... end element (http://www.w3.org/1999/xhtml, blockquote) end element (http://www.w3.org/1999/xhtml, body) end element (http://www.vuxml.org/app/vuxml-1/, description) while an old XML processor with no support for namespaces will see events like these: start element description attributes [] start element body attributes [(xmlns, "http://www.w3.org/1999/xhtml")] start element blockquote attributes [(cite, "http://...")] ... end element blockquote end element body end element description You can even ignore the namespaces if you like. You just need to "remember" when you are processing stuff inside a <description> element versus not. AFAIK, XML::Node is based on XML::Parser which is based on expat. expat supports namespaces perfectly well, so it is surprising if the Perl modules built on top of it do not. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org From nectar at FreeBSD.org Mon Aug 30 18:30:31 2004 From: nectar at FreeBSD.org (Jacques Vidrine) Date: Mon Aug 30 18:30:33 2004 Subject: vuln.xml *is* XML (was Re: vuln.xml is not XML) In-Reply-To: <20040830203241.V35009@xeon.unixathome.org> References: <20040830133416.X35009@xeon.unixathome.org> <20040830203241.V35009@xeon.unixathome.org> Message-ID: <4C792F15-FAED-11D8-84D2-000A95BC6FAE@FreeBSD.org> On Aug 30, 2004, at 7:34 PM, Dan Langille wrote: > With CDATA, it works, With CDATA, it breaks: some markup is incorrectly treated as text content. > without, it fails and I have to treat every , > and <blockquote> as a node, not markup. without, it works perfectly. That is exactly how it is supposed to work, Dan :-) I guess I understand now what you are trying to accomplish. If you want to stuff processed XML (i.e. an XML nodeset) into a database, you must first convert the nodeset into a character stream in the usual fashion. Cheers, -- Jacques A Vidrine / NTT/Verio nectar@celabo.org / jvidrine@verio.net / nectar@freebsd.org