[Machine readable output from userland utilities] report

[Eitan Adler]

On 26 May 2014 10:30, David Chisnall <theraven at theravensnest.org> wrote:
> On 26 May 2014, at 15:34, Zaro Korchev <zkorchev at mail.bg> wrote:

[ Zaro and I were convering about this on a different thread ]
>
>> I considered using libucl and libnv. The problem with libucl is that it does not support streamed output so, after a discussion with my mentors, I decided to implement this prototype version using YAJL. This is not a final decision but that's what I'm using at the moment.

The design will look like:
- applications write to libnv for serialization
- pass libnv pairs to wrapper library
- which in turn unwraps them and passes them off to formatting library
such as libucl or yajl or whatnot

>
> That's fine, as long as you're prototyping an interface for an abstraction layer it doesn't matter much what is on the back end, I just wanted to make sure that you were thinking about libucl as an eventual back end.

This should be on the roadmap.  By the end the wrapper should work
with many applications and multiple formats.  That said, as I said to
Zaro earlier, I would rather he focus on getting more utility support
than a a lot format support.  1 or 2 for the latter is fine.

> I've added Vsevolod to the cc list, as he's the author of libucl - perhaps he can add the missing functionality that you require.

Cool.

> I definitely agree that streaming is important - we want to be able to construct pipes of these, although hopefully the total amount of data won't be huge.

>> I may use libnv soon - I just haven't had need for it yet. It has one limitation that I'm concerned about - it does not support arrays (the only supported composite data type is key, value pairs).

> Arrays in libnv came up at BSDCan.  Apparently someone (Pawel?) has patches for arrays, but didn't commit them because there were no consumers in
the base system that needed them.  It sounds like you've just
volunteered as a beta tester ;-)

pjd, any comments?

> It would also be good to consider prepending a header to each stream so that tools can consume them without having to be aware of the format.  JSON has the nice property that it can be spotted quite easily be examining the first 4 bytes (in any unicode encoding).  I'm not sure if UCL and NV have the same property - if they do, then we don't need to worry.

Good point. I did not think of this.  I think it makes sense to always
prepend a 'format id' or 'module name' since not all formats are
guessable.


-- 
Eitan Adler
Source, Ports, Doc committer
Bugmeister, Ports Security teams