svn commit: r284198 - head/bin/ls

[Slawa Olhovchenkov]

On Sun, Jun 14, 2015 at 11:22:03AM +0100, Bruce Simpson wrote:

> On 14/06/2015 11:00, Slawa Olhovchenkov wrote:
> > On Sat, Jun 13, 2015 at 05:13:31PM -0700, Craig Rodrigues wrote:
> >
> >> The people I talk to use scripting languages like Python or Ruby,
> >> and devops frameworks like Ansible, Saltstack, Puppet, and Chef.
> >> They may do some quick prototyping and UI work with Javascript and HTML/CSS.
> >> Being able to generate JSON directly from system-level tools,
> >> and then analyze that in a Python script,
> >
> > You need JSON from system-level tools for analyze that in a Python
> > script? Realy? Not plain text? or tab/space separated?
> >
> 
> So, I am broadly in favour of keeping libxo -- providing the problems 
> with its introduction are fixed.
> 
> I'm not even thinking of the "DevOps" space, but the R&D space, where 
> Python and R are king.
> 
> Man pages are small beer and can be dealt with easily.
> 
> Control flow and regressions from previous functionality are another 
> issue, and I am not for a moment going to duck out and suggest those 
> responsible for introducing it aren't also responsible for fixing these 
> specific issues.
> 
> But I have yet to see a coherent argument here -- size(1) numbers, RSS 
> figures etc. -- about how it allegedly adds bloat. Most of what I've 
> seen so far is POLA, NIH resistance, and hand-wavery.
> 
> If anything it helps to future proof the code as it stands, and make it 
> easier for us to actually engineer the system for performance.
> 
> I tend to look upon counter-arguments against that last point as "The 
> more we obfuscate, the harder it is to get found out that the code isn't 
> actually that good."
> 
> So, if you read my previous response to this thread, I've clearly 
> pointed out that: there are specific problems in parsing the output of 
> these system tools as they stand.
> 
> If you don't believe this, you can have my yesterday morning/afternoon, 
> where I was post-processing the output of 4000 individual "vmstat -z" 
> and "vmstat -m" reports.
> 
> Another 1000 this morning, with more to follow. "Oh boy," I'd say to 
> myself, "I wish I had libxo!"
> 
> This argument is not limited to base system utilities. For example: 
> iperf 3.x has had a similar reworking of its reporting format to include 
> JSON.
> 
> This is a massive improvement over iperf 2.x, which does not even 
> clearly document its CSV fields; you have to read the source for that. 
> JSON actually tags each field.
> 
> This reduces the time from experiment to analysed result significantly, 
> just because I can easily see what each god damned number means.
> 
> Given, you need to read the source to understand why its naive 
> sequencing algorithm breaks in multipath networking scenarios, but one 
> should not have to do this just to get basic throughput results tabulated.

Post-processing massive outputs and individual outputs is different
task. Anoder one -- one-line scripts. I am don't like some aspects of
ls/find outputs (for example -- date/time format changed depends of
age). Some utilites already have key for script-processing (zfs -H,
for example). I am don't see how libxo integrated to ls, but I see
that in other utilites format changed options don't tigh integrated:

 # zpool list -v 
NAME          SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank         46.2T  37.3M  46.2T         -     0%     0%  1.00x  ONLINE  -
  mirror     2.72T  2.15M  2.72T         -     0%     0%
    da2          -      -      -         -      -      -
    da0          -      -      -         -      -      -

 # zpool list -v -p
NAME          SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank         50818053046272  39088128  50818013958144         -     0%      0  1.00x  ONLINE  -
  mirror     2.72T  2.15M  2.72T         -     0%     0%
    da2          -      -      -         -      -      -
    da0          -      -      -         -      -      -

only 'total' format changed, not components.

And this is more complex problem for scripting/automating that JSON/plaint text.

Also, I am don't like pyton. I am use perl. For JSON parsing I am need additional modules.
I am can't easy use JSON output in one-lines shell pipes -- system utilites (like sort, awk, cut
and etc.) don't understand and parsed JSON. But understand tab/space/null separated stream.

vmstat have slight different problem -- no easy filtering.
gstat can't output in file/stream in looped mode.

All this problems don't resolved by JSON output.

I am don't talk about need or don't need libxo, good or bad code in libxo.
I am just about needing json output for system utilites.

I am see tons of utilites. Some utiltes born in FreeBSD. Some born in relative projects
(NetBSD, OpenBSD). Some GNU. And etc. Who do JSONification all of this? Who ported to libxo ZFS
utilites? clang oputput?
Or some utilites will be JSON, some -- not?
As I see, JSON output good only for Ansible and direct output to web interface.