[GSoC] Machine readable output from userland utilities

[Erik Cederstrand]

Den 20/05/2014 kl. 18.59 skrev Zaro Korchev <zkorchev at mail.bg>:

> I'm working on the project "Machine readable output from userland utilities" and I want to share my ideas and thoughts.
> 
> [...]
> Do you have any suggestions or ideas?

Microsoft PowerShell does something similar. They print human-readable output to the shell, but pass C# (I think) object instances along when piping data to other utilities. The idea of separating formatting from data is good, but it also leads to insanity because what you see is never what is passed on to the next utility. The utilities need to support the filtering and processing you would otherwise do with awk, grep, sed etc., and you need to look up two manual pages - one for the human output, and one for the C# models. Also, you can only combine utilities that were meant to be combined. This is a huge loss of flexibility compared to what I'm used to from UNIX.

Has anyone given general thought to where the "Machine readable output from userland utilities" idea is headed? In particular, what should happen if I stick a pipe at the end of a command? Do we want utilities to understand XML/JSON for input, too? So for example 'rm' understands JSON from 'ls'. Unless this was really smart thought out, only tools that were prepared to work together would be able to do so, and you would need to look up which utilities work together.

Even if we leave input alone and focus on output, we might want to let sed, grep, awk and friends understand JSON/XML input so we can e.g. use XPath syntax to grep for certain elements in XML output, or filter certain JSON elements from the output. At least I think FreeBSD should gain command-line tools that offer at a subset of the JSON and XML functionality that e.g. Python offers when working with these data formats. Using simple awk and grep with these formats would be regex madness.

I've had long hours fighting with grep, awk and others to get simple scripts to behave properly due to the machine-unfriendly format of some tools, so I really like the idea of machine-readable output. sysctl offers some good examples of do's and dont's in this regard, and I feel that the concept of machine readability needs to be thought through very carefully to be exactly as awesome as when I first started grep'ing, awk'ing, sed'ing and piping my way through the UNIX shell.

Erik