Interesting examples, especially in light of the recent debates about Unix shells and PowerShell.
In PowerShell, the `ls` command is a simple command that produces output without flags to control output format, ordering or filtering. Like with higher order functional programming, those tasks are delegated to cmdlets like Format-List, Format-Table, Select-Object, Where-Object.
Some PowerShell examples:
To list the files and directories (children) of a directory:
ls
To get just the names or the full path:
ls | select Name
ls | select FullName
To discover which properties are available from `ls`:
ls | gm
To select only files larger than 200MB:
ls | ? Length -gt 200MB
(ls is alias for Get-ChildItem, gm is alias for Get-Member, ? is alias for Where-Object)
It is somewhat ironic that PowerShell is more true to the principle of "do one thing and do it well" than the Unix shells.
I recently discovered that I can do this on Linux, too!
After over two decades of using Unix and Linux, I ran into jq, a tool for querying and transforming structured JSON pipelines: http://stedolan.github.io/jq/
This can be used to do many of things you demonstrate with PowerShell. Here's an example from the docs:
{
"name": "Nicolas Williams",
"message": "Add --tab and -indent n options"
}
... more JSON blobs...
You can output a series of individual JSON documents, or convert the output to a regular JSON array. You can also output raw lines of text to interoperate with existing tools. And this works especially well with commands like 'docker inspect' that produce JSON output.
I think that PowerShell and jq get this right: More command-line tools should produce explicitly-structured output, and we should have powerful query and transformation tools for working with that output. For day-to-day use, I do prefer jq's use of JSON over a binary format.
I've written a tool inspired by jq that builds on top of ramda and LiveScript. The motivation behind it was to be able to write pipelines with tools that already exist, instead of having to learn new syntax that can't be used outside that domain.
That's cool, though powershell deals in actual objects which are more powerful than better structured input and output. For example, the objects returned by ps have methods for interacting with them.
I would argue the object approach limits the universality of the PowerShell way as a general purpose computer interface because it binds it to the necessity of a particular flavor of object system (e.g. .Net) and mutable data which is venom to general purpose pipe and filter systems.
jq looks very interesting though note that it builds upon an underlying notion of text to serialize JSON.
You can parse streams of text in PS too, so it's not like cmdlets are making the pipeline any less powerful.
As for binding to .NET, I don't think that's very limiting. A surprising amount of stuff is already easily hosted in .NET.
I would argue that all PS needs to be more competitive is: Ports to Linux/*BSD/OSX and a terminal implementation for Windows that doesn't suck.
Cmd.exe is a piece of shit that needs to die:
- Command editing. Is support for at least ^A, ^E, ^W, and friends too much to ask?
- Completion. Who wants to cycle through 4253 possibilities one at a time?
- Copy/paste. Programs actually STOP executing when you select text, like as if you ^Z in Unix. Even with quick edit enabled so you don't have to ALT-SPACE-E-ENTER<select stuff>ENTER, the fastest way to paste is 4 keys: ALT-SPACE-E-P.
- No SSH. Microsoft is addressing this. It's borderline criminal that Windows doesn't just ship with OpenSSH server already.
- No screen/tmux. I can't even talk about how it deals with resizing without using a lot of profanity.
- Lack of echo by default is seriously annoying.
In short, make the terminal feel like Putty and the editing/completion features feel like bash and I think PS could give all existing shells a run for their money.
As a Linux user / developer, it's surprising to hear colleagues talk about how important it is to separate content from presentation regarding, say, TeX, but ignore the benefits of Powershell doing the same. Unix users actually expel cycles trying to express things like times and file size as regular expressions to operate on strings, rather than dealing with them using their real data types.
A big part of this is that remoting has sucked - most webb app developers don't use Windows - so a lot of people who should have been able to find out for themselves hasn't been able to. Microsoft supporting SSH (as they announcedd recently) should fix that.
it's surprising to hear colleagues talk about how important it is to separate content from presentation regarding, say, TeX
This debate never ends because, like the static vs dynamic types debate, it ignores the main tradeoff in favour of a one-size-fits-all solution. In both debates (separation and types) the tradeoff can be expressed most generally as an up-front vs deferred time investment. Static types require more up-front time investment with the advantage that they save time later. The same goes for having separated content from presentation.
Now, in light of the tradeoffs above what is the appropriate choice to make? That depends on how much time you're going to spend writing the script/document/whatever and how much time you're going to spend running/maintaining it later. For scripts, it makes no sense to have a heavyweight type system if you're only going to run the thing once and throw it away. Likewise, for documents it makes no sense to put in the extra time planning all the styles if you're only going to make a document (say, a shopping list) and then throw it away. For a long-term document such as a book or a thesis it makes a ton of sense to use something like TeX.
> Static types require more up-front time investment with the advantage that they save time later.
What exactly is it that saves time? Is it to not have to type as much? In my experience if you use a lang with good type inference you type a lot less with strict typing than without, since you don't have to have unit tests to guard for things like typos/param order/param types.
Typing on the keyboard is not what takes up most of the time in programming; thinking is. With static types you are forced to put more time into thinking about how to make the program compile whereas in a dynamic scripting language you only care about the result output. Your script may be ill-typed but if it gives the correct output for that situation then it doesn't really matter.
Static types save time later by giving you the opportunity to do a lot of refactoring with high assurance of correctness. Unit tests bring back some of this ability to dynamic languages but they require extra time to write and maintain the tests.
I just can't think of many situations when a dynamically/weakly typed program produces a meaningful result in that case. In my experience the errors are the same in both the (strong/static) case and the (weak/dynamic) case, with the only difference being whether the issue shows up at compile time or runtime. Dealing with N runtime issues is slower than fixing N compiler errors when you discover runtumr problems one at a time, but compiiler errors you get in batches of up to N.
Yes if everything is untyped then types are less useful. And for interactive scripring, compilation feels a bit strange too.
(Of course, I think text-only Unix tools as a way of interacting with a computer is a fundamentally broken idea).
Powershell is an interesting idea, though maybe not the best implementation of that idea.
"Yes if everything is untyped then types are less useful."
Everything sharing a common representation does not make types less useful. I would rather my program crash then confuse Bid and Ask prices.
"And for interactive scripring, compilation feels a bit strange too."
I wonder if you've used a REPL for a typed language. I agree that peeling out a compilation step would be odd, but adding types doesn't actually have to change the interaction that much.
You can always choose not to use types even when you have them. If a method has multiple args of the same (system) type it's poorly typed, at least in a language that allows type aliases.
Linux/Unix can't even get the concept of "one thing" right. Sometimes it's separated by spaces, sometimes by lines (and then you need the IFS gore to make it behave the way we want)
I mean, the shell is great, once you get around all those idiosyncrasies, but we can evolve.
However, after a few years of using both, my conclusion is that PowerShell isn't a better shell. It is programatically superior (for scripting) but inferior as a CLI language. And your examples are one of the reasons why.
"Do one thing and do it well" is about the goal of the tool, not about its features. With all its options, `ls` is still just about listing directory contents.
And that motto is still just a guideline, not an absolute rule. What makes the *nix shell great is that it follows that motto enough to be sensible and structured, but not enough that it becomes a burden. Just like a human language does.
ls is an alias for Get-ChildItem which is simply "get me a list of objects attached to the path". That's about as orthogonal as you can possibly muster.
If only it wasn't hosted in the awful cmd style terminal with it's frustrating style of edit functionality and lousy completion. If it had the feel of a Unix terminal I think I could get used to it.
I am really curious about how much parsing and formatting occupy *nix source code. (IIRC 30% of ls), and if it would be a good idea to decouple the way you mention it. my json-infused distro is still on my mind.
ps: this lisp machine talk mention how 'programs' (I guess functions) exchanged plain old data rather than serialized ascii streams making things very fast. There are caveats of adress space sharing though but it's another nice hint we could investigate other ways.
That's an interesting point. abstracting out parsing/serialization from the data piped between commands would lead to more consistent argument handling for all commands.
somebody wrote docopt (started as python lib) as a generic POSIX usage string parsing (part of the standard). Maybe it could lead to simpler argument parsing and 'user interface' generation, whether static documentation or shell completion scripts.
Also structured output may lead to more relation tools, less regexful ad-hoc parsing, maybe some kind of typecheck so callers can be notified when they need to be rewritten.
This will emit "deserialized" objects from myrouter. PowerShell remoting works across machine boundaries by defining an xml based format for serializing and deserializing objects.
During serialization, all properties are serialized, recursively to a configured max depth. The local client will see the deserialized objects - ie they are not "live" any more.
Note that the PS team recently announced that they would support SSH as a remoting mechanism. I suspect that they will still send CliXML objects when using powershell to powershell communication.
Powershell uses an industry standard for http based remote commands. It should be adaptable to SSH.
Or `irm myrouter {ls c:\logs} | ? length -gt 50kb` to match the original example, right?
Serializing structured data is a pretty solved problem, and I wonder if decades of UNIX going out of its way to screw this up is confusing the parent poster. XML is certainly a way to do it that works just fine, but there are a million others. None of them require awk or counting spaces.
> It is somewhat ironic that PowerShell is more true to the principle of "do one thing and do it well" than the Unix shells.
It's not particularly surprising that a system designed a long time after the original is more consistent (I wonder how the Plan 9 shell fares in this regard?).
Another UNIX principle is that programs deal with text. PowerShell is huge violation of that principle. You can't easily write PowerShell command with C or Rust or Java, as far as I understand. Yet you can write command-line tool with any language and use it with combination of hundreds existing tools.
Except what about Dates? Or apps were floating point precision is important? Or needing to deal with very large integers? Also, have you thought about how large the output would be for ls as JSON? Think about the I/o usage if every time you invoked ls it had to do a deep walk of the file system to get all the possible data ls might output with all its command line arguments.
Methods are useful to organise and abstract programs, they're really quite bad at composition. Many functions operating on the same primitives is much more composable than small sets of functions operating on their own custom primitives.
Behaviour could be implemented by shell commands. That way, if you wanted to implement a method for a particular object, you could just write a binary/shell script/whatever that reads the object from stdin and writes its result to stdout
I don't understand your point, since JSON is a textual format.
Sure, the fact that Unix chose text meant they created a lot of different formats, some non-standard, but I don't see this being an issue, except for configuration purposes.
Do you deal with the serialized JSON text directly, or run JSON.parse() and JSON.stringify() to turn your JSON into objects?
Do you ever use regular expressions to parse unstructured text when using a Unix shell? Do you think 'grep' and thinking of a regex is more or less efficient that using 'where'?
Exactly. His point is that whatever operations you perform from that point on is on the object and not the string that was used to represent the object as it was being communicated. The string is just an intermediate representation. It's the object that's relevant to your business logic.
Fair enough. I was coming at this from my current experience where a couple of different programs I have grab JSON from a server: the Python one puts it into an object, the bash script doesn't because I couldn't be bothered :)
JSON is not a textual format. What it is, is right there in the name: It's an Object Notation. Now, JSON objects are most frequently serialized to text, but they are still objects. The format is an orthogonal matter.
Why does it matter whether the exchange format is a regular or a context-free language? It is a textual representation of binary data either way.
Sure, it is harder to pattern-match context-free, and we have a convenient syntax for textual regular languages, but we create tools such as jq so that we can pipe JSON and extract data.
PS will take the text output of any command line program and turn it into .Net strings that can then be manipulated like any other object. You can write PS cmdlets in C/C++, there is an SDK. It will also marshal COM objects into and out of the appropriate .Net types.
When dealing with output from command line apps I usually take the time to parse out the data I want from the strings and turn them into strong types if I'm writing a script...If I'm just trying to get something done with a command prompt I just do whatever gets it done the fastest.
In the 2012 R2 time frame Jeff Snover said that they opened up cmdlet authoring to subsystem teams to use C++ to build cmdlets. Maybe they haven't released it yet?
That may mean that you have to implement at least some part of it as a .Net class. That may mean that they are doing COM components that inherit certain interfaces...it may mean that they are doing PInvoke...to be honest I haven't looked into it. It may be that it's still internal. Huh. I should look that up.
Half way through a powershell script you can switch to C#, VB, JavaScript if you want and implement a pipeline or shell out to a C++ program, talk to something over the network or even Cygwin if you really want.
I'm not familiar with PowerShell, but wouldn't following through with this principle mean that the default program has to give maximally verbose output to the piped formatter, since the formatter can only filter rather than extend the original command.
I imagine to achieve the default functionality your command must be significantly more verbose, if this philosophy is followed through 100%, or you have a lot of per-configured default formatters (and you have to remember what to pipe to which). Maybe I'm misunderstanding this, it's very neat in principle though.
PowerShell cmdlets return (active) objects in the same process space as the hosting process (usually the shell). As such they can expose expensive-to-compute properties as late-evaluated properties that are only evaluated if and when they are invoked later.
Take for instance Get-Process:
$p = ps powershell
(ps is an alias for Get-Process). Now $p is the process information onject about the PowerShell process itself.
If you request the content of $p at some later point in time, you will notice how the CPU has increased, for instance. The point is that a lot of the properties are computed when read, and hence do not incur overhead in the pipeline.
> PowerShell returns objects, not strings. Which means they can have as much information as you like without being overly verbose :-)
How does the user know exactly which output he is going to get? The formatting program cannot know anything about default expected output - so it either must be specified explicitly, or objects must distinguish between default_for_formatter_to_output fields and more_verbose_hidden_fields - I'm really not a fan of the amount of man <command_name> using Unix involves, - but is the alternative really much better?
There are some sane defaults in the PS case along with heuristics that try to map arguments between pipe-out and pipe-in cmdlets. The parsing/serialization and formatting are done by the environment, not by an individual program.
It works pretty well. In the end there is also some convention that helps it work more consistently.
In practice, it is kind of cool to be able to construct new objects as the pipeline progresses so that can customize that behavior. I've found that with some aliases for default commands it can be reasonably succinct as well.
Each object specifies how to format itself by default.
So ls, by default, gives Mode,LastWriteTime,Length,and Name.
But if I check all of the properties I get PSPath, PSParentPath, PSChildName, PSDrive, PSProvider, PSIsContainer, BaseName, Mode, Name, Parent, Exists, Root, FullName, Extension, CreationTime, CreationTimeUtc, LastAccessTime, LastAccessTimeUtc, LastWriteTime, LastWriteTimeUtc, Attributes
The reason that a program should strive to be unixlike is portability. The nice side effect is little bloat. Being unixlike is about the scope of a program rather than trying to make the implementation as basic as possible to the extent of not taking flags. If a program relies on another program to be useful, that's clearly not portable. It's no different if it's a library or another executable.
These cmdlets are excessively small in scope and that makes them intrinsically tied to one another. They're useless without their sibling cmdlets. In effect they're one giant program.
So I completely disagree that these are more unixlike.
I do wonder why they use TitleCase for somethings, and lowercase for others ("ls", "gt"). I think it kind of makes this non-obvious and non-predicable. (reminds me of PHP a bit).
>I am not sure how many of the commands have been converted
There's a page [1] on the FreeBSD wiki with a list of converted and ongoing conversion. I don't know if the list is complete but the page was last modified 2015-05-22, so it should be fairly up to date i guess.
Except how do you represent every possible data type without losing fidelity? I made this point in another leaf, but it's worth echoing again. A perfect example of this is Dates. There isn't a defined JSON type for that so we'd be back to basic text parsing again (which will cause anyone who has dealt with parsing dates to recoil in agony).
When you try to ram a one size fits all approach to everything, you end up with abominations like SOAP with XML. I simply don't get the fascination with trying to use the same tool for every job even if it's not applicable.
Text streams are just an ad hoc structured format. Even an out of date format is better than one you have to invent or decode on a per case basis.
The whole xml vs json debate feels pretty pointless, they are for all practical purposes equal (in size, complexity etc). Sure xml has some weird design decisions around namespaces and so on, but if you use it as a simple hierarchical markup it's pretty much equivalent to json with different brackets? and often easier to parse for humans because of closing tags instead of }}} and of course, comments.
The xml-hate I think isn't really xml-hate it's the reaction against the xml-y things from 10-15 years ago: the IBM/Struts/Blah we all had to wade through. These days it feels frameworks pick json over xml even when it's clearly an inferior choice (such as for config files). Json is an object/message markup, not a good human readable config format.
>> Ten years ago this would be --xml. Who knows what it would be ten years from now. Text streams are timeless
> Text streams are just an ad hoc structured format. Even an out of date format is better than one you have to invent or decode on a per case basis.
I won't argue which one sucks more, XML or JSON. They are both inferior to what you call "ad-hoc structured" text files.
XML and JSON are both hierarchical data models. It has been known for fourty years that these are inferior to the relational model because they make presumptions about the access paths of the consuming algorithms.
Put differently, hierarchical data models provide just a view of the information that is actually there. (Implicitly -- consistency and normalization are not enforced). Relational databases on the other hand are concerned with the information and provide much better mechanisms for enforcing consistency and normalization.
By coincidence, the "unstructured" text files in Unix are just miniature relational database tables. Think passwd, shadow, hosts, fstab, ... . Consistency is not technically enforced (that would be a huge overkill at this level of abstraction), but there are even checker programs like pwck.
A good standardized relational model format would be cool, and I'm sure such formats exist. Feels like we could do better than spitting out randomly (I.e per- tool) formatted data with so-so encoding support!
A sequence of tables in csv with decent encoding support would go a long way towards a good machine parseable relational text output.
It's really two separate discussions though: what is s good input/output format for Unix-like tools, and what makes a good format for a config file.
> It's really two separate discussions though: what is s good input/output format for Unix-like tools, and what makes a good format for a config file.
I don't see where these are not one single problem. Everything is a file.
> A good standardized relational model format would be cool, and I'm sure such formats exist. Feels like we could do better than spitting out randomly (I.e per- tool) formatted data with so-so encoding support!
I'm actually currently trying to realize such a thing in Haskell, for usage in low traffic websites. There are obvious advantages in text DBs compared to binary DBs, for example versioning.
But I doubt we can do better than current Unix text files if we don't want to lock in to some very specific technology.
> I don't see where these are not one single problem. Everything is a file.
A very general format could solve more problems but as I said earlier I think the lack of comments in json makes it sub par as a config format for human editing.
I generally like the idea of tabular text files where the last column is free form text.
If you need more commenting freedom or flexiblity, why not make another indirection and generate the data from some source which is tailored to your needs? After all, relational data is often not suited for manual input. It's meant for general consumption by computers.
For example, as a sysadmin, passwd / shadow / group is not enough to model my business objects directly. I keep my business data in a custom database, and generate the text files from that.
I really don't care what a format is called so long as it fulfills the basic requirements: 1) can be written and parsed with the std libs of the language in question, and 2) supports comments if it is to be used by both humans and machines, such as in config files.
How would that work for late-evaluated data? A PowerShell object doesn't have to provide all of the data up-front, and can give updated data when checked at a later date. JSON is still just text, it still needs to provide all of the data you might need up front.
Not necessarily. Text streams don't have to by default provide all possible data for the next process in the pipe. Sure, you could keep all the command line arguments you had before to make JSON output manageable, but then you have two problems rather than one.
Exactly. I think the Unix philosophy of "everything is a file" has an impedance mismatch with "streams of text". Files and directories are hierarchical lists of objects with certain properties, similar to JSON.
The "directories of files" abstraction is versatile and useful, simply because it is a versatile and simple data structure. Streams of text are too limited.
In PowerShell, the `ls` command is a simple command that produces output without flags to control output format, ordering or filtering. Like with higher order functional programming, those tasks are delegated to cmdlets like Format-List, Format-Table, Select-Object, Where-Object.
Some PowerShell examples:
To list the files and directories (children) of a directory:
To get just the names or the full path: To discover which properties are available from `ls`: To select only files larger than 200MB: (ls is alias for Get-ChildItem, gm is alias for Get-Member, ? is alias for Where-Object)It is somewhat ironic that PowerShell is more true to the principle of "do one thing and do it well" than the Unix shells.