In my last blog post, I discussed a few issues I see with Linux as an operating system (as opposed to the higher-level stuff; eventually I’ll put a few words into that, but right now I’m still running with an interesting idea drawn out of a question from a friend of mine: “how do you fix Linux?” In my last post I listed three major “spaces between” in Linux software.
- The space between user paradigms. There’s a divide between the developers who use the command line and those who use the GUI, and invariably the users who use the paradigm the developers don’t favor get the short end of the stick.
- The space between languages. Interoperability with various languages is pretty weak, and it results in a lot of work being lost because of it.
- The space between programs. The tool-and-filter philosophy of the “Unix way” is nice for smaller, easily grasped problems (example:
grepping through a file for instances of a phrase, then piping that intotailfor the last 20 instances of it). It rapidly grows hairy and ugly when you start getting into more complex problems.
In this post, I’d like to put forth an idea that, while a fairly huge prospect, I think would be a great leap forward for Linux.
As a couple of commenters mentioned in the previous post, it sounds like what I’m describing is a managed code solution. It will then surprise absolutely no one that that is in fact what I’m talking about.
I’m not talking about something absolutely foolish, like a managed code kernel or something equally cretinous. Nor am I talking about piling up C coders like cordwood and setting them alight (I mean, come on…all that burning hair? Enough to make you start retching…). What I’m talking about is an expansion of the capabilities of the standard Unix programs (and other Linux-specific ones) to support newer methods of interaction with the standard Unix tools, methods that empower users and developers alike. I’m talking about replacing the current Unix tools packaged with most (all?) Linux distributions with managed code* equivalents.
.
Let’s start, first, with the three points I made above, and discuss how a managed code framework would benefit developers and end users in all of those areas.
- The space between user paradigms. When using Mono, at least, there’s a bit of a push toward writing more modular applications; it “falls out” of the framework and quickly starts to feel natural. Somebody writing a tool in Mono isn’t (usually, hopefully) going to just parse the command arguments and work their magic in Main(). Instead, they’ll break the functionality out into an object (or a bunch of them). The natural mapping that comes to mind is that of the command line options to properties on their “control” object, and it becomes a lot easier for a developer–whether it’s the same person who made the CLI application and library or not–to put together a GUI application to provide the same functionality.
- The space between programs. (Yeah, I swapped up this one and the languages issue, because in this case user paradigms and programs are pretty close.) Like I said before, pipes are a great method for people who want to use them–more power to you if that’s your thing. It’s not mine and it’s not a lot of other people’s. And with managed tools, we both win. You still get your command line applications. You can
catandteeto your heart’s content. Meanwhile, I get the code objects that you on the command line are using wrapped with a command line interface and I can deal with them directly, as I prefer to do. Everybody wins.And I realize that this sounds a little pathetic at first blush, but when you aren’t stuck piping text around and can work with actual objects, debugging becomes easier. You don’t have to wonder why your output file is blank just because you didn’t order the command line arguments correctly or the tool didn’t like the way you phrased one of the settings–instead an exception can be raised with actual valuable information that can help you solve the problem. Just try and tell me that doesn’t sound awesome. - The space between programming languages. This one’s a meatball, whether you’re talking about the Java Virtual Machine (JVM) or the Common Language Runtime (CLR). Offhand, I can’t think of many even remotely popular languages that don’t have CLR implementations either in the works or complete. I’m still convinced that C# is the greatest thing since sliced bread, but just off the top of my head I can think of C++, Python, Ruby, Scheme, and PHP. Heck, there’s Prolog, Ada, Delphi, Object Pascal, and at least two or three COBOL implementations that Wikipedia knows about (in fact, click here for their list of languages running on the Common Language Infrastructure, though I don’t know which ones are supported by ). The killer feature is that objects written in any** of those languages can be used in any of the others. If the maintainer of the managed version of
ifconfigwants to write it in Python, more power to him! I don’t have to care what language he’s using; so long as the objects necessary to manipulate network interfaces are made public, I just have to reference his libraries and I can issue the in-application equivalent ofifconfig eth0 upall day long. Even Java developers can get in on the fun. I don’t know of a Java-on-Mono compiler (though one may exist that I don’t know of; it’s definitely possible, as Microsoft does it with J#), but there’s IKVM to run JVM code on Mono. Though I don’t know whether there’s bridging to .NET objects; Java’s not my thing.
This shouldn’t be construed as an attack upon the current stuff that’s there, because those tools are time-tested and good for what they are. Nor should the people who like the way it currently works be immediately hostile to the idea, because I can’t think of a reason they’d see any real difference. There’s no reason the command line flavor of the tools couldn’t mimic the GNU tools or the BSD tools or whatever else; people who choose to use them just like they always have*** can do so.
Now, I know that most of the functionality afforded by the various low-level Unix programs is available in Mono itself. That’s not the point. The applications themselves are important because they encapsulate known, well-used functionality in a package that everybody’s already familiar with. The ability to get all the data you’d normally get out of, say, ifconfig, in a format that you can actually work with programmatically without a bunch of ugly hackish text processing, is, to me, a big win.
.
Believe me, I recognize the huge amount of work this sort of thing would entail, though I think it’d be less than I’m estimating due to having something to look at. It’s not a glamorous project, either; ripping the plumbing out of a house never is. And it’s definitely only a start; even if this were accomplished, there’s a lot more out there. But I think it’s definitely a project that would benefit Linux as a whole. I don’t mean to be flip, but managed code is the future wherever performance can be “good enough” instead of “absolutely critical.” Linux needs to move it along.
I’m not trying to say I’m going to start this project. I don’t know if I have the chops to do it. But if you’ve got ideas or suggestions on the topic, or might be interested in working on it, drop me a line. At the least, it’s an interesting discussion topic.
.
.
* - When I say “managed code,” you can take that as “Mono” if you really want to. Writing these replacement programs in languages targeted to the JVM would be totally doable, and I’m not knocking the JVM, its developers, or those who use languages on it–I just can’t stand Java after using C#. I also develop on Windows, so C# and the CLR are a natural fit for me.
** - There are occasional exceptions where the object model doesn’t yet translate perfectly between CLR languages, but work’s regularly underway to round off those sharp corners.
*** - About the only difference might be in startup time. If the libraries need to load, starting up a Mono application can be a bit slower than a native code application. I can think of a number of ways to solve this issue off the top of my head.
![[del.icio.us]](http://images.del.icio.us/static/img/delicious.small.gif){kind=small}
14 Comments
This sounds like a great idea, really. It would also potentially have the effect of bringing the “UNIX” way of doing things to the Windows side in a meaningful way, making windows more usable and ultimately bringing windows users to Linux.
Cheers!
JustACoder:
Your reply is kind of funny to me, because “the UNIX way” is a little bit of what makes my teeth sometimes grind when using Linux. The UNIX way is good, if your transfer medium is pipes and sockets. I personally would rather have the ability to consume the applications entirely if I need their functionality, rather than having to communicate with separate programs.
The issues with bringing a Mono-based set of base tools (i.e., the stuff Cygwin packages) over to Windows would lie mostly in the API calls (Mono.Unix handles file information operations where you need to know the permissions, for example, and there’s no analog on Windows), but it’d make sense that it’d be doable.
Not only would this be great for the reasons you mentioned, but it would make it easier to run older applications on a newer distribution without having to port it to use a newer library, as the CLR was designed to allow for multiple versions of a given library to co-exist.
It would seem this would solve some of the problems for 3rd party vendors developing applications for Linux.
> it would make it easier to run older applications on a newer distribution
> without having to port it to use a newer library
Y’know, I didn’t even think of that. Versioning in the GAC would definitely be a plus.
Cretinous? Is this what you think about sharpOS devs?
> Cretinous? Is this what you think about sharpOS devs?
I didn’t even know somebody was trying it (though I guess I should have expected it).
And no, I’m not talking about the developers, I’m talking about the concept. I don’t think it’s a good idea. I think a managed code kernel is foolish, if only for performance reasons–I don’t think it’s a terribly good idea to be building everything *else* upon a relatively slow kernel if it can be avoided. There remain a lot of benefits to native code, and I can’t think of a decently-performing way to make up for those benefits with an all-managed system (though I’ll definitely concede I could be wrong). I guess it’s an interesting project from an academic point of view, but Linux is being used in production environments. Suggesting something like a managed code kernel would be cretinous of me.
No, no, no, no! You don’t need Mono (ie, .NET) to have the features you want. (Ugh. Ptooey!). Read my comments in the first part of your blog article. You just need a language-neutral interprocess communication protocol like ActiveX. Actually, Linux already has one of those. It’s called “DBus”. The problem is that no one knows how to use DBus except for the guy who wrote it, and those poor souls who spent countless hours pouring over piles of scantily commented code to try to figure out how it works and how to use it (putting in so much time that they probably wouldn’t have taken more time if they instead wrote their own interprocess communication from scratch). Document Dbus, and add a “scripting host” on top of it, and you’ve got want you need. And there won’t be any need to deal with C#, Win32 APIs, or anything but what is already available for Linux.
Yeah, MS and 3rd party sources were savvy enough to realize that if you don’t document some API/tool well, and provide good tutorials/examples of its use, then you may as well not have released the API/tool at all. So there’s lots of good .NET documentation/tutorials/examples upon every webpage you hit during your google search. And that’s undoubtably why you think it’s the only game in town. It may as well BE the only game in town because hardly anyone knows anything about any other alternative.
But do a google on DBus and every page you hit has the exact same bit of “documentation” (if you can call it that) — often with the _exact same wording_ — because it was cut and pasted from the DBus author’s techno-babble “documentation” of DBus. And nobody else knows anything more about it, so that’s all you _ever_ get. That’s all ANYONE can tell you about it. So as far as anyone knows, it can’t do any of things that you’re describing in your blog. In fact, most programmers can’t even tell you what it does… period. They think it’s some “udev utility”.
Why is that? It’s the docs/tutorials/examples.
And why do you think that Mono is the only game in town? Again, it’s the docs/tutorials/examples. You’ve been clearly told what .NET is, what it can do, and shown how you use it. So you know.
But you don’t need .NET to do all this stuff you’re talking about (unless you really do want the automatic garbage collection and psuedo-compiler — not that you can’t get that stuff with some non-NET languages if that’s what you want). You just need someone to tell you what the alternatives are, what they can do, and show you how to use them. And that latter thing is a very big problem in the Linux community because of the appalling state of docs/tutorials/examples — where the original author poorly documents his work, and then every Linux guy in the world (and his dog) sets up a mirror site to host that very same poor documentation so that, if you try to google for good docs, you end up spending months surfing from page to page reading the same damn useless crap “documentation” cut and pasted from the same source. Argh! Someone shoot these people, please!
I don’t want to sound as to be thinking I’m the only one I own the truth but:
jg: Without agreeing/disagreeing with you about the documentation issue you arise, you seem not to know the difference between inter-process communication and inter-program communication. Please keep reading. It’s not all about IPC.
Ed: AFAIK, the kernel is written in managed code but not interpreted this way, but with the help of AOT, it’s compiled to native code at compile time (no JIT). For more info, read sharpos docs.
>It’s not all about IPC.
Yes it _is_. You _can’t_ have inter-program communication without inter-process communication. Period. That’s because every program runs in its own process. Period. A pipe is a one form of inter-process communication. (And it’s a really primitive, slow one — usually not supported by GUI programs — as the orig author found out when he was piping the input/output of data between programs). Since all inter-program communication _must_ (and let me emphasize that again… _must_) be based upon some sort of inter-process communication, then the state of inter-program communication is _directly_ dependent upon the state of inter-process communication. In other words, you _can’t_ have this “managed environment” you’re talking about _until and unless_ you have widespread support for something other than a really crappy inter-process communication like pipes. And you ain’t gonna get it no matter how much you talk about it, until existing implementations are much better documented. Never.
So for example, if the primary form of inter-process communication is pipes, then:
1) It isn’t going to be well-supported by GUI software. GUI software is event-driven, so the main thread can’t just make a call to some _potentially blocking_ I/O routine like read(). It can’t even do a select(). (Linux doesn’t have what Windows calls “overlapped I/O”). You need to spawn a separate thread to deal with that stuff. And multi-threaded programming can be complicated, and is overkill for some software.
2) It’s going to be SLOW and primitive. So don’t expect any sort of complex language-neutral communication protocol (ie, what you guys are describing, but mistakenly calling it a “managed environment”) to be built upon it. That would suck.
And yet, pipes are in fact exactly what most Unix/Linux software uses for inter-process (ie, inter-program, if you prefer) communication. Why? Because it’s available upon all platforms. But mostly, because the docs/examples/tutorials for every other alternative are so bad, very, very few programmers know how to use anything else. You can find lots of lots in detailed info/examples about using pipes. Using DBus? Good luck. And of course, Dbus isn’t available everywhere, because the people who would port it to other environments don’t know how it works, or what it does, either.
I wrote some Linux software that allows programs to directly communicate with, and exchange data with, each other. I wanted it fast and flexible. So I didn’t use pipes. I did what I presume DBus is doing. I used memory mapping, and other interprocess mechanisms such as signals. It’s fast and flexible and works well. _But_, it isn’t language neutral. Since the app interface comes in the form of a shared lib written in C, it supports C or C++ and any other language that can deal with that. I wrote a Python extension lib to support Python. And that was all the Linux language support that I was willing to do, because when I went looking for things like DBus, I was appalled at the lack of docs/tutorials/examples. It wasn’t worth my time to deal with that.
But the Windows version of my software uses ActiveX (in conjunction with the original memory mapping scheme) to yield a language-neutral inter-program communication. It supports every language that can deal with ActiveX (which is nearly every major language on Windows). It’s fast, flexible, and works well. And it was easy to do because I had the dev support to do it. So, my Windows Python example communicates with the Java example, without even needing to know it’s a java program. And the Java example communicates with the VB example. And the VB example communicates with the REXX example, etc. Yeah that’s right. Even REXX on Windows can participate.
Please don’t make blanket statements, without any additional supporting evidence such as “It’s not all about IPC. You seem to not know the difference” when I’ve already demonstrated in clear, ample detail exactly why it makes a difference, and I do know what it’s all about. Simply stamping your feet and sneering “No, you don’t” is childish. If that’s all you have to say, then you should _not_ be stating that someone else doesn’t understand the issue.
And that’s what got the orig author clamoring after this thing he’s calling “a managed environment” (just because that phrase is a famous one marketed by Microsoft to promote .NET so everyone thinks that it means something more than it does, but what he’s _really_ describing is a language-neutral inter-process communication).
It all happened because he discovered that pipes are a really primitive, crappy way to get programs to work together (ie, do inter-process communication), which also happens to be poorly supported in GUI apps (for reasons I outline above). And he wants some inter-process communication (inter-program, “managed environment”, whatever) which is not only more flexible, but also language neutral.
Ain’t gonna get it given the current state of Linux docs/tutorials/examples for anything beyond pipes. To quote Billy Joel “Dream on, but don’t imagine it’ll all come true”.
I think the idea you are putting forward can be realised without tying everything to the CLR or JVM. We can have those libraries you are talking about to be implemented in C++ then have bindings to them to different programming languges. For e.g if the functionalities of ethtool is implemented in libethtool then you P/Invoke into libethtool and get your GUI program cleanly written.
>Since all inter-program communication _must_ (and let me emphasize that
>again… _must_) be based upon some sort of inter-process communication,
This is the main point where we disagree. Inter-program communication doesn’t rely at all on inter-process communication (as I understand it, and as I borrowed the term from Ed’s last post). When you use inter-program communication, you’re just calling a library (a function, with params, to return values), and that doesn’t have anything to do with communicating away from the scope of the sole process.
>Please don’t make blanket statements, without any additional
>supporting evidence
If we had to be absolutely 100% sure of what we write, blogs wouldn’t exist. Anyway, that’s why I used the verb “seem”, a kind-of AFAIK statement (although I keep disagreeing from you now).
>(just because that phrase is a famous one marketed by Microsoft to promote
>.NET so everyone thinks that it means something more than it does, but what
>he’s _really_ describing is a language-neutral inter-process communication).
.NET inter-process communication is Remoting (IPC channel), and you haven’t mentioned that for now.
Managed code isn’t going to work.
At least for the space between programs. There is actually a reason that that many different languages exist. You will never be able to share Ada and Ruby byte code. Those languages have completly different goals (e.g strong typing etc.). It can only work if you only use the features of the lowest common denominator (which would be C).
“There are occasional exceptions where the object model doesn’t yet translate perfectly between CLR languages, but work’s regularly underway to round off those sharp corners.” - This is /never/ going to happen.
I would add to this concept a interface that could be accessed from a custom shell through reflection to actually allow prompting of all command line parameters. This concept is implemented on some big IBM computers. It is really helpful to command line users.
One Trackback
[...] Ed Ropple’s Homepage Code. Baseball. Whatever. Skip to content « The FSF: Pull Trigger, Blow Off Foot (Again) Fixing Linux, Part Two: Bridging the Gap » [...]