Fixing Linux, Part One: “The Space Between”

Much of the “the space between language” is solved by using a managed environment like .NET. What you describe is very much the old bindings-hell. Mono (.NET) solves this. One binding and I can use the library/code in question from C#, Python, PHP, or any other language for which there is a managed implementation.

As for “the space between usage paradigm” have you used Power Shell? While the product is closed source the concept and usage in unparalleled beauty. Applications can simply provide commandlets and you can easily do the same things you can in shell scripts (or other scripts) without all the shell-ish madness since it is OO - you can loop over files without worrying about spaces in the filenames, etc… since you are iterating File objects instead of file names, etc…

Vala[1] is another good bridge for the space between languages. It provides many of the benefits of a managed language (automatic garbage collection, simplicity of programming, ease of binding (due to gobject backend)) along with many of the benefits of C (no virtual machine, bindable from /any/ language (not just ones blessed for the current vm), portable to any hardware with a C compiler).

1. http://live.gnome.org/Vala

Being a user of GObject is not the same as being a user of GTK+. GTK+ is stacked on _top_ of GObject. There is no dependency the other way. Vala is itself a CLI program, and is using GObject (heavily).

Yeah, I see your point. I guess Vala is a viable option for a _standalone_ library, or a library depending on other C/Vala libraries. Still not too bad, but a platform like Java or .NET still has its uses I agree.

DBus was supposed to address the issue of programs being able to have advanced control over, and communication with, each other, as well as exchange data. It should have made obsolete the tradition of communication via “piping the text input/output of programs via the command line”.

But like too, too many free software projects, all the time was spent on coding (and usually writing code scant of comments, so it’s really, really difficult to glean what the code does and how it works), rather than documentation/tutorials. So very few people ended up using it because they just didn’t know how to do so without good docs and tutorials. Surprise, surprise! When are open source programmers going to learn that, if you aren’t going to document your code in an explicit and clear manner, then you may as well not even release it publically, because very, very few people are going to use it. You’ll get more usage if you pare your feature list way down, and instead spend more time writing docs/tutorials. THAT’S how Microsoft got all that 3rd party developer support for their equivalents to things like DBus (ie COM and ActiveX) — because MS (0r the Windows development community) releases lots of docs, examples, and well-commented example source. Linux is in desperate need of good doc and tutorial authors. If you want to make a difference, start there. Hell, start with DBus. If you can explain it well to developers (other than the ones who created it), maybe they’ll start using it more, and that will make your work infinitely easier when they do.

In “blessed for the current vm”, I wasn’t differentiating between .NET and Mono, but rather CLR and JVM (and whatever the next one will be).

As far as portability goes, if you only care about x86 and x86-64, then you’re fine with Java or Mono. However, desktop-type software is being run on more architectures (ARM, Blackfin), and a C baseline is an advantage in those circumstances.

>I’m not talking about inter-process communication, I’m
>talking about inter-program communication

Um, you can’t have inter-program communication (without piping the text input/output of programs via the command line) without first having inter-process communication. *IF* the Dbus folks had ever documented their work, and provided clear tutorials/examples, so that others could easily implement it in their programs, then adding user scripting on top of that would be as easy as it was for Microsoft to create VBscript and Jscript, just as soon as most windows software provided a COM (ie, ActiveX object) interface. (And frankly, MS was uncharacteristically unforthcoming with ActiveX scripting docs. Even so, they eventually released enough adequate docs/examples so that, within about 2 years, most major apps supported ActiveX scripting). And of course, the Windows’ version of Python also has the ability to interface with any programs’ ActiveX interace, thanks to that documentation/tutorials/examples. Want to control Works from your python program? Just create an instance of its ActiveX object, and start calling its functions. Easy. Want to exchange Works data with Excel? Then create an instance of Excel’s ActiveX object too, and your python scripts goes to town calling functions in both objects to do what you want to do with those 2 apps.

It would have already happened by now in Linux if the documentation and tutorials had been done. Your python script would just create an instance of some program’s “DBus object” (or whatever the DBus scripting component would call it), and start calling its functions to control the program, and exchange data with it. It wouldn’t matter whether the program was GUI-based, or command line. (You wouldn’t need to know). And it wouldn’t matter what language the program was written in. There’s your “managed environment”.

Did you know that a Python script can actually call a function written in another language, such as Java or Ruby, passing data to it, and getting back its value? Yep. All it takes is for both your Python interpreter to support ActiveX and also the other language’s interpreter to do the same. That’s how your can embed both Jscript and VBscript code in the same object tags on a web page, and have a Jscript function call a VBscript function or vice versa.

Could be happening in Linux… if developers had actually documented their stuff, and spent some time writing good tutorials/examples rather than adding more features to software that is already underutilized due to poor docs/tutorials/examples. And then after a few years of no one using his stuff, the open source developer gets discouraged, abandons it, and moves on to fork some other abandoned project, repeating the same mistake of not taking the time to document it.

The reason Linux doesn’t have this is because no one can build a scripting implementation upon inter-process communications like Dbus for two reasons:

1) The docs/tutorials for the inter-process communication frameworks like DBus are so awful/inadequate, that the overwhelming number of programmers don’t know how to implement it in their software.

2) There’s nothing useful you can do with a scripting implementation if it isn’t using some inter-process communication also used by lots of programs. ie, If virtually no apps have a DBus interface, what good is it to have a scripting implementation that works via DBus?

And even if, magically, all known Linux software were to suddenly have some sort of DBus interface via the wave of a magic wand, the guy who writes the scripting interface now needs to utilize DBus too, and he ain’t gonna do it easily given the horrible state of Dbus documentation.

It’s all down to inadequate/poor documentation, tutorials, and examples for Linux open source alternatives. People are coding the stuff (without even commenting their code adequately), and then tossing it out there without letting others know how it works or what it can do. Thud. That’s the sound of something falling flat on its face, and not being a viable alternative as a result. What’s the point of having access to source code if you don’t know what it does, how it works, and how to use it? That’s a rhetorical question. The answer is: There’s no point at all, and even if such software could potentially address the problems you have, it isn’t doing so now, and very likely will never do so.

Hammer it into every open source developer’s mind that his software is only as good/useful as his docs/tutorials/examples, and things will start to improve. And not until then.

>I want to be able to consume another application *within* mine and
>avail myself of its functionality.

That’s what ActiveX scripting does! For example, it lets your Python program totally control every aspect of some other program, from starting it up, to closing it down, to doing anything with it that its ActiveX scripting object exposes. It also lets you call functions in any ActiveX shared library, regardless of what language the lib was written in. And it even lets you call functions written in any interpreted language (ie, a Python script calling that Ruby code) provided the interpreter supports ActiveX (and most all Windows script engines do today).

I’m beginning to think that you’ve never used ActiveX scripting. You owe to yourself to do so, so that you can see how something like this is exactly what you’re saying you need.

Think of ActiveX as a way to allow a program written in any language (ie, Python, Ruby, Perl, C, etc) to treat any other program (written in any language, such as Python, Ruby, Perl, C, etc) as if it’s a shared lib whose API is callable by that first program. It’s that “managed code environment” that you’re lusting after.

If you want more technical info about ActiveX scripting, I wrote an entire series of articles over at Code Project:

http://www.codeproject.com/KB/COM/com_in_c1.aspx
http://www.codeproject.com/KB/COM/com_in_c2.aspx
http://www.codeproject.com/KB/COM/com_in_c3.aspx
http://www.codeproject.com/KB/COM/com_in_c4.aspx
http://www.codeproject.com/KB/COM/com_in_c5.aspx
http://www.codeproject.com/KB/COM/com_in_c6.aspx
http://www.codeproject.com/KB/COM/com_in_c7.aspx
http://www.codeproject.com/KB/COM/com_in_c8.aspx

But that doesn’t address the problem of: How and where do we get info like the above, about Linux alternatives, so that they can be used as much and as effectively, as ActiveX scripting is used on Windows? Hmmmm?

>A managed code userland offers not just the ability for you to use >*functions* from other programs (written in other languages or not), but >entire objects.

But an ActiveX object *is* treated like an object in an object-oriented language. Take this bit of VBscript for example:

Set objConnection = CreateObject(”ADODB.Connection”)
objConnection.Provider = “ADsDSOObject”
objConnection.Open “Active Directory Provider”

You’re not setting a variable named objConnection.Provider. You’re setting the Provider member of an instance (ie, object) of the “ADODB.Connection” class. And you’re not just calling a function named Open. You’re calling the Open method in the same object. That first line has actually created/imported an entire object into your VBscript program. (And incidentally, that object is written in C++). You have done exactly what you want to do (and for some reason assume that you can’t do with ActiveX).

And you can do the same with Python. It takes one line of code to create/import an entire ActiveX object (written in any language) into your program, and from there you treat it just like any other object. You access its data members the same way, and call its functions the same way.

Same thing with any other object oriented language using ActiveX.

Dbus was supposed to make all this possible. The reason it hasn’t is because no one knows how to use it, and so its potential remains completely untapped for such use.

That was my point all along.

All managed code gives you is some automatic garbage collection. But your problem isn’t memory leaks. Your problem is that you want the kind of control and flexibility you get by being able to interface to some program as if it had a directly accessible API, wrapped in your own language’s object-oriented implementation. Well, that’s the whole idea behind ActiveX. Essentially .NET is just some automatic garbage collection added to ActiveX, plus the requirement that all interpreted languages include a “compiler” that produces one of these “byte code” things. (ie, Essentially a platform neutral sort of assembly code). But again, your issue isn’t speed either, so you don’t care about the byte code thing either. Your issue is that you need simply what ActiveX gives you… and you need it on Linux. Well… there’s DBus. But how do you use it? And do more than a handful of people actually use it in their software? There’s the REAL problem.

You edited your preceding post after I wrote my above reply to add a lot more questions. I think from my description of the type library, you can deduce a lot of answers to those questions. For example, VB.NET’s IDE has an “object browser” that queries the type libraries of all installed ActiveX components to get the names of all functions/data members, and number and datatype of all args and returns to functions. So of course, that’s how the IDE’s “IntelliSense” works. And when you install a new ActiveX component, it automatically works with it. You don’t have to update your IDE, or rewrite some sort of “keyword file” or whatever. The IDE automatically recognizes it, and utilizes the standardized type library in all ActiveX components to extract the needed info.

“Exceptions” are generally handled by IError “sub-objects” in an ActiveX component. So yes, there is a standardized method for error reporting designed to be used by various languages.

Obviously, a lot of implementation issues were considered by MS developers when they came up with this. Certainly some developers have not been entirely pleased with everything. (Frankly, I’ve had some really scathing things to say about IDispatch. That seems to be a sore point with others too. I would have implemented that functionality very differently. And don’t even get me started about ActiveX’s debugger helper API). I’m not sure what “articles” you’ve read about ActiveX, but the dirty little secret is that, contrary to what Ubuntu fanbois will typically tell you, not everything associated with Microsoft is awful and unusable. In fact, some of it frankly blows the competition away. For whatever faults it has, ActiveX is one of those examples because Linux doesn’t even have something comparable to that, despite the existence of things like DBus.

Yes, MS is moving away from ActiveX scripting to .NET. Hey, they can sell you something new if you want it. What company doesn’t want to have something new to sell you? And there are some things that .NET will give you, like the garbage collection, and speed-up of interpreted languages I alluded to. But I think it’s a bit overrated, has its own problems that earlier alternatives didn’t have, and frankly, I think that a language like C# is so overly abstract that it ends up producing code and programmers that don’t know how to write stable, reliable, efficient apps. You may disagree, but that’s what I see.

But the bottom line is: ActiveX scripting would do the things you stated in your blog you wanted to do (and certainly better than what you’re doing now), and Linux doesn’t have the same level of support for such. My take on this is that it’s largely due to the failure of most open source programmers to document and demonstrate their code as well as MS’ community of developers do. Dbus should have done this already. It hasn’t. Why? It’s the docs/tutorials/examples. What do you think Ballmer was really saying when he said it’s about “Developers! Developers! Developers!”? He’s saying that MS is going to make sure devs have what they need to get their work done as easily, efficiently, and effectively as possible. And for a start, you can’t do it with the state of Linux docs/tutorials/examples.

Your friend asked what you would do. I’m telling you my answer to absolutely everyone involved with Linux is to start right there with the docs/tutorials/examples, especially for the programmers who bring you the software you need to do your work. Ballmer knows it. Shouldn’t someone else too?

You know I had a similar feeling when I started playing with Linux. I’m a paid up MSDN subscriber kinda guy. I would lay awake at night dreaming of how to fix linux. But if you look at how you’d “fix” linux, do you see yourself implementing windows-like solutions? I do. Question is, do these windows-like changes have any other effects. The conclusion that I came to is that the windows-like solutions would put in place all the constraints that makes windows less useful than linux. This became vastly clearer to me when I stopped playing with linux, and started using it to make money.

Example. I set up vsftpd today. I had to muck about with firewalls, openssl, vsftp.conf and pam. I edited numerous text files. The end results is I have real and virtual users logging in to my ftp box and it all works. Migrating this to another machine is a matter of copying over the keys and the config files. I know where they are and it works. I write this shit down in my own private blog that no-one can see. Its almost like, like, keeping a journal, like, when you have to get your MEng!

Another example: I needed to format some data for a web page. It was just simple to sed the stdout of a program than it was to muck about with the program. Could have. Didn’t need to.

You talk about libDoesX, libRubyDoesX etc. You seem to want a C based libDoesX that Ruby, C# etc can bind to. I have a couple of problems with this:

1) C and C++ are dead as far as I’m concerned. Nothing useful should be done in them unless you are programming a kernel or a video game (and I’ve done both). You cannot write good code as quickly as you can in a modern language.
2) Mono-culture. If everyone has to use libC++DoesX, then libC++DoesX is a bottleneck. Its a lowest common denominator. And chances are its not even the fastest at presenting information to the various different higher level languages.

Unlike Microsoft, the free software world is not a mono culture. It does not matter if two groups do the same work in parallel, because those two groups will implement their own best solution. If the first group wrote libDoesX in C, it would take them longer. The second group would find that this libDoesX doesn’t do quite what they need, and starts to modify it - hating having to use C the whole time. I believe that each group would save time writing it from scratch in their own language.

Another disadvantage of monoculture is libDoesXandYandZandDependsOnAandBandC as a result, and each of those then has to depend on everything else. Someone did a call graph for an http request in apache vs IE. What a difference. This is why Server 2008 with *no* features installed takes 800Mb to boot.

I now have a number of linux machines that work, work well and work securely. The perform vastly faster than windows, because they don’t take 800Mb of RAM just to boot.

****

In my view, the “space between” is the “space between” *testing* paradigms: although one might thing of sed/awk/perl as separate applications, they are not - they are like methods in the larger application whose source is made up of a myriad of .sh/.conf files. This has problems:

1) most people who would put their .cs files into svn, don’t put their whatever.conf in. They just don’t equate it to source code.
2) there are no unit testing frameworks for bash^H^H^H well f*** me there is, but I’ve never seen anyone use it.
3) executables are great, because they are a single lump of stuff immune to tampering. however, .sh applications can be broken in many ways by changing files seemingly completely unrelated. The executable in this case is human editable, i.e. breakable. Binary blobs are not. As a result, binaries are less fragile and more trust-worthy (unless you are pwned).

Now that I have a lot of unix boxes, I’m working on this myself.

The problems you mention in the article are really at the very heart of the problems with, not only *nix, but Windows and most operating systems. The operating system provides an enviroment that is inherently fragmented (different languages and libraries) and unsafe (C pointers, no OS control over processes).

Last night, I spent several hours reading about managed OS’s, including SharpOS, Singularity, and Midori. After reading a lengthly research article about Singularity, I concluded that a managed OS running managed code is going to be the future.

All processes are completely separated from the kernel and from other processes and can only communicate with each other via pre-defined, strongly typed, ‘contracts.’ You can use C#, managed Ruby, managed Python, or whatever language you want, and they can all reference the same assemblies due to strong-typing and the nature of managed code.

Another benefit of a managed OS is the inherent modularity of the OS (due to the microkernel approach), making it an ideal platform to be use for everything from mobile phones to cloud computing environments.

One of the currentl downsides of managed code is the need for the CLR. In Singularity and SharpOS, there is no CLR. Applications are compiled to IL code, verified for type-safety, and then compiled to native code (Singularity project acknowledges the need for strongly typed assembly), thus eliminating the overhead of managed code.

I would love to see the FOSS community grab ahold of SharpOS and turn it into a functional OS, proving that it can innovate and provide real solutions for the future.