jql

Jambi means Progress

2006-07-28T21:05:00.000-07:00

I've been waiting a long time for this shoe to drop. It's time to publish some details [corrected URL] and/or code, gunnar. Inquiring minds want to ~~steal~~ know your binding technique and use it for the empowerment of KDE.

Intrometaspection

2006-04-30T15:48:00.000-07:00

So, how is KDE4 going to understand its own API? It's now May of '06, and the most promising projects I was keeping an eye on haven't been publishing much.

Java Bindings

aseigo leaked word of Trolltech working on Java bindings for Qt — based off the KDevelop parser, IIRC. It was supposed to be available in Q1 '06, but nary a whisper of it now. It wasn't ever really official, but I was looking forward to everyone having something standard to steal base KDE bindings on. I hope it's not dead (or castrated like QSA).

KJSEmbed

The presumed standard language binding. Is it intended to be a full Qt binding? Or something more practical, for use in KDE applets and such? I have no clue what state it's in right now. I'll cross my fingers.

The Introspection Problem

It is really hard to get a detailed API listing for Qt (let alone KDE) to use in a program. This has serious consequences for KDE, the best known of which is difficulty with language bindings. The lack of an API listing leads to half-measures and ugly workarounds. Richard is successful with his cluster of language bindings because he knows how to tickle the data out of Kalyptus. I just wish it was easier.

Bindings: XPath is my friend

2005-06-17T19:37:00.000-07:00

I've had good success with my gcc -fdump-translation-unit parser generating XML, and then using XSLT to convert the XML into C++ source-code. I expect to create an XSL stylesheet for the C++/moc4 Smoke binding, another one for the O'Caml binding, and yet another one for the C# binding. All using the same raw data generated from GCC. Scroll down to see an example of what that XML contains.

This data replicates what was availble through the Smoke bindings. Instead of trying to shoehorn all that data into giant, esoteric data structures in a monolithic library (which works great -- don't get me wrong), we're going to leave it as XML. Each class which has a Smoke binding will live in its own .so file, and should be dynamically loaded upon request. Inside that file will be the full XML data which describes that class. Upon loading the library, the class XML will be inserted to a global Smoke DOM, which will be the basis of the Smoke automation system.

Once we have the DOM loaded, we can preform the same searches Smoke uses, but with XPath.

To get a list of all the methods in QPushButton:

//type[@name='QPushButton']/struct/method

To get a list of the callable methods:

//type[@name='QPushButton']/struct/method[not(@access='private')]

To get a list of virtual methods defined:

//type[@name='QPushButton']/struct/method[virtual]

The list of static methods we can call:

//type[@name='QPushButton']/struct/function[not(@access='private')]

The list of Whatever::setFoo() methods which accept 2 arguments:

//method[@name='Whatever::setFoo' and (param/last() = 2 or param[3]/default)]

The Smoke method lookup will be XPath-based. You will need to lookup the XML node corresponding to the function if you want to call it via Smoke. Once you have the node, you will know the types of all the arguments and anything else you could possibly want to know about the method, its class, etc. You can check if a method exists in a class with a virtual destructor by looking up ../destructor/virtual. It's far more powerful than Smoke.

How about speed? Well, I don't expect this will come anywhere close to C++ speed. However, with smart caching of the XPath query results, the runtime penalty during tight loops can be minimized.

For static languages which require compiled bindings, the XPath gives way to XSLT, so there's no significant runtime penalty there. The Smoke library will still load the DOM, and the function calls will still be routed through the meta-calling convention, but that's not so bad.

As promised, I'll spam you with XML:


<lib>
  <type name="QWorkspace" id="6" href="/opt/qt4/include/QtGui/qworkspace.h#42">
    <struct>
      <base class="QWidget" access="public"/>
      <constructor name="QWorkspace::QWorkspace" access="public">
        <return>
          <type name="void"/>
        </return>
        <this>
          <pointer>
            <type name="QWorkspace"/>
          </pointer>
        </this>
        <param name="parent">
          <default/>
          <pointer>
            <type name="QWidget"/>
          </pointer>
        </param>
      </constructor>
      <method name="QWorkspace::addWindow" access="public">
        <return>
          <pointer>
            <type name="QWidget"/>
          </pointer>
        </return>
        <this>
          <pointer>
            <type name="QWorkspace"/>
          </pointer>
        </this>
        <param name="w">
          <pointer>
            <type name="QWidget"/>
          </pointer>
        </param>
        <param name="flags">
          <default/>
          <type name="QFlags&lt;Qt::WindowType&gt;"/>
        </param>
      </method>
      <method name="QWorkspace::sizeHint" access="public">
        <return>
          <type name="QSize"/>
        </return>
        <this>
          <pointer>
            <type name="QWorkspace" qualifier="const"/>
          </pointer>
        </this>
        <virtual/>
      </method>
      <method name="QWorkspace::paintEvent" access="protected">
        <return>
          <type name="void"/>
        </return>
        <this>
          <pointer>
            <type name="QWorkspace"/>
          </pointer>
        </this>
        <param name="e">
          <pointer>
            <type name="QPaintEvent"/>
          </pointer>
        </param>
        <virtual/>
      </method>
    </struct>
  </type>
</lib>

If you run that through my crufty deparser XSL, you end up with something that looks like a C++ header. If you have an XSLT-happy browser (aka. Mozilla/* or IE6), you can perform the transformation in your browser, or you can view the finished result, care of libxslt.

Smoke : KDE :: COM : Win32

2005-06-11T18:23:00.000-07:00

Adam points out the deficiencies of Smoke as a language binding. On the whole, I have to agree with him. Smoke doesn't match what we should recognize as a real language binding.

Smoke is an automation system

Similar to COM in Windows, Smoke is an in-process automation system. Distributed Smoke (aka. DSmoke) would be a relatively simple add-on to the built-in marshalling system. The Perl/Ruby Qt interfaces are to the Smoke automation system rather than Qt/KDE themselves. It's analagous to a Win32 interface which only used CreateObject() and COM.

Adam goes over a list of what could be gained from having the ability to create real binding for KDE. However, it would likely require maintaining compiler support, and wouldn't be entirely portable to MacOS/Windows. I think Smoke is a necessary evil, while a real set of language bindings should be considered optional.

In short, worse is better.

Smoke for KDE4

2005-06-10T00:33:00.000-07:00

I've seen mumblings from the KDE folk about increasing their binding support in KDE4. Specifically, aseigo has mentioned making KJS and Python bindings standard.

My recommendation is to base the bindings on Qt4's metaobject system. The idea would be to adopt the Qt4 meta-calling convention for the next version of Smoke.

How would the Qt4 Smoke bindings work?

The goal would be for every function in every class to be available through QMetaObject::invokeMethod() Like Smoke does now, it would allow AUTOLOAD/missing_method based language bindings.
However, the metaobject system only gets us 50% of the way there. There are some obstacles to overcome:

There must be a way to call static functions -- like constructors.
Not every Qt class inherits from QObject. We need to interface those as well
Virtual functions!

I haven't worked out how to solve these issues entirely, yet. Here's where my thoughts have lead me so far.

Static functions

The answer here is probably to generate a factory class which is aware of the static functions.
Lets try writing Hello World..


// Lets say the smoke4 bindings export factory() functions for each class
extern "C" void* qapplication_factory();
extern "C" void* qpushbutton_factory();
extern "C" void* qstring_factory();

// public call interface of smoke
extern "C" void metacall_wrapper(void* object, int index, void** args) {
   // call qt_metacall()
   QObject *o = static_cast<QObject*>(object);
   o->qt_metacall(QMetaObject::InvokeMetaMember, index, args);
}

extern "C" int indexOfMember_wrapper(void* object, const char* member) {
   // indexOfMember() looks up the method name in the metaobject
   QObject *o = static_cast<QObject*>(object);
   return o->metaObject()->indexOfMember(member);
}

extern "C" void* qt_metacast_wrapper(void* object, const char* className) {
   QObject *o = static_cast<QObject*>(object);
   return o->qt_metacast(className);
}

extern "C" void smoke_call(void* object, const char* member, void** args) {
   // helper function to do everything at once
   int _i = indexOfMember_wrapper(object, member);
   qt_metacall_wrapper(object, _i, args);
}

int main(int argc, char **argv) {
  // lets assume these *factory() functions are coming from smoke
  void *appfactory = qapplication_factory();
  void *pbfactory = qpushbutton_factory();
  void *strfactory = qstring_factory();

  void *app;
  void *_a0[] = { &app, &argc, &argv };
  smoke_call(appfactory, "new(int,char**)", _a0);

At this point, the app variable contains the return-value from new(). In order for us to be able to call qt_metacall() on that variable, we need to mandate that the return-value from new() can be safely cast with static_cast<QObject*>. If we want the derived pointer, we can use qt_metacast().


   // allocate a new QString
   void *string;
   char *_d1 = "Hello World!";
   void *_a1[] = { &string, &_d1 };
   smoke_call(strfactory, "new(const char*)", _a1);

In order to automate the QString object, the constructor returns a proxy QObject instead of a bare QString, so we can invoke methods on it with qt_metacall(). The question here is how do we get at the bare QString object if it's being wrapped in a QObject? The obvious solution would be to hijack qt_metacast() in the proxy object to break the encapsulation. This would be the only legal method to see a naked QString pointer. It would also allow us to cast up/down the inheritance chain for non-QObject classes by implementing it in qt_metacast().


   void *qstring = qt_metacast_wrapper(string, "QString");
   void *button;
   void *_a2[] = { &button, qstring };
   // and construct the new QPushButton
   smoke_call(pbfactory, "new(const QString&)", _a2);

Finally, we finish everything off;


   int w = 100, h = 30;
   void *_a3[] = { 0, &w, &h };
   smoke_call(hello, "resize(int,int)", _a3);
   smoke_call(hello, "show()", 0);
   int ret;
   void *_a4[] = { &ret };
   smoke_call(app, "exec()", _a4);
   return ret;
}

That covers Hello World. As a perk, every function in every class is now a valid slot. From a language binding, you can even connect() to a function in a non-QObject class.


my $str = new QString;
# there's no QString::set(const QString&) function?
$str->connect($listview, currentTextChanged => 'operator=');

Virtual Functions

If we've figured out the calling convention, how do we implement virtual functions? As with Smoke v3, it'll be necessary to override every virtual function in every class. I suspect if you fiddle with gcc4's symbol visibility and compile the virtual-function subclass into the same library as the original function, the linking overhead for doing this should be ~0. That overhead was a nasty performance hit for the smoke library, especially since it was built as a monolithic .so for all of Qt.

Each virtual function should emit itself as a signal. By default, each virtual function would be connected to its own slot. When a binding language wants to override a virtual function, it'll be free to connect() the signal to any valid slot. Even a slot in a different object! The binding should take care to disconnect() the default slot connection if the programmer reimplements the virtual function.

If the virtual function hasn't been connect()ed up, we're free to optimize away the signal emission if we want and call the original function directly.

Just to put some code to this idea...


#include <QWidget>
#include <QSize>
#include <QCloseEvent>
#include <QMetaObject>

namespace SMOKE {

class QWidget : public ::QWidget {
public:
    virtual ::QSize sizeHint() const;
    virtual void closeEvent(::QCloseEvent*);
};

// SMOKE::QWidget implementation

::QSize QWidget::sizeHint() const {
   ::QSize _ret;
   void _a[] = { &_ret };
   qt_metacall(::QMetaObject::InvokeMetaMember, metaObject().memberOffset() + 12, _a);
   return _ret;
}

void QWidget::closeEvent(::QCloseEvent *e) {
   void _a[] = { 0, &e };
   qt_metacall(::QMetaObject::InvokeMetaMember, metaObject().memberOffser() + 13, _a);
}

}

Method discovery

Now that we can call all these methods, we need to know they're there. Somehow we need to provide a listing of available methods to the language bindings. QMetaObject doesn't provide what Smoke needs in this regard.

The current Smoke method discovery could be adapted to use the Qt4 moc data-structure, which is just byte offsets into a string which contains every method and type used separated by \0. The Smoke::Index values would be converted into metaObject() offsets. A few other tweaks would be necessary, but moc seems to have similar requirements to Smoke as far as method descriptions go. We'll probably want to keep Smoke3's method$$ munging syntax since it's so handy for overload resolution.

Signals and Slots

If you notice from my connect() example above, I think we can do away with specifying signal and slot arguments. For signals declared in the binding language, we're going to cheat! Every signal will be declared as having zero arguments. When you call connect(), with a language signal being emitted, the call doesn't get passed on to QObject::connect. Instead, the binding needs to keep track of its own connections. Picture this:


sub foo : signal;

# this actually declares a function which looks like:

sub foo {
    my($self, @args);
    our %slots;
    my @candidates;
    for my $slot (@{ $slots{'foo'} }) {
        push @candidates, $slot
            if compatibleArguments($slot, \@args);
    }

    # of the slots with the given name which can accept these arguments
    # pick the one with the longest string length
    @candidates = sort {
        length($a->signature) <=> length($b->signature) || $a cmp $b
    } @candidates;

    magically_call($candidates[0]);
}

For slots, we can dynamically create slots which match all the matching signals:


$foo->connect($bar, signalName => slotName);

# signalName(const QString&)
# signalName(int)
sub slotName : slot {
    # this function gets called for either/both of those signals,
    # unless it specifies a signature in the slot declaration
}

That exhausts all I've thought through so far.