Javascript

View Extended Entry

Posted by in Javascript | Add / View Comments()

Heres the problem.. as soon as you start building large applications usings ExtJs your users say two things

• Its slow to load
• Can you translale it into their language

The speed issues are mainly due to

• Loading of many javascript files (With our currently quite small applications, there are still more than 30 single javascript files - one class per file, which can take over  10 seconds to load.)
• The javascript contains comments and whitespace (As I like to make my code readable and maintainable)

To address these issues, there are a few javascript compression tools out there, however, most have rather painful setup requirements, like getting java working, and are not very hacker friendly.

Now that all my interface is embedded into javascript code, including all the button labels etc. l have been wondering for quite a while how I was going to translate the application and keep it up-to-date (in the Flexy template engine we extract the strings from the HTML, and have a tool that manages PO files based on them). Since this did not seem feasible for Javascript, I was begining to dread the thought of finding all the strings and wrapping them with something like a gettext type call.

So it turned out my little dmdscript hacking may have some use...
It took a small amount of hacking to add tokenizing support to dmdscript, by adding a new class for the javascript interface and few minor changes to dmdscript's internal tokenizer I got this little script to work.

   var data = File.read(fname);
   var x = Script.tokenize(data);
	
   for(var i =0; i < x.length; i++) {
       if (x[i].tok == "whitespace") {
           if (x[i].val.indexOf("\n") > -1) {
		print("\n");
		continue;
           }
           print(" ");
           continue;
       }
       print(x[i].val);
    }
    print(";\n");

Basically it reads the file, then creates an array of objects, each one representing a token. (I should perhaps change the token to an integer and use  Script.Token.whitespace as a constant... ).

The script will basically strip all the whitespace and comments, and convert whitespace into either a line break or a space.. (as linebreaks are actually relivant in Javascript).

The next issue was how to deal with language conversion. The code above can be used to test for strings by doing:

for(var i =0; i < x.length; i++) {
    if (x[i].tok == "string") {
       println("got string: " + x[i].val);
    }
}

So I came up with the idea of using the quote style to determine if a string needs translating.
single quotes = does not need translating
double quotes = a translable string.

It was amazingly easy to change my existing code to do this, I could then generate a list of strings for each file, and output a JSON structured file which can be used to store the translations mapping.

#djs ScriptCrusher.js -- --strings /tmp/Hello.js > /tmp/zh_HK.js

Would generate a file:

"Hello.js" : {
    "Hello":       "Hello",
    "World":      "World",
}

Which can be edited and translated.

Then when Crunching the script up, it uses the map to convert strings from one language into another.
eg.

#djs ScriptCrusher.js -- --langfile /tmp/zh_HK.js /tmp/Hello.js \
	> /tmp/Hello.zh_HK.js
#djs ScriptCrusher.js --/tmp/Hello.js > /tmp/Hello.en.js

(I've not done the merging of old/new yet, but it should be pretty trivial).

djs should be pretty simple to build: (requires gdc - available in Debian and Ubuntu + probably others)

#svn co http://www.akbkhome.com/svn/gtkDS
#cd gtkDS
#sh djs.gdc.sh

The full ScriptCrusher is here..
http://www.akbkhome.com/svn/gtkDS/test/ScriptCrusher.js

Happy Hacking..

View Extended Entry

Posted by in Javascript | Add / View Comments()

Most of the fun development projects had to be put on hold over the last few months, due to good ole paying projects.

ExtJS security holes

The majority of my paid projects are now using ExtJS, (and one has now used ExtJS 2.0, which is a definite improvement in both features and design). But I did run into what I considered to be a rather serious problem. ExtJS by design is a XSS nightmare... so be careful.

Basically the grid component and a few of the other components are designed to use JSON data comming from the server and render the data it recieves onto a styled HTML table. The problem I discovered, was that rather than using DOM.createTextElement() or similar to render the data it recieves. It uses innerHTML. So data is not escaped in any way prior to rendering onto the page.

It was discovered while testing one of my first major products, A complete Spam/Mail control solution, I discovered that it was not rendering email addresses <someaddress@example.com> , due to the <> characters. I dug into the code and realized the XSS hole, and posted it as a bug on the extjs forum.

Unfortunately it appears that rather than regarding this as a huge hole, they would rather trust developers to read the fine print in the documentation about rendering data to the screen. Personally I would be embarrassed by this attitude on my project.. 'security is someone elses problem'.. but I guess that's their problem... - I have to keep security patches for extjs handy....

GtkDjs updates

I finally got a short time yesterday to update the code in Gtkjs to use the latest dmdscript source, overlayed with my JS2 stuff and language binding add-ons. It builds and runs all the test code in the test folder, and Walter fixed the closure issues alot cleaner than I had...

I've also updated the 'downloadFiles.sh' script so that It fetches the correct language HTML files for the generator. It appears that gnome has moved all the documentation around. (It also means the generated code is more in sync with newer releases of Gtk and the many extensions.)

Mailfort teaser..

As I mentioned above, I'm working heavily on a new product, Mailfort, anyone sending email to me, who I dont know, is likely to get a Identify confirmation email. We are currently testing with friends and family, along with a couple of companies, and hopefully more details will appear soon. But the teaser is that I've been working on Spam and mail control solutions for myself and a few clients, which has finally evolved in something generic enough that we can begin to sell. A number of my friends have been complaining about their Spam problems, so they are now slowly being introduced to the beta test product (with very positive feedback) so hopefully we will get the full website and product rolled out soon.

View Extended Entry

Posted by in Javascript | Add / View Comments()

Other than Enums which are now created and available in Javascript, I've tidied up the generation code quite a bit, so static methods are generated correctly. Less brute force generation is used, and made a start on the documentation generation.

So now most of the donkey work of the bindings are done, it's down to the last 20% (the long tail from what I've been reading recently). Something like the main developer[s] get's 80% of the project done, but since there's always those little niggly things that dont work it takes 1000's of developers to fix the last 20%... - interesting statistic, which explains why it's such a pain fixing all those bugs that crop up..

Anyway The GtkDjavascript Manual has been born.. While again not quite ready for production, (It desperately needs user comments, Enums, Signals ,hrefs linking all together and many other things.). I think it's a step in the right direction.

The navbar, and the XML method descriptions are generated from the code generation script. (The main body is a nice example of extjs - and should be portable to gtkdjs eventually) -

The biggest bug on the docs is that stuff that is manually overriden (eg. Gtk.init()) is flagged as broken in the manual. as I've not added any code to add the fact I've overridden it to the manual docs..

Anyway.. ticking along nicely.. now I wonder how much use it is....

View Extended Entry

Posted by in Javascript | Add / View Comments()

I'm still digging through the gtk api, checking it works as I go. (and still deciding on a name for the bindings). I've been looking at perhaps one of the two most complex widgets in Gtk, GtkTreeView. The tutorial for which looks like it weighs in at half the Full Gtk manual.

At present I've got through the first section, and got a roughed out GtkListStore to work with it. (TreeStore should come next).

For those who dont know GtkTreeView, it uses the concept of a Model, and a View.

The Model contains all the data (in rows or a tree) - GtkListStore is a model.

The View (eg. GtkTreeView and it's columns) describe how to render the view, and what columns to use from the model.

This API is pretty raw at present, Some of it can be tidied up in Javascript, others may need more work in the bindings

First part of using the tree is creating the Store.

var store = new Gtk.ListStore(
    "gchararray", 
    "gfloat",
    "gpointer",
    "gboolean",
    "GdkPixbuf");

At present I've used the Gtype names to define what is being stored where. - This may need simplifying, and integrating somehow with the writing mechanism as validating data types written to the tree is a bit haphazard at present.

Next, to add data, you must use what I think is the worst named Class in Gtk, Gtk.TreeIter. a better name would have probably been something like  GtkRowReaderAndWriter (bit of a mouthfull, but alot clearer.)

To use this Row reader/writer, we first have to create a blank version. Then call the append Method on the store (a bit ass backwards, but again this could be fixed with some Javascript prototype modifications.

var iter = new Gtk.TreeIter();
store.append(iter);

The append method loads the Row reader writer, so that it can access the new row.

Next we add data to the row. This is done via G.Values at present, although this may change in the future...

store.set(iter, 0, new G.Value("test"));
store.set(iter, 1, new G.Value(1.0));
store.set(iter, 2, new G.Value({ a: 1, c: 2})); 
store.set(iter, 3, new G.Value(true) ); 
var pixbuf = Gdk.Pixbuf.newFromFile("gnome-logo-icon.png");
store.set(iter, 4, new G.Value(pixbuf) ); 

all this could easily be abstracted by defining a simple prototype

Gtk.ListStore.prototype.appendRow = function(rdata) {
	var iter = new Gtk.TreeIter();
	this.append(iter);
	for(var i =0;i<rdata.length;i++) {
		this.set(iter,i, new G.Value(rdata[i]));
	}
}
.....
store.appendRow(["test", 1.0, {a: 1}, true, Gtk.Pixbuf.newFromFile(....)]);

Next step is to build the Widget. And the renderers (the classes that turn the data into visable data). Gtk comes with a few renderers, It will be interesting to see if creating them in Javascript is feasible.

var view = new Gtk.TreeView();
var renderer = new Gtk.CellRendererText ();
var pixrenderer = new Gtk.CellRendererPixbuf ();

Adding columns is again a little raw, as we cant use the varargs methods the C does, so we have to combine a few calls.

var col = new Gtk.TreeViewColumn();
col.setTitle("Name");
col.packStart(renderer, true);
col.addAttribute(renderer, "text", 0);
view.insertColumn( col, -1);

Again a nice wrapper could be written.

Gtk.TreeView.prototype.insertColumnAuto = function(config) {
	var col = new Gtk.TreeViewColumn();
	col.setTitle(config.title);
	col.packStart(config.renderer, true);
	col.addAttribute(config.renderer, 
		config.renderer.isPrototypeOf(GtkCellRenderText) ?
		 "text" : "pixbuf", 0);
	this.insertColumn( col, -1);
}
...
view.insertColumnAuto({
	title: "text here",
	renderer: new GtkCellRendererText(),
}

Merging the data and the view is done simply by setting the model.

view.setModel ( store );
window.add(view); // add it to a GtkWindow..
window.showAll();
Gtk.main();

I also had a few experiments with callbacks.

view.connect("row-activated", function(view, path, col) {
	var model = view.getModel();
	var iter = new Gtk.TreeIter();
	
	if (model.getIter(iter, path)) {
		var name = new G.Value();
		var name2 = new G.Value();
		model.getValue(iter, 0, name);
		model.getValue(iter, 1, name2);
		println (
			"The row containing the name '"+
			name.getString() + 
			"' has been double-clicked. holding" + 
			name2.getFloat()	
		);
		model = new Gtk.ListStore(model);
		model.remove(iter);
		println ("removed?");
	}
	
	
});

At present this is a little klunky, but again with some prototype modifications it should be pretty easy to solve.
Fetching values has to be done by creating an empty G.Value, them fetching and converting them.

The last line illustrates removing a line. - since we cant cast in Javascript, I've modified the ListStore constructor to accept either a list of strings (eg. creating it) or a store object which does effectively the same as casting.. enabling you to call the remove method on GtkListStore.

Anyway back to fighting fires with real work today...

View Extended Entry

Posted by in Javascript | Add / View Comments()

As I dig away at the Gtk wrappings, I'm testing the ideas that should make the bindings interesting.

One of the ideas is that extending core Gtk Objects should enable easier usage. I started with the basics of interface design, Making a Menubar as  simple as possible to build, and bind..

var menuBar = new Gtk.MenuBar.quick([
	new Gtk.MenuItem.quick("File" , function () {
		println("File");
	}),
	new Gtk.MenuItem.quick("Edit" , function () {
		println("Edit");
	}),
	new Gtk.MenuItem.quick("Search" , function () {
		println("Search");
	})
]);

In the above example, I've created an extended class Gtk.MenuItem.quick, and Gtk.MenuBar.quick (it could just be Ext.MenuBar / Ext.MenuItem...) - Just adding it to the Gtk.Menubar Class namespace seemed clever at the time.

As you can see, creating the menu, along with adding the handlers is very simple. - I could even use a standard Javascript object as the constructor (even more like ExtJs): eg.

new Gtk.MenuItem.quick({
	title: "Search" ,
	icon:  Gtk.Stock.SEARCH, // this doesnt work yet!
	handler: function () {
		println("Search");
	}
})

This morphing of the Gtk bindings is all done in the Javascript code,

Gtk.MenuItem.quick = function(label, activate) {
	Gtk.MenuItem.quick.superclass.constructor.call(this,label);
	this.connect("activate", activate);
}	
Ext.extend(Gtk.MenuItem.quick, Gtk.MenuItem,  {});


Gtk.MenuBar.quick = function(ar)
{
	Gtk.MenuBar.quick.superclass.constructor.call(this,label);
	for (var i in ar) {
		this.append(ar[i]);
	}
}

Ext.extend(Gtk.MenuBar.quick , Gtk.MenuBar,  {});

(using snippets of code from Extjs) to implement the extending.

On the backend however, I had to make a few changes to dmdscript, both the DObject definition (which is the class that handles Javascript objects in the script engine). Noteably adding generic support for bound void*'s to all Javascript objects. (so binding things like curl, mysql, sqlite etc. should be considerably easier..)

Along with this, dmdscript as available from digitalmars did not support  the {function}.call(this, args) syntax, a stub was there, and it was pretty simple to add.

Anyway the next big challange is understanding GtkTreeView, GtkTreeModel and GtkTreeIter so that building Trees can be made  more Javascript (and Extjs) like..

View Extended Entry

Posted by in Javascript | Add / View Comments()

Signals for the most part are working. I went with the GTK'ish style, rather than the more JS style, eg.

var a = new Gtk.Window(0);
a.connect("delete", function() { Gtk.exit(1); });

That's not to say that it's not trivial to use a more javascript style = in the client side!!!!

Gtk.Widget.prototype.on = function (sig, call) { return this.connect(sig,call); }

There is still quite a bit todo, especially around the more complex widget's like textview, tree and lists. but I suspect the majority of the main widgets should just work.

I'm kind of wondering If I'm announcing this to the wrong audience though, as I dont really get any sense of excitement. I have this wierd vision that GTK and Javascript could be a leap forward in application development. Having written a couple of projects both in scripted and compiled languages, The whole save/[compile]/run/test loop can get very slow, especially as the application grows.

The idea, that to edit the toolbar, and it's behaviour, I would pull up the debugger/editor in the running application (That it'self would be written in gtkjs). Navigate in the tree of objects (AKA Firebug DOM inspector) to the class definition or prototype for the menu. - click edit - bring up a scintilla or GtkSourceView editor, edit the code.. save it.. (editor both saves to disk overwriting the specific code that it needs to), along with compiling and altering the in-memory method - Then type something in the debug command line to for the refresh of the menu... - Or even have the editor know that saving that function would need some js run to test it.. - and it affecting the running application - changing the behavior and look!.

Not only that, the program would not crash, only output to the debug console any unhandled exceptions... - (with line/file/method etc.)

I'm just in awe of the concept of adding a small feature or fixing a bug would by so mind blowingly simple, that you would lower the barriers even further to make more of your users your development team..

Then again other days I wonder if I've been doing to many of those drugs....

View Extended Entry

Posted by in Javascript | Add / View Comments()

My last entry described how to create objects for dmdscript, I made a brief comment, that gtk javascript might be quite cool.. Well after a bit of hacking I got hello world to work. (well and a bit more..)

Gtk.init();

var w  = new Gtk.Window(0);
w.setTitle("hello world");
w.showAll();
Gtk.main();

run with

/tmp/gtkjs hello.ds

I started with the GtkD bindings code generator, and extracted out the HTML parser code (it parses the HTML API docs). The based on the code from the last post, and some of the ideas from GtkD generated all the classes for dmdscript.

At present:

• 317 Classes, Including Atk, Glib, Gobject, gthread, gdk, gdkpixbuf, glade gtk and pango
• 2554 Methods should be working.

What's left to do..

• Signals - I've got a prototype working for this, so It's mostly a matter of  writing the generation code.
• Enums - They are in the D code, but no exported to the Javascript side.
• Argument checking.. - Should be pretty trivial to add..
• Return checking.. again pretty trivial..
• Custom code for non-generated methods, Structs etc.
• Tiding up the APILookup files. - - they are pretty similar to the GtkD code at present, and my generator just ignores alot of them.
• Lots of code tidy ups.. (the class registration is a bit of a brute force hack)
  

If you want to try it out... (developers only.. dont expect to write applications with it yet.)

svn co http://www.akbkhome.com/svn/gtkDS
cd gtkDS/wrap
#download API Files (needs curl / tar etc.)
sh downloadFiles.sh
#compile it - needs dmd installed..
sh buildit.sh
cd ..
#build the bindings - requires compd from dsource.org/projects/dool
compd gdkds.compd
/tmp/gtkjs test/hello.ds

Now I just wonder what use it is ;)

View Extended Entry

Posted by in Javascript | Add / View Comments()

Having spent the last few weeks working with ExtJs, I'm been drinking quite a bit of Javascript Coolaid. (Actually being british, I never understood that phase, but I guess most people understand it these days.)

Doing all this a while back led me to investigate General Javascript engines, I had this weird vision that Javascript and Gtk may make an interesting combination.

As I mentioned in my last post about testing code the whole write-save-(compile)-run-test would in an ideal world be reduced to write-save-test. And since very few languages actually allow you to redefine objects, object methods, functions etc after you have actually run the application (without weird add-on etc), Javascript is pretty interesting in this area.

Anyway there are quite a few EMCAscript engines out there, most of them in C, and a few in other Bytecode or even scripting languages. But only one that is written in that new little magical language D!...

dmdscript, (also on dsource as walnut with a few hacks and the start of a new version), Has some serious benefits over all the other implementations.

- The source code is clear, readable, and not cluttered with memory allocation crap.
- It's fast
- It's small = small again making it easier to understand
- It's free of any major dependancies (apart from the compiler.. dmd or gdc )
- Writing extensions is (at least in theory) simple, and not full with learning about crazy cryptic macro's or custom memory allocation stuff (sound like anyone's favourite language?)

The only downside... - Dont expect alot of support, It looks like the project is one of Walter's little whims, that he lost interest in a bit. And licence-wise I'm not sure if GPL is the perfect licence for the project.. - but that's just a gut feeling, I'm far from a Licencing expert, but I suspect using it as a add-on for another project that is not GPL would be difficult. And I'm not sure how adding LGPL to a GPL application works either... Anyway..

I had a small amount of free time at the end of last week, Just having nailed the last few bugs in a project. So I thought I'd investigate dmdscript extensions on the pre-cursor to looking at modifying the GtkD bindings generator to create the dmdscript bindings...

First stage investigation, gave a little bit of imformation from the dmdscript site, along with a short note on the dsource Walnut project subversion server. But unfortunatly It's not really enough to get you started, and there are now 'Step by step' instructions on writing extensions, or adding Objects.

So here's my step by step guide to adding an object with methods to dmdscript... - in the extended entry..

View Extended Entry

Posted by in Javascript | Add / View Comments()

I thought I'd go some of my random thoughts of working with extjs, as creating quite large applicaitons with extjs is quite new, and as the projects I've used it on have been refactored a few times, I'm beginning to develop approaches that make code more maintainable, and faster to develop.

This is quite a big post - so the contents in the Extended body....

View Extended Entry

Posted by in Javascript | Add / View Comments()