Latest

Some thoughts on the language server and its usefulness in the roobuilder

Roo Builder for Gtk4 moving forward

Clustered Web Applications - Mysql and File replication

GitLive - Branching - Merging

PDO_DataObject Released

PDO_DataObject is under way

Mass email Marketing and anti-spam - some of the how-to..

Hydra - Recruitment done right

RSS and Categories

All Categories

PHP

PHP-GTK

PEAR

Hong Kong

Linux

General Crap

XUL

Digitalmars D

Javascript

Gtk

Roo Javascript

Xtuple ERP

13 Feb 2011

SQL change management done right.

I'm slowly going through mtrack at present, breaking the HTML rendering into templates, and making nice clean classes for the fetching of underlying data. All this is due to implement quite a few ideas once it's in a state to be easily changed.

The more deeply I go through mtrack though, there are parts I find that you look at  and think, "That's a really smart way to handing that problem". Like the change control auditing, where each component has a ID for the last updated (and created) This maps to a event table reord containing who/when data. Much cleaner than the more common practice of using two datetime and user fields.

However, as always, there are parts of the code where you want to pull you hair out and say, No way should you solve a problem like this. Mtrack's SQL change management is one of those areas.

It's approach is not uncommon, and I've seen it before. Basically it uses an XML file to define the schema and then has a series of files schema1.xml,schema2.xml,schema3.xml... and so on. 

The installer works out what the current version is, then compares schema's between the current and selected version. and works out how to update the database. There is a driver for PostgreSQL and SQLite. 

My fundamental issue with all this is that while on the face of things it does not seems like a bad idea, however, it ignores the plain simple fact that there is already a language used for Database Schema definitions and modifications, and it's called SQL!

Anyone who uses SQL can quickly read a SQL file and understand what it's doing, even if it's in a different dialect (SQLite etc...), but putting that type of information in a XML file just adds so much noise. Worse of all, it involves learning and remembering a small subset of knowledge, that is only relevant to a tiny problem domain. Then the worst thing is it's just plain annoying to read.

For my projects I'm luckly only having to deal with a single database vendor (MySQL usually). To manage changes I keep SQL updated, this file contains all the table definitions along with the later changes. To update the database, I just pipe it through the mysql command line with the '-f' (force) switch. This is a trivial, simple and effective way to keep the database schema in sync wit the project code. There are even wrappers in the Component Framework code to do this simple task for you

The big problem however is that SQL, for all these great benefits has turned out to be horribly incompatible between database vendors. If I carry on with the above idea of keeping my database modifications in a SQL file,  then I would end up with one for each of the databases I want to support. Then I not only have to keep each of the schema files up-to-date, but have to remember the syntax for multiple database vendors, and apply each change to every file, not really a good long term plan.

So rather than keep multiple files up-to-date, I wondered, why not convert the SQL schema changes from one dialect to another, and just keep an SQL file as I currently do, and make it feaasible for anyone to install using the database of their choice.

This is one of those 'why has no-one done this before moments', but for the life of me, I could not find anything that came up to quickly on google. So I had a look at what was 'close enough' for this idea to work, and what a supprise, most of the code for this is already in PEAR.

The SQL Parser package, as very basically introduced here, http://www.sjhannah.com/blog/?p=16, provides pretty much all the backend code for a solution to this. However, there was not previously any code in either the parser, or writer/compiler to actually deal with DDL commands like alter table etc.

I've just commited the changes for this, so you can now very easily extend the current SQL_Parser_Compiler class to output your favourite dialect of SQL, based on reading an SQL file containing the base changes.

For a example of how to use it in real life, here's a nice simple example from the forked mtrack codebase.

And here's the commit that makes it possible..

And finally, a good example of a SQL file that can be run through the parser/generator.

Posted by in PEAR | Add / View Comments()

26 Jan 2011

Browser rending weirdness OSX and firefox

Sometimes rendering bugs can be very annoying, IE has it's share of misbehaving, but usually firefox, chrome are consistant and never need much testing. However on one site, prior to going live someone testing on OSX kept saying that the menu bar was broken in Firefox.

Testing on Windows and Linux it looked fine. however on their browser, the horizontal line of buttons for the menu broke into two lines. It was not until I finally did remote control on her PC was I able to see what was going on.

Basically, on first load it rendered fine, but as soon as you went back to the page, the buttons would break. Inspecting it in firebug indicated that the second time it rendered the DOM tree was actually different to the HTML source.

The button code looks a bit like this

<a href="somurl"><div class="btn"><div class="btn-right"><div class="btn-left"><div 
   class="btn-body">Title here</div></div></div></div></a>

The idea is that the div's provide the rounded button edges, and allow nice hover effects all in CSS.

however inspecting the reloaded page indicated the tree looked like
<a href="somurl"></a>
<div class="btn"><div class="btn-right"><div class="btn-left"><div
         class="btn-body">Title here</div></div></div></div>
Where the a tag had broken away from the div, strangely only on one button.

It was only after I ran it through the w3 validator and  got an error message about block elements inside and inline one, that I had the idea to rewrite that HTML to use span's, rather than div's. Along with adding display:block to all the CSS for the button elements.

The resulting HTML being
<a href="somurl"><span class="btn"><span class="btn-right"><span class="btn-left"><span 
        class="btn-body">Title here</span></span></span></span>
And amazingly enough it rendered perfectly on all browser.. strange but true....

22 Jan 2011

mtrack and work flowing commits into a live site

Working with outsourced resources can be frustrating at times, In the past I've used various techniques for this, normally it involved outsourcing a large bulk project at a fixed cost, setting down a few rules about code quality, design etc. and letting them get on with it. This works reasonably well, as the mess that get's created is controllable if they have followed the rules.

More recently I've been working with outsourced contractors who work on an hourly basis. The results have been mixed, and as we do it more frequently we are beginning to refine our working process.

Last Thursday however the client for one of the outsourced projects called frantically wanting the live site refreshed to show the development changes. Luckly we had decided to go with revision control only access to the site some while back (and as one of my previous posts mentioned the mess before that, and our problems with git, we concluded the contractors where not capable of using it, so we had ended up with a subversion frontend committing into a git backend)

To make the site go live is just a matter of running git pull in the live directory (and ocassionally git reset --hard to clear out any crap.) However after urgently updating the site to the current development state, a horrible problem was noticed on the front page. And I was tasked to try and fix it really quickly. 

To my horror though, as I had taken a hands off approach on this project (due to budgeting requirements), the code was in a far worse state than I feared. A few weeks ago we had started trying to force the contractors to follow some basic good coding practices, like commit messages with meaningfull descriptions, do not use the development server as a test server and always commit unix line breaks etc. This was all done via commit hooks on subversion, and commits where being rejected frequently (much to my evil amusement).

However this was not enough to prevent code that had been created many months before getting worse with age (bad decisions, with no sensible review process being made worse by more feature requests.). The result was that what should have been a simple one line fix to change the formating of a currency output, ended up with trying to understand some 400+ lines of jiberish. 

At this point, we concluded enough is enough. the cost savings of not reviewing this code previously was going to cost more and more in the future unless this mess was stopped. Hence mtrack came into the picture..

For those un-aware, mtrack is Wez Ferlong's project to replace trak, with a PHP based implementation. It's relatively new, and looks like it was developed for a need that Wez had internally.

My idea was that we would continue to allow the developers to commit into the repository, the only difference would be that they would have to add ticket numbers to the commit messages, and we would have a simple review process for the code using mtrack, so that we would only close an issue when it had been fixed to a reasonable level of code quality (and worked properly)

Setting up mtrack is extremely simple. the introduction instructions will get you started, however as I found doing anything more complex that what is provided requires modifying significant chunks of the code.

The overall sense I get from mtrack is that it has the potential to be an outstanding project. It could rival things like bugzilla, however, it could seriously do with some tidying up, and rethinking of some parts.

Anyway read the extended article for more detail thoughts as I started to implement our desired workflow with mtrack. It may help if you had diving into mtrack as well.



Posted by in PHP | Add / View Comments()

18 Jan 2011

Easy way to make Word documents with images from PHP

 It's yet another "solve this quickly and cheaply" problems....

How to produce nice Word documents with images and all, without ending up doing lots of hard coded PHP calling some obtuse library in a way that would be difficult to maintain.

This one took a bit longer than expected, my first idea was to generate HTML files, then run them through abiword's command line, as that was the hints I got from googling.

abiword --to=doc  myfile.html

Unfortunately, although looking at abiwords source code, it should have  worked, nothing actually came out. After trying a few other magic tools like unoconv I was getting close to admitting defeat.

However, after a nice walk away from the desk and putting the thinking cap on, it came to me.. abw is abiwords xml file format. It's about as simple as HTML. So why not generate a abw file, and convert that into word.

Images in abw files are just base64 at the end of the file, each given a specific name, and reference in the body of the document where you want to use them. Trivial to generate... and even easier if you cheat a little by not using an dom based xml generator for the content (good old Flexy templates). then just run them through abiword as planned.

abiword --to=doc  myfile.abw

Quick, easy and pretty Word documents generated from web pages..
Posted by in PHP | Add / View Comments()

28 Dec 2010

Looking at opencart


There's been quite a demand recently for online shops, While I do have a custom online shop, It's currently quite specific to a client, and does not have a generic frontend. So any kind of re-purposing involves creating a new design. Obviously this looks like a good long term plan, along with open sourcing it properly.

However in the meantime, I discovered opencart a few weeks ago, and for a quick and dirty shop delivery it's not to bad.

On the positive side, installation and basic setup of products is very simple, we checked the code out into git-svn and ran through the installation process, commiting changes, then used the git-ftp code to upload it to the target server.

This gave a managable installation, with full revision control, and after adding the clients logo to the site, setting up things like paypal and shipping etc. we let them loose adding their products.

All of this was quite smooth, but then came the real fun. "But can it do this..." the classic second sentence after you get started..

Their first request was to add extra fields on the product description.

In a perfectly designed system (eg. the shopping system I had already designed), this would be a matter of.

  1. add some extra fields to the database (eg. lead_time VARCHAR(32) ) - eg. in a local_mods.sql file.
  2. add some extra HTML  to the two templates or using app.Builder.  (in the 'overridden templates' folder) - which Template_Flexy supports
This would be feasible as the model layer would not need to understand much about random extra columns in the database that did not really mean to much to the application, but are always essentail  for the client. This is all possible as FlexyFramework/DataObjects/Template_Flexy combo handles extra columns without any modification to controllers or models. No more than 10 minutes work..

To follow the same approach with opencart however would have involved
  1. modify the database
  2. modify the model code for the product (checking each method to make sure they added any relevant hand coded SQL)
  3. modify the admin model code for the product (checking each method to make sure they added any relevant hand coded SQL)
  4. checking the controller class so that it added extra variables to the view layer, the controller layer for just viewing a product is over 400 lines, in what should be done in around 50. 
  5. checking the controller class for the administration side to see if it correctly send the data without understanding the model had changed (which actually it did)
  6. adding some HTML (and lightly sprinkled PHP) in the template for the frontend, using rather verbose, non-templated language, and ensuring it's all correctly escaped.
  7. adding some HTML (and lightly sprinkled PHP) in the template for the backend, using rather verbose, non-templated language, and ensuring it's all correctly escaped.

In the end I concluded that even making all these changes would result in a forked codebase that would become increasingly more complex to maintain, as the client demanded more changes, and the opencart released more versions.

What I ended up doing was using their category mechanism, and creating categories for different lead times, then modifying the front end controller to read the hidden extra categories, and creating a forked template for the frontend.

Even this is still a problematic fork, as SQL needed modifying in model, and extra code was added in the controllers

Having said all that, the experience was far smoother than the mornings job of creating out a development version of a Joomla site, that suffers from the horrific problem that the codebase can be modified by the Web frontend making revision control an absolute nightmare. It's a even better lesson in poor design....
Posted by in PHP | Add / View Comments()

23 Dec 2010

And now for some Christmas entertainment, git, outsourcing and PHP error messages.

Well, you either have to laugh or cry at this.

One of the projects I help out on, mostly by looking after the production server, uses Joomla, the owner has employed a number of outsourced resources via odesk to do the development, in the belief that it's cheaper that way. 

After a few months of them modifying the code and occasionally breaking the site, I changed the configuration so they had to commit via git, which in turn would copy the commited code onto the live server. So there was at least some tracking of changes they where making. 

Before this, they had been in the habit of modifying a file like index.php and calling it index-2010-04-12.php then uploading via ftp, and there was about half a dozen versions of each file on the server, (basically quite a mess)

So since git had been so mindblowingly usefull for all my projects, I made the assumption that it would be effective as a revision control system on the server.

Unfortunatly over the last few months that decision has come back to bite us. 

a) pretty much all the contractors they employed used windows (usually a sign of an inexperienced developer)
b) git on windows is no where near as mature (cygwin being slightly better that the msysgit version), especially when handling http-backend based repositories.
c) the contractors ran into all sorts of problems dealing with the command line and messages like 'error: Untracked working tree file 'images/M_images/Thumbs.db' would be overwritten by merge.  Aborting' - requiring either teamviewer sessions to walk them through the fixes or rather fruitless email conversations explaining what they should do.
d) the contractors had no idea how to use command line mysql (I even saw this in one of their logs - #git mysqldump -h... )
e) they would send me screenshots of the cygwin terminal???? copy and paste anyone...

So in the end I've given up on letting them use git directly, and installed subversion with a commit hook that copies the change into the git central store.

But that was the least of the problems I've seen. When the client started looking at his billing he was beginning to wonder what he was paying for. In one bill some 8 hours had been billed, yet there had been little to no change on the site. odesk provides screenshots of what the contractor was doing so I thought I would  have a look.

It appears the problem lay with their lack of understanding of PHP (the language they claim to know to be able to do these jobs). Our server is running PHP 5.3, which produces a few new warning messages, causing problems with older code. I've insisted that we do not downgrade, as the exim exploit illustrated, depending on the distribution provided packages for something like this can be the difference between a instant upgrade and messing around with build issues while a server is wide open for attack.

Anyway the problem they ran into was

Deprecated: Assigning the return value of new by reference is deprecated

this was in the nusoap library, and caused by this syntax.

 $this->wsdl =& new wsdl(.......

The fix is very trivial, just remove the & before the new (it's not needed in PHP5). however, rather than doing this, first they tried googling it, and apparently after not understanding the error message, they decided to comment out the whole body of the loadWSDL()  method, then went on to find all the calls to the method, and comment them out.

I guess it might be worth changing that error message to  "There is an '&' before 'new' on line XXX, it is not needed, so you should remove it."

Then again, as I keep reminding the client, you get what you pay for....

Posted by in PHP | Add / View Comments()

10 Oct 2010

Leadership lessons, or are MBA courses any good?

And now for something different.. I have a friend doing a course at HKU on Leadership, and with the feedback I'm getting, the course might just be missing the plot a bit.

From my side of the fence, ever since I gave up on my Masters, I have been generally dismissive of post-graduate courses at Universities. There was a serious sense of 'If you can't do it, teach it..' (or now if you really can't do it, blog about it..). My sense is that there is a reasonably large body of acedemia that suffers from the fact that they are kept on board as a result of quantity of research publications. They however they do not appear to understand the principles behind teaching (which teacher training wonderfully encapsulates). I wonder if tutors at university should be required to undertake a teacher training course, rather than slowly migrating from research into tutoring.

Along with all this, I wonder if the type of students these course attract are the exact opposite to 'self-learners', and are more used to a spoon feed education, especially here in asia. 

Anyway back to the subject at hand. HKU was teaching a course based around a Harvard case study about a climb of mount Everest resulting in a number of deaths. The students are given the course material and are asked to produce a short piece explaining it and illustrating their ability to extract the leadership lessons that caused the failure.

This is where adult reading (not the picture kind) comes in handy. In Malcom Gladwell's book Outliers, a chapter is dedicated to an interesting study of plane crashes, and pilot assertiveness. If you read the book in the wrong way it appears to sensensionalize the idea that different countries and cultures have different levels of assertiveness which lead to greater crashes with pilot's of different countries. While this inference tends to lead to great press and I presume book sales. It is not really the message that come accross if you read it in detail.

In essense the book points out that individuals (and perhaps to some degree culturally based) are less assertive, in that they feel unable due to their role in society of workforce to assert their opinions. In the case of the pilot's it was the quiet mention that there might be a small issue with the weather. In the case of the climb it was more a case of Doctor's not asserting the seriousness of health issues, or the sherpa's inability to express that they where concern with various issues about the trip. All these would have likely been non-fatal if both communication had been clear and strong, and the subordinates had felt comfortable in the life threatening situations in overriding the leaders actions.

From the sense I got, the course was being taught on leadership, but unfortunatly learning that there are problems occuring with these type of situations has very little to do with leadership, and more to do with realizing that assertiveness training is an essential part of senior management, and more essential in situations which may be life threatening. 

What is worse is that the course dynamic's of putting groups into teams and tasking them with these studies looks like it is a perfect example of why failure happens. Rather than using the course to entrench this positive assertiveness it basically leaves the groups to their own devices to produce a result and does not monitor their interaction trying to illustrate the individual's in the groups own abilility to assert their opinions.  Would this type of couse not be better taught better with some kind of situation based role playing and instant feedback. 

19 Sep 2010

Barcamp Hong Kong 2010

Had a great time at Barcamp Hong Kong yesterday, Saw some really good talks on Y-Combinator , portable Mobile app development with a Webkit wrapper and gave a rather disastrous talk on Javascript (due to technical problems with my laptop, nvidia cards and projectors...)

Anyway here's the slideshow I gave in full..

I will update this post later - with a few more details

01 Sep 2010

Big step forward in Modular Database Applications with DataObjects

Being a Software Developer is all about developing applications faster, and delivering quicker. At the same time ensuring that quality is not lost and readibility is kept. DB_DataObjects is one of the key tools in my productivity toolkit. It was originally designed as a way of ensuring that shared database related code ends up in the right place, In the Model layer of the application, rather than the View or Controller (or worse mixed in to some hybrid PHP HTML garbage..), along with doing cool stuff like Query building etc.

Over the years, of using DataObjects, I've built up quite a library of reusable DataObjects, which to some degree can be plugged in to any project. A Person object that handles login/authentication and authorization (working with the Permissions and Group objects). A Image object that handles storage of Images, and Files that can provide File type conversion for rendering (using things like unoconv, etc.)

More recently, I've been using the Roo Javascript library as the UI component, the Pman Components ontop of the very lightweight HTML_FlexyFramework. The result is a very modular set of Application development parts. That can quickly be thrown together to build applications. Here's how they all fit together and how it just got a whole lot more modular and flexible...


Posted by in PHP | Add / View Comments()

29 Aug 2010

When unstable is just not unstable enough

..It all started by having yet another look at the gir problem..... Keeping them up-to-date, and trying out the latest cool Gnome toys.

« prev page    (Page 4 of 24, totalling 233 entries)    next page »

Follow us on

OUR BLOG