Big step forward in Modular Database Applications with DataObjects

Published 2010-09-01 00:00:00
Being a Software Developer is all about developing applications faster, and delivering quicker. At the same time ensuring that quality is not lost and readibility is kept. DB_DataObjects is one of the key tools in my productivity toolkit. It was originally designed as a way of ensuring that shared database related code ends up in the right place, In the Model layer of the application, rather than the View or Controller (or worse mixed in to some hybrid PHP HTML garbage..), along with doing cool stuff like Query building etc.

Over the years, of using DataObjects, I've built up quite a library of reusable DataObjects, which to some degree can be plugged in to any project. A Person object that handles login/authentication and authorization (working with the Permissions and Group objects). A Image object that handles storage of Images, and Files that can provide File type conversion for rendering (using things like unoconv, etc.)

More recently, I've been using the Roo Javascript library as the UI component, the Pman Components ontop of the very lightweight HTML_FlexyFramework. The result is a very modular set of Application development parts. That can quickly be thrown together to build applications. Here's how they all fit together and how it just got a whole lot more modular and flexible...



HTML_FlexyFramework Provides the basic 'Controller', It's one of those very lightweight, get out of your way, 2  small classes bit of code. In some sense's it's not really  a Framework (well 2 classes could hardly classify as that). More a generic Controller, and a Base class for a Page, just implementing the bare bones stub features, like get/post,getAuth() and implementing the output method to run HTML_Template_Flexy.

What I've built ontop of that is the Pman Components, originaly based on a project codenamed Pman (for project management). Which ended up being the most re-usable code I've developed to date. It's the foundation for around 6 completely different projects now, sharing a large amount of modular code. The basic code of the Pman Project is Pman.Base and Pman.Core, in themselves they do not provide much more than the basic Interface outline. 

The directory structure for projects

  • /index.php  (the bootstrapper and installed configuration- the link should show a rather messy example)

  • Pman.Base (git submodule)
  • Pman.Core (git submodule)
  • Pman.Admin (git submodule)
  • ......... (any other submodules for the project)

  • 'Pman.Base' files and where they end up in the repositor.
  • Pman.php (alias of Pman.Base/Pman.php) - this provides base feature for any custom code, like jerr(), jdata() and jok() for sending database via JSON in standard formats.
  • Pman - Technically where all the real files for the submodules go..
  • Pman/Roo.php (alias of to Pman.Base/Pman/Roo.php) - The base interface for pretty much all Database Queries from the Roo Javascript Library. - this may end up in Core..
  • Pman/I18N.php (alias of to Pman.Base/Pman/I18N.php) - To be phased out eventually - most of the code is now in Pman.Core/I18N.php
  • Pman/Login.php (alias of to Pman.Base/Pman/Login.php) -The base provider of Login Queries - both initial login, background information refresh and logout
  • Pman/Images.php (alias of to Pman.Base/Pman/Images.php) - Generic provider code to deliver images to the interface (either as attachments, thumbnails or the real thing.)
  • Pman/templates/images (alias of to Pman.Base/Pman/templates/images) - contains images and icons.. that will probably be moved upstream into Roo.

  • 'Pman.Core' files aliased into the 'Pman/Core/' 
  • Pman.js = The Previous base class for Pman. Some of this code has been moved to Roo.XComponents, but it still provides a few usefull methods for all applicaitons, like Pman.download() etc.. 
  • Pman.Dialog.*.js - some standard dialogs for editing generic data.
  • DataObjects - the Core DB_DataObject defintionts.
  • template - the standard master template 

  • Project Specific code 
  • Pman/TheProject/*.js
  • Pman/TheProject/*.php

There's quite a bit more code and data in those Modules, but what changed yesterday and has a significant benefit to the long term re-usabilty of all this code is the autogeneration of cached data for DB_DataObjects schema files, automatic merging of DB_DataObject links files, and the generation of code for Roo Javascript's Reader data.

A Changing Data Model


The background here is that, as I re-used more and more of these componets, there would frequently be an issue where a particular project needed the modification of a core database component. In our example it was the adding of 'dispatch port' to the Company Table. This was a specific requirement for one project, which could have been impemented using a Companys_Extra table or some kind of 'properties' hack (which is what Midgard used to do back in early 2005). However the best solution is really to modify the base Table, so that it can be searched/ordered etc, and in the long run I'm looking at having a 'Designer Module' similar to SugarCRM / netsuite, that allows the end users to actually modify the database structure, adding colums as they need. 

The Knock-on effect however for this small change is, unfortunatly it often breaks other code depending on the compoents, due to the process that is involved.

a) Modify the database changes file. normally in the DataObjects directory of the Component
b) Run it against the Database, updating the Data Definition for tables in that module.
b) Run the generator on the module, so that the schema.ini file is regenerated.

This works fine on the development server, and usually fine on the Live server for that project, as you remember to run the database changes from the relivant Module when you do the next install, Where this falls apart is usually a month later...

You go back to working on a project that was on hold for a while, and start by updating the modules in the project. Suddenly Authentication fails, or no data is available in the interface... - All this is due to the fact that the Database Queries being built expect certain columns to be there (based on the schema.ini file) and since you have forgotten to update the database all the queries break.. Not usually a big issue as the builder module has some code in it to apply all the database changes code, by runing the bootstrapper in CLI mode. - But still it's annoying.

The other issue is that you really do not need the Company Table to contain a 'dispatch port' when no other project is going to use it.

The Solution..

I have been thinking about the solution to this for some time, however, just before I jumped into refreshing some code in a project, I thought this would be the ideal time to solve this problem, and speed up my development process in the long run.

The change involved
options:
dataObjectsCache = true (defaults to ON)
dataObjectsCacheExpires = 3600 (database Cache file expires every hour..)
$FlexyFramework->generateDataObjectCache(force=false) method
which can be called to force regeneration of the ini files (stored in the session directory)
HTML_FlexyFramework_Generator - a tiny wrapper around DB_DataObject_Generator, that is loaded and called only when regeneration takes place. 

- when you load the main page (not when you do JSON calls) it calls generateDataObjectCache(), which will generate the schema files if they have expired. - if you are running with Pman[isDev] then it will run them every time (eg. force generation)..

- new get Request argument _requestMeta
which adds to the data return value the JsonReader meta arguments which include schema type information, so you do not need the schema information in the UI (reducing the size of the UI considerably). note schema data is only returned if rows of data where also returned..

- new property metaFromRemote - a flag to indicate that the meta data was recieved from a remote source (and hence probably does not need fetching again).


- load method modified to check for metaFromRemote, if it is set to false, then _requestMeta is added to the data Query. (hence schema is only queried once).


All this should mean that I can move application specific database changes to core tables into their respective modules, and out of core..

Comments



Add a comment (requires javascript!)
Name
Email
Homepage
Comment Title
Comment
I am a link spammer?