RooJSolutions http://roojs.com/index.php/View.html en http://roojs.com/Roojscom/templates/images/roojs_square_logo_150.png RSS: RooJSolutions - /index.php 150 150 Cli parsing in FlexyFramework, PEAR Console_GetArg 2011-08-11 00:00:00 http://roojs.com/index.php/View/241/Cli_parsing_in_FlexyFramework_PEAR_Console_GetArg.html <a href="http://roojs.com/index.php/View.html">Article originally from rooJSolutions blog</a><br/> <font size="2">&nbsp;And another rare article get's published, I've been slacking off posting recently. As I've been busy getting some interesting sites online. The biggest being a rather fun viral advertising campaign on facebook<a href="http://www.facebook.com/deargoodboy"> www.facebook.com/deargoodboy</a>. Which I ended up project managing, after originally only committing to do the facebook integration.</font><div><font size="2"><br></font></div><div><font size="2">Anyway back to the open source stuff. One of the tasks I've had on my todo list for a while is revamping the CLI handling of my framework, Which probably has a tiny following, but is one of those increadably simple, yet powerfull backbones to all my projects.</font></div><div><font size="2"><br></font></div><div><font size="2">While this article focuses on the changes to the framework, it should also be of interest to others developing frameworks, and anyone interested in using PEAR's<a href="http://pear.php.net/Console_GetArg"> Console_GetArg.</a></font></div><div style="font-size: 13px; "><br></div> DataObjects links.ini and Archiving database data - an ideas of march post.. 2011-03-22 00:00:00 http://roojs.com/index.php/View/231/DataObjects_linksini_and_Archiving_database_data__an_ideas_of_march_post.html <a href="http://roojs.com/index.php/View.html">Article originally from rooJSolutions blog</a><br/> <div><span style="font-size: small; ">Trying to keep up with the Ideas of March - and get a few more posts out.</span></div><div><font size="2"><br></font></div><div><font size="2"><a href="http://pear.php.net/DB_DataObject">DB_DataObject</a> is the workhorse behind most of my project work. It saves a huge amount of development time, and makes code considerably simpler to understand, modify and maintain. Recently I've added a few new features to this granddaddy of ORM's, and solved a perpetural data management problem at the same time.</font></div><div><font size="2"><br></font></div><h3><font size="2">AutoJoins</font></h3><div><font size="2">Last month saw the commit of the autoJoin method to pear's svn server, this is a cut down version of something that's already in use in the <a href="http://www.roojs.com/mtrack/index.php/File/default/Pman.Base/Pman/Roo.php?jump=">Pman Roojs interface layer</a>. The basic problem it solves is that when you are doing rapid development, you do not really want to be spending time writing the code that add's all the tables together into a join.&nbsp;</font></div><div><font size="2"><br></font></div><div><font size="2">for example a typical list people with their office details would look something like this.</font></div><div><font size="2"><br></font></div><pre><font size="2">$person = DB_DataObject::factory('person');<br></font><font size="2">$office = DB_DataObject::factory('office');<br></font><font size="2">$person-&gt;joinAdd($office,'LEFT');<br></font><font size="2">$person-&gt;selectAs();<br></font><font size="2">$person-&gt;selectAs($office, 'office_id_%s');<br></font><font size="2">$results = $person-&gt;fetchAll();<br></font><font size="2">$ret = array();<br></font><font size="2">foreach($results as $r) {<br></font><font size="2">&nbsp;&nbsp; &nbsp; &nbsp; &nbsp;$ret[] =$r-&gt;toArray();<br></font><font size="2">}<br></font><font size="2">echo json_encode(array('total'=&gt;count($ret), 'data'=&gt;$ret));</font></pre><div><font size="2"><br></font></div><div><font size="2">This would rely on a links.ini file describing the table connections</font></div><div><font size="2"><br></font></div><pre><font size="2">[person]<br></font><font size="2">office_id = office:id</font></pre><div><font size="2"><br></font></div><div><font size="2">With the new autoJoin() method, this code is reduced to&nbsp;</font></div><div><font size="2"><br></font></div><pre><font size="2">$person = DB_DataObject::factory('person');<br></font><font size="2">$person-&gt;autoJoin();</font><font size="2"><br></font><font size="2">$results = $person-&gt;fetchAll();<br></font><font size="2">$ret = array();<br></font><font size="2">foreach($results as $r) {<br></font><font size="2">&nbsp;&nbsp; &nbsp; &nbsp; &nbsp;$ret[] =$r-&gt;toArray();<br></font><font size="2">}<br></font><font size="2">echo json_encode(array('total'=&gt;count($ret), 'data'=&gt;$ret));</font></pre><div><font size="2"><br></font></div><div><font size="2">This not only simplifies the code, but if we change the database design and add a company field to the person table and change the links.ini file to</font></div><div><font size="2"><br></font></div><pre><font size="2">[person]<br></font><font size="2">office_id = office:id<br></font><font size="2">company_id = company:id</font></pre><div><font size="2"><br></font></div><div><font size="2">Then the resulting data will include the person's company details, just by adding a single line in the links.ini.The original code in Roo.php enabled filtering and distinct queries to be built automatically - this may one day end up as some of the options as a parameter for autoJoin().</font></div><div><font size="2"><br></font></div><div><font size="2"><br></font></div><h3><font size="2">Archiving data</font></h3><div><span style="font-size: small; ">This link.ini schema map file has another huge benefit, which I've now realized with a quite simple piece of code to archive data from a database, so it can be removed and restored later. This was necessary on two projects, one where the database contained press clippings about a client company for a media monitoring project. In this case the owner wanted to remove all the clippings and users relating to a specific client as they where no longer using the service, and it was taking up disk space. Along with this it was obviously slowing down the queries due to the large data size.</span></div><div><font size="2"><br></font></div><div><font size="2">The other problem area was our mail filtering and anti-spam product. It &nbsp;had a rolling archive of all mail delivered to a client, which was&nbsp;automatically&nbsp;pruned using cron jobs. On most of our servers this was limited to about a week, however on one client we had data in the database for over a years worth of emails for 500 staff. With that amount of data, it was consuming a considerable amount of disk space and certian searches where rather slow.&nbsp;</font></div><div><font size="2"><br></font></div><div><font size="2">My concept to solve both issues was basically the same. using the relationship data in the links.ini file, it is possible to work out what data in the database is both 'owned' and required when exporting and restoring data.</font></div><div><font size="2"><br></font></div><div><font size="2">In the mail example, each email is referenced by a few rows in the Attachment table (so they could quickly search for files). So when we export the emails we could also export and flag for deletion the related Attachments. Along with this the mail row had references to the Senders, who we would have to exist if we restored the data, however we did not need to delete them.&nbsp;</font></div><div><font size="2"><br></font></div><div><font size="2">The solution to all this is currently a class available in the Pman Admin module called <a href="http://www.roojs.com/mtrack/index.php/File/default/Pman.Admin/Dump.php?jump=">Dump</a>. It could quite easily be moved into a library DB_DataObject_Dump. As long as you have your DB_DataObject configuration set up correctly, it takes a table and a query, and locates all the records from related tables that can be deleted or are required for a restoration then creates a number of files.</font></div><div><font size="2"><br></font></div><div><font size="2">a <b>restore.sql </b>- which can recreate the data which has been removed, using a large number of INSERT statements for the core data, dependant records and related records.</font></div><div><font size="2">a <b>deletes.sql</b> - which removes the core data and dependant from the database.</font></div><div><font size="2"><br></font></div><div><font size="2">Along with this it also supports hooks (or a dumb class based method API), which are just methods that can be implemented in the Dataobjects being dumped, that enable you to copy and restore the files relating to the database records (for example our original emails.) The second part of the script uses that information to generate the following shell scripts</font></div><div><font size="2"><br></font></div><div><font size="2">a<b> restore.sh</b> - which copies files from the backup into the original location</font></div><div><font size="2">a <b>backup.sh </b>- which copies the files from the original location to the backup location</font></div><div><font size="2">a <b>delete.sh</b> - which deletes the files from the original location.</font></div><div><font size="2"><br></font></div><div><font size="2">All I need to do now is write a nice UI for it all...</font></div><div><font size="2"><br></font></div><div><font size="2"><br></font></div><div><font size="2"><br></font></div><div style="font-size: 13px; "><br></div> SQL change management done right. 2011-02-13 00:00:00 http://roojs.com/index.php/View/229/_SQL_change_management_done_right_.html <a href="http://roojs.com/index.php/View.html">Article originally from rooJSolutions blog</a><br/> <div><span style="font-size: small; ">I'm slowly going through mtrack at present, breaking the HTML rendering into templates, and making nice clean classes for the fetching of underlying data. All this is due to implement quite a few ideas once it's in a state to be easily changed.</span></div><div><font size="2"><br></font></div><div><font size="2">The more deeply I go through mtrack though, there are parts I find that you look at &nbsp;and think, "That's a really smart way to handing that problem". Like the change control auditing, where each component has a ID for the last updated (and created) This maps to a event table reord containing who/when data. Much cleaner than the more common practice of using two datetime and user fields.</font></div><div><font size="2"><br></font></div><div><font size="2">However, as always, there are parts of the code where you want to pull you hair out and say, No way should you solve a problem like this. Mtrack's SQL change management is one of those areas.</font></div><div><font size="2"><br></font></div><div><font size="2">It's approach is not uncommon, and I've seen it before. Basically it uses an XML file to define the schema and then has a series of files schema1.xml,schema2.xml,schema3.xml... and so on.&nbsp;</font></div><div><font size="2"><br></font></div><div><font size="2">The installer works out what the current version is, then compares schema's between the current and selected version. and works out how to update the database. There is a driver for PostgreSQL and SQLite.&nbsp;</font></div><div><font size="2"><br></font></div><div><font size="2">My&nbsp;fundamental&nbsp;issue with all this is that while on the face of things it does not seems like a bad idea, however, it ignores the plain simple fact that there is already a language used for Database Schema definitions and modifications, and it's called SQL!</font></div><div><font size="2"><br></font></div><div><font size="2">Anyone who uses SQL can quickly read a SQL file and understand what it's doing, even if it's in a different dialect (SQLite etc...), but putting that type of information in a XML file just adds so much noise. Worse of all, it involves learning and remembering a small subset of knowledge, that is only&nbsp;relevant&nbsp;to a tiny problem domain. Then the worst thing is it's just plain annoying to read.</font></div><div><font size="2"><br></font></div><div><font size="2">For my projects I'm luckly only having to deal with a single database vendor (MySQL usually). To manage changes I keep SQL updated, this file contains all the table definitions along with the later changes. To update the database, I just pipe it through the mysql command line with the '-f' (force) switch. This is a trivial, simple and effective way to keep the database schema in sync wit the project code. There are even wrappers in the Component Framework code to do this simple task for you</font></div><div><font size="2"><br></font></div><div><font size="2">The big problem however is that SQL, for all these great benefits has turned out to be horribly incompatible between database vendors. If I carry on with the above idea of keeping my database modifications in a SQL file, &nbsp;then I would end up with one for each of the databases I want to support. Then I not only have to keep each of the schema files up-to-date, but have to remember the syntax for multiple database vendors, and apply each change to every file, not really a good long term plan.</font></div><div><font size="2"><br></font></div><div><font size="2">So rather than keep multiple files up-to-date, I wondered, why not convert the SQL schema changes from one dialect to another, and just keep an SQL file as I currently do, and make it feaasible for anyone to install using the database of their choice.</font></div><div><font size="2"><br></font></div><div><font size="2">This is one of those 'why has no-one done this before moments', but for the life of me, I could not find anything that came up to quickly on google. So I had a look at what was 'close enough' for this idea to work, and what a supprise, most of the code for this is already in PEAR.</font></div><div><font size="2"><br></font></div><div><font size="2">The <a href="http://pear.php.net/package/SQL_Parser">SQL Parser</a> package, as very basically introduced here,<a> </a><a href="http://www.sjhannah.com/blog/?p=16">http://www.sjhannah.com/blog/?p=16</a>,&nbsp;provides pretty much all the backend code for a solution to this. However, there was not previously any code in either the parser, or writer/compiler to actually deal with DDL commands like alter table etc.</font></div><div><font size="2"><br></font></div><div><font size="2">I've just commited the changes for this, so you can now very easily extend the current SQL_Parser_Compiler class to output your favourite dialect of SQL, based on reading an SQL file containing the base changes.</font></div><div><font size="2"><br></font></div><div><font size="2">For a example of how to use it in real life, here's a nice simple example from the forked mtrack codebase.</font></div><div><font size="2"><a href="http://git.roojs.org/?p=web.mtrack;a=blob;f=web/CreateSchema.php">http://git.roojs.org/?p=web.mtrack;a=blob;f=web/CreateSchema.php</a></font></div><div><font size="2"><br></font></div><div><font size="2">And here's the commit that makes it possible..</font></div><div><font size="2"><a href="http://svn.php.net/viewvc?view=revision&amp;revision=308303">http://svn.php.net/viewvc?view=revision&amp;revision=308303</a></font></div><div><font size="2"><br></font></div><div><font size="2">And finally, a good example of a SQL file that can be run through the parser/generator.</font></div><div><font size="2"><a href="http://git.roojs.org/?p=Pman.Core;a=blob;f=DataObjects/core.sql">http://git.roojs.org/?p=Pman.Core;a=blob;f=DataObjects/core.sql</a></font></div><div style="font-size: 13px; "><br></div> PEAR state of play, why move to PEAR2 2009-09-21 14:16:30 http://roojs.com/index.php/View/181/PEAR_state_of_play_why_move_to_PEAR2.html <a href="http://roojs.com/index.php/View.html">Article originally from rooJSolutions blog</a><br/> Dataobjects, Flexy Releases - and Ext-Roo builder for FlexyFramework. 2009-03-13 12:22:00 http://roojs.com/index.php/View/177/Dataobjects_Flexy_Releases__and_ExtRoo_builder_for_FlexyFramework.html <a href="http://roojs.com/index.php/View.html">Article originally from rooJSolutions blog</a><br/> crazy require_once optimization rears it head again. 2007-07-14 10:14:51 http://roojs.com/index.php/View/148/crazy_require_once_optimization_rears_it_head_again.html <a href="http://roojs.com/index.php/View.html">Article originally from rooJSolutions blog</a><br/> There's been a long thread the last couple of weeks covering require_once and some rather crazy ideas to change the PEAR standards to do all sorts of strange things to load code up.<br /><br />What is most ridiculus is the benchmarking that is being done to 'prove' that we should all move our code loaders to some 'allfiles.php' and we are magicially going to running be 15% faster. <br /><br />What this whole concepts fails to put into place is loading all the PHP files up, either tiered one to another or all from one place is such a small part of a PHP page responding from a request.<br /><br />Think of what is happening when you look at this page.<br /><ul><li>You start by looking at a bootstrapper (that sets config option) - probably quite quick as it just set's some variables.</li><li>Then you run into the Framework class - that works out which file to load based on the URL you are typing. - This does quite a bit of text maniplation, making best guesses about where things might be. (this is quite generic and could be made specific to the application quite easily)<br /></li><li>We then do the page action stuff, like pulling data from databases which requires quite a few PEAR libraries. And does some really slow crap like connecting to databases and pulling data. This stage probably does far to many queries and pulls down to much data that it doesnt use.</li><li>Then we do output stuff, Normally we are using a compiled template, (so we dont load all the template parsing code - well at least we save a bit of effort here). so we need to pull down at least the Flexy class, then the compiled template.</li></ul>Now looking at that whole process and thinking it's a bit slow, You would probably go through the following steps to optimize it.<br /><ul><li>replace it with a staticly generated page!!!</li></ul>Well, yeah, that's it!.... <br /><br />Only kidding... - but if you do this to a normal page, you would probably want to do the following<br /><ul><li>reduce / remove database calls</li><li>cache as much as possible.<br /></li><li>reduce / remove libraries needed and code used.</li><li>move any data intensive code into C</li><li>use an opcode cacher.<br /></li></ul>At no point would you get so silly that you would re-jig the 'require_once' calls just to get 15% saving on the code loading component, because wait for it..... the code loading component of your application (when using APC) would take such an insignificant percentage of the total running time.<br /><br />Working out how to remove a whole file or pre-process some bit of data would be a far better use of your time. And if you where running a team focusing on this you would probably have fired the guy who wasted all that time on such a pointless task as moving the require_once lines around.<br /><br />Sounds like moving the deck chairs on the titanic?<br /> Multilanguage setup for Flexy and FlexyFramework. 2006-11-24 15:02:00 http://roojs.com/index.php/View/124/Multilanguage_setup_for_Flexy_and_FlexyFramework.html <a href="http://roojs.com/index.php/View.html">Article originally from rooJSolutions blog</a><br/> Having just released the increadibly minor update to the Flexy Template Engine in pear, I can now release the documentation that goes with it... <br /><br />FlexyFramework, with HTML_Template_Flexy include the ability quickly create multilingual websites, by translating the templates.<br /><br />I've set this up a few times now, and kept meaning to document it. - So here it goes.<br /><br />Having set up FlexyFramework, (see the previous post about this), <br /><br />... The full instructions are in the Extended entry....<br /><br /> Flexy, the condom of template engines updated 2005-07-23 00:00:00 http://roojs.com/index.php/View/95/Flexy_the_condom_of_template_engines_updated.html <a href="http://roojs.com/index.php/View.html">Article originally from rooJSolutions blog</a><br/> After seeing quite a few "fixed XSS problem" cvs commits on the mailing lists, I finally remembered that a release of Flexy was overdue. The thought, "that would not have happened if they had used flexy" came to mind a bit too often.. Flexy is designed to be safe by default (like a firewall where you have to try hard to open ports)<br><br><a href="http://pear.php.net/HTML_Template_Flexy">HTML_Template_Flexy</a> has not had much added to it in the last 6 months, probably as it just 'works', tries not to get in your way, or do to much.. But there have been a few bugs left over from the last release in January, along with a few tiny feature requests. So I finally got round to tidying up a release.<br><br>There is nothing mind shattering in this release, But users and potential users may be interested in the extra pages in the <a href="http://pear.php.net/manual/en/package.html.html-template-flexy.php">manual.</a> They should come online by monday:<br><ul><li>flexy:nameuses<br></li><li>flexy:tojavascript jsvar="phpvar"</li></ul>Along with a pretty complete list of configuration options that I updated a few weeks ago.<br><br>As far as I know there is not to much more missing.. but some of these are a bit critical<br><ul><li>{_( translation markers )_} , and how to use the translation toolkit.</li><li>flexy:raw="somevar" for putting text within tags (as otherwise the syntax would break HTML editors.)</li><li>Using some of the modifiers from the Savant Plugin (hint {xxx():numberformat}</li></ul>I'm sure there is more, but that's all I could think of at present..<br><br><br>