Linux

I had to laugh the otherday reading one of the Microsurf's blogs about how they suddenly had to deal with customer comments, and suprisingly enough, not many where positive. I think the guy was from the .Net team, and I considered he probably get's it easy, at least he's not on the IE team..

I gave up microcrap years ago, and only now use it for interoperability testing, where most of the time it fails, (XP included..) but I was supprised this week, when working with a new team here, when someone asked me why he should use linux rather than microsoft.

Youth and alot of Microsoft giveaways had ended up with the impression that this stuff fulfilled every need and desire. I have to admit I was a little shocked, I rarely find people who actually like what microsoft attempts to deliver. Most are resigned to the fact that it's a pile of crap but they are either to lazy to learn something new or, blindly believe that the next version will be better.

I started explaining the concept, that as a developer you have the ability to actually fix things with linux. Where he seemed to think that microsoft actually fixed things, especially as his old company had some developer/partner package.. But digging deeper it was pretty clear that solving problems just meant news about workarounds comes faster.. Actually fixing stuff still required something close to a miracle. (CSS fixes for IE came to mind)

But what really would have been a killer example was reading about Webdav support on XP, basically it's completely broken (there are even Knowledgebase articles saying so from 2003). Yet again re-inforcing my view of Microsoft as a company that may fix bugs in each release, but breaks just as many new things with each effort. So however many blogs you see coming out of microsurfs, the knowledge that they will completely ignore their customers (which thankfully I managed to remain from being), means that they still have a long while before anyone actually expects their software to work.

View Extended Entry

Posted by in Linux | Add / View Comments()

After compaining for so long about spam getting through via my @php.net mail address, the php.net mail admins have at least partially answered my prayers, this has started appearing in the headers.

X-Host-Fingerprint: 00.00.00.00 the.host.name Windows 2000 SP2+, XP SP1 (seldom 98 4.10.2222)

After a bit of investigation, it appears the very nice tool p0f is being used transparently to recognize the sending machine type. I spent a bit of time here playing with this, although not perfect (it cant recognize hosts behind a NAT box), It's pretty darn good..

It does lead me onto the theory though

The real solution to spam is just to block all Windows boxes from public email servers....

Seriously, I cant imagine anyone dumb enough to have a public facing Windows box, sending email. And what the hell, they should have to pay a relaying fee if they need messages delivered (if they are dumb enough to pay for windows, they are sure dumb enough to pay for a relaying service..)

This solution is alot simpler and more reliable than SPF.. hell, lets fire up an RFC, and get this standard layed down for good.. (oops I forgot to patent it..)

View Extended Entry

Posted by in Linux | Add / View Comments()

My father just retired, and rekindled his interest in doing Video conferencing (He's in the UK, I'm in HK). So after a bit of checking for linux compatible usb webcams, I settled on a Creative webcam Live!

Set up was not to bad, it uses a spca50x chip, which builds and compiles quite easily. The only oddity was that the spcatools that can be downloaded from the driver site didnt appear to work. However, gnomemeeting and streamer displayed the output perfectly.

I ran into huge trouble though with networking and video conferencing, I initially set up gnomemeeting behind my firewall (eg. the webserver with ipmasq on it.), but after a few experiments it became obvious that the whole relaying of h323 video connections was increadibly complex, and really not worth the hastle. In the end I just moved the camara to my firewall/webserver/comms server (eg. it runs irc/gaim).

I did at one point get video conferencing working within the house to a win98 box with netmeeting, but getting the video to actually display in netmeeting was very hit and miss.

In doing all this, I wandered off topic and started looking at alternatives, the two big ones being SIPs and Skype. It's pretty difficult to get an overview of SIP, but by the looks of things it looks like it's not much better than the current gatekeeper type of concept of gnomemeeting/netmeeting. The only thing going for it is that all the hardware manufacturers have been building routers and phones specifically for this protocol. (I think there's even a free SIPS router for linux). But in reality it looked like it had the same architecture as gatekeeper, which was the core to the fundimental problem with this point to point via a special gateway concept. The gateway was always an extra consideration that was a hastle to set up.

Skype, is something that I've been hearing more and more about, and from reading their technical FAQ, they seem to have spotted the above problems and thought of a half decent solution, basically by turning Skype clients that have direct internet access into relays (or quazi gatekeepers), along with using P2P type technology to do the connections (eg. doing originating connections from inside firewalls, to outside, rather than trying to cleverly route external connections to an inside machine).

The only shame about the Skype stuff is the propriatory protocol and client, which makes it highly seceptable to an open source implementation decimating their market, and the fact that smart people cant hack the client to add nice features like video conferencing.

But I'm still waiting to see how my gnomemeeting works with netmeeting, lets hope that my parents broadband is not firewalled to death by their ISP..

View Extended Entry

Posted by in Linux | Add / View Comments()

There were some interesting comments from my last post on Hyping by blog. Jackson Miller pointed out very bluntly, isnt that what blogs are for, and while he is partly correct, if a blogs are purely a advertising / news feed for a project, then they are not really blogs, but more project sites. What makes blogs interesting normally, is not that they publish release announcements, but that you get some insight into other things a developer may be doing, often unrelated to a project they are well known for. So I guess the conclusion was, if some one starts hinting you are Hyping and not blogging, perhaps you are..

I saw a few posts on Artima more recently, that included detailed analysis on RoR indicating alot of what I considered RoR to be. Ruby, while having interesting features, doesnt appear to have what could be called an elegant language construct, or a particularly huge following, which for me are part of the consideration on whether to invest time into experimenting with it, (which C#/ASP.net did justify, but produced similar returns). RoR, turns out to be little more that a clever combination of tools to write skeletons and some reasonable libraries, which while useful, really doesnt justify the excitement, but I gues it's an improvement on the ASP.NET, where the solution is not forced in your face so much, and alternatives are frowned apon (try googling for the equivilant of mysql_escape_string in .NET, and you will see what I mean)

Imap Continued.

Bincimap unfortunatly was unable to deliver the promise that it looked like it could. This week I got a call saying that Outlook express (or more like 'lookout express') users where having problems. It's pretty common knowledge that although outlook says it supports IMAP, it's implementation is buggy to the point of unusable. I know this from googling mailing lists and seeing the amount of kludges and workarounds that appear to have gone into IMAP servers, just to support this pile of crap.

Normally when you get problems with outlook and imap, you brush it off as intermittenant problems with a crap piece of software, and suggest they upgrade to a real email client (thunderbird, or evolution come to mind). But sometimes, company owners or important sales staff are not really that open to changing the ill gotten ways, so Outlook support has to be suffered.. (at least at an hourly rate!!!). So this time (after a few goes at modifying the settings on outlook) I decided to examine what was going on a little closer, including doing protocol dumps.

The key problems where that deleted messages (and ones that had been moved to another folder) would reappear as unread, new when you pressed the send/recieve.

To my amazement, outlook spawns new connections and does alot of imap operations concurrently, without a care in the world on how complex this may be to the server (eg. 3 connections all doing operations on the inbox folder). And menu operations often open new connections, and drag and drop operations dont. - It's all a bit like a beginners VB program, completely undesigned, and thrown together a few minutes after hello world worked.

I've given the protocol dumps to the bincimap developers, but over the weekend, I also discovered that my wife's palmphone, was unable to read email. I can pospone problems with companies a few days, but I better fix my wife's issues faster!. So after another marathon protocol dumping sessions, it became clear that bincimap was sending a little too much information for snappermail to understand. So I quickly switched over to dovecot imap.

I feel a bit disapointed here, us fickle users, jump from one ship to another so easily. I did get the chance to look at bincimap's souce, and it was very clean C++, and pretty well designed. And having given the author (Andreas Aardal Hanssen ,who was very responsive) a reasonably high quality set of bug reports, I didnt feel to bad deserting to another application.

Dovecot on debian proved amazingly simple to migrate to, the only change required after apt-getting was modifying /etc/dovecot/dovecot.conf and changing the line protocol = imaps

Other than that, restarting evolution, which should provide another good blog review, and I have now finally tested, used and configured all 5 major open source imap servers..

View Extended Entry

Posted by in Linux | Add / View Comments()

mbox must be about the worst designed format ever, this week in a small office I consult for, a few of the staff started complaining they couldn't open some of their mailboxes. It didnt take long to realize that the server was overloaded. 10 people, each had an inbox averaging 500Mb, and outlook checking email every 3 minutes, a few of them also doing a full scan of their imap folders checking for new mail, which range from 2->4Gb.

The poor server was suffering badly, so yet again, I investigated IMAP servers. I've tried cyrus, courier, uw-imap and while each has advantages cyrus and courier have tended to be a little annoying to set up, messing around with auth and protocol issues. uw-imap is the root cause of the above issues (although mbx format does help alot). I was interested to find bincimap (binc is not courier). The overview points out that it's a pure maildir backend mail server (which usually perform pretty well with a cache, and dont have interface issues with folders containing folders not being usable). It also appeared to be pretty simple to set up, although no examples for use with exim where obvious.

I tested the installation on my development box, and after a bit of hunting around and guesswork, I put together simple instructions for converting a exim4/uw-imap installation to exim4/bincimap. The only downside to the conversion was that I lost all my "important" flags from thunderbird. I was actually quite impressed that my own instructions where so amazingly simple.

mono/ASP.net and C#

My experiments with ASP.net and C# have been contining, highlights of this week where discovering that codebehind, without VS.NET is a real waste of time, the codebehind concept assumes that you want to have a compiled .dll. So if you are developing a web page with codebehind, your roundtrip testing becomes edit/build/(install)/test.. I may as well write it in C!!!, at least you can compile that on the fly (tcc a very nice small C compiler).

After playing around with various <% language="C#" src="..."> options, I eventualy came up with the kludge of doing quasi virtual includes

<!-- #include "lib.cs" -->

It's far from clean (it feels a bit like working with function libraries), but at least it works.

Next on my challenges was getting ASP.Net working with mysql, This is a challenge in it'self (I ended up copying the bytefx.dll into my web root's bin directory to get the import working). From what I gather, ByteFX who wrote one of the main mysql connection toolkits for .NET, (which looks like it is now owned by mysqlAB) is unfortunatly not very well documented. The example on the go-mono.org site works, but It did not take long to be reminded that if you want to work with C#, you have to think the Microsoft way. The one true solution, or the one true solution (both of which look good on the face of it, but are shit round the edges.)

Any good PHP programmer knows that sending raw data from a URL to the database is a security nightmare waiting to happen. So we have these wonderfull features like addslashes, and mysql_real_escape_string(). - normally hidden nicely in a DB abstaction layer. We also get to use bound parameters in some database backends that are designed that way. The .NET way is take parameters or give up.. - you cant escape strings, you must use parameters. (excuse my memory here - this example may need fixing)
eg.

mycmd.executeText = "select * from sometable where name=@name";

MysqlParameter param = new MysqlParameter("@name", MysqlType.VarChar);

param.Value = "some'test";

mycmd.Parameters.Add(param);

A couple of problems with the above code come to mind, (apart from it's suffers the .NET problem, adding as much noise as possible to a piece of code, trying hiding the purpose).

• MysqlParameter docs are difficult to find, and mostly in javadoc style format - which not exactly informative.
  
• @ is used by mysql for variables, so apparently the @ will get changed to ? later...
• Debugging what is actually going to the server is impossible! (as far as I could tell). I ended up turning debugging on , on the server, just to be sure what data was ending up at the server.

I can't say it's all bad, but it get very difficult to see the gem's between the rocks when you spend your time writing simple methods to solve common problems all the time.

Client ever asked you to turn a HTML table into a excel spreadsheet?

Given one hour to convert a complex piece of data retreival code to output to an excel file. The thought came to me, why not do it in javascript, based on the existing HTML. write a small piece of javascript that iterates through a HTML table, and posts a form with the data as a CSV to a 2 line PHP script.
This is the HTML

<form method="POST" action="quickexcel.php" onsubmit="return toExcel('data')">
<input type="hidden" id="exceldata" name="exceldata" value="">
<input type="submit" name="_submit" value="Download as Excel">
</form>
<table id="data"> ...... table with data ...... </table>

you can see the javascript here (it's pretty simple)
and the two line php file..
<?php

header('Content-type: application/vnd.ms-excel');
echo $_POST['exceldata'];



View Extended Entry

Posted by in Linux | Add / View Comments()

"So should we throw away that old portable", my wife frequently asks if I leave it lying around. An 8yr+ old toshiba PII/300 portable, that has windows 95, supposidly for her use.. but generally ignored due to the fact that it's too slow to fart.

It's a perfectly good machine except for the speed and the battery life is now about 2 seconds. So I occasionally embark on a hunt for a nice small linux distribution on a live cd, that would run a desktop via XDMCP.

Unfortunatly I've yet to succeed.. - Todays efforts included

dsl (Damn Small linux) ~50Mb - boots up the machine, finds all the hardware and network perfectly (including a wireless card). Has a reasonably usefull looking desktop for browsing the web. But unfortunatly uses tinyx compiled without XDMCP support.

PXES ~11Mb - a 'network ready' really thin client boots with a nice option to specify XDMCP, but totally fails to start the pcmcia cards and hence the network (kind of necessary for XDMCP!)

So unless someone suggests an answer (as I dont really want to get into the live cd building business), the box will go back to annoying my wife, being left around the house..

And on a slightly more successfull note, I am still realing from amazement at how well debian linux runs on a Sun Enterprise 2 machine a friend of mine had lying around his office - a more than 6year old machine, that apart from the crappy display card performs about the same speed as my 3year old development Intel box. (and has pretty much the same software installed thanks to debian)

View Extended Entry

Posted by in Linux | Add / View Comments()

Defer unknowns has proved very successful, with success rates of more than 1000 spams per day removed from my email box.

My original code parsed the exim log, and built black,white and greylists by selecting checkboxes ona web page. It was effective, but involved a little maintenaince. (daily review, and mostly blacklisting IP addresses.)

By looking at the general pattern of this, it became clear, that most spammers fire off a large number of hijacked PC's and just run through a big email list. If it fails (eg. defer), they just give up on that machine, and pass it along to the next (often changing the signature). - It's a known trick to do defer greylisting on this. Basically first time that ip contacts you, you respond, defer, try later.. next time, you let it through. (In my new scheme, I only black/grey/white list the ones that tried more than once - which should significantly reduce the amount of maintenance, and makes spotting good IP addresses alot easier.

The other beauty of the new solution is that it doesnt involve parsing logs anymore, it's almost a pure exim/mysql solutions, with my manual categorizing a considerably simpler web page.

The exim config I'm using is available in the extended entry.
or have a look at the simple spam manager interface

I guess if you want to run this on a bigger site, you might want to go to the mysql conference where you can find out reall answers form mysql developers, and experts. (and if you go to the php conference at the same place/time, you can see me talking about php5 and pear.)

View Extended Entry

Posted by in Linux | Add / View Comments()

That last entry noted the huge torpedo that Windows had fired at my Racing yacht (DBDO). I spent a good day or two exploring options, apart from the ones mentioned previously (using pdo as a backend). I looked at having a way using callbacks to implement the libgda stuff in clientside php on windows.

All in all the options where looking pretty dire, I was going to have to spend alot of time just to create a crippleware version for windows (well I guess if you use windows you shouldnt expect better). But It was not really what I regarded as a interesting or usefull use of my spare time.

On the sidelines, I had also looked at building libgda on windows, (or Win4Lin in my case), I had tried mingw, dev-cpp, but having to deal with a operating system that was never designed to make building things easy, it just became fustrating and annoying.

I dont know how I came across it, perhaps researching mingw, but I accidentally ended up on a page discussing cross-compiling mingw. It seemed to infer that you could build the windows .exe and .dll's on linux, and not have to go near the windows hell..

After some experimentation, (trying the debian package for it, and downloading a whole enviroment - google for cross compiling mingw), I got something resembling an enviroment that ran configure, and even actually made a few files.

After tying to build libgda for a while,I concluded that using configure/make on it was pretty pointless, (It adds a huge overhead, got various thing wrong most of the time). So I ended up hacking up a php build script for it which successfully compiles libsql, and libgda (although I havent linked or tested it yet)

Ah well, back to real work for a while, panic at least partially over...

View Extended Entry

Posted by in Linux | Add / View Comments()

Probably like Sterling and almost all my clients, spam is a huge problem. A couple of years ago I started using exim4 with eximscan-av (an excellent set of tools), and configured it so:

• no .pif/.exe/....
  
• clamav (removes all remainng virus emails, like zips)
  
• No HTML only emails (or messages without text/plain)
  
• No korean / chinese subjects.
• No Multipart messages to my @php.net address.
• Some attempt at blocking virus bounces..

This achieved pretty good results It originally caught about 300 out of 340 spams per day. (and lets through 200-400 good messages). However after a while it became clear that a few spammers had realized that some people did this, and they started sending plain text spam.., so the number getting through slowly increased.

About a year ago, I decided that IP blacklisting was really the only other solution. So I wrote an exim Log parser, along with adding the Subject to the message log line, this enabled me to quickly scan and blacklist IP's on a daily basis, on average I blacklisted about 10-20 a day. However with the advent of the virus spammers, this began to seem like a neverending task..

In an effort to fight back, I finally rewrote the blacklist parser code, and rejigged exim a little.

• any incomming email (IP) that is not 'greylisted' or 'whitelisted', is defered.
• once a day, A cron job parses the log, look at the defer messages, and compare the email From address to the IP
• If they are close (eg. same Class B) - it gets greylisted.
• If there is no match whatsoever, (fake from domain etc.), it gets instantly blacklisted.
• All virus IP's are auto blacklisted
• All HTML email senders are auto blacklisted
  
• once in a while, I check the parser web page, which lists all the emails recieved in the greylist, and white or blacklist them manually..

The solution is not perfect however, as it does depend heavily on me having created a good whitelist to start with, and for @php.net mail servers not to change to often (as they did recently, and got blacklisted accidentally by me, on the old manual system). But all these minor irritants, are not as bad as the alternative, having to wade through a ton of spam everyday..

The end result, About one spam per day (usually a nigerian scam one, via a freemail service) and I still get the occasional project enquiry (delayed by a day)..

Such a shame I miss out on all those fantastic offers for viagra...

View Extended Entry

Posted by in Linux | Add / View Comments()

Posted by in Linux | Add / View Comments()