actually I walked the same path a couple of month ago. there is no perfect answer to the question (that was asked here a couple of time) but there is one more sophisticated then the sqrt(r-r) etc answer and more easy to implent directly with RGB without moving to all kind of alternate color spaces. I found this formula here which is a low cost approximation of the quite complicated real formula (by the CIE which is the W3C of color, since this is a not finished quest, you can find older and simpler color difference equations there). good luck
Edit: For posterity, here's the relevant C code:
typedef struct {
unsigned char r, g, b;
} RGB;
double ColourDistance(RGB e1, RGB e2)
{
long rmean = ( (long)e1.r + (long)e2.r ) / 2;
long r = (long)e1.r - (long)e2.r;
long g = (long)e1.g - (long)e2.g;
long b = (long)e1.b - (long)e2.b;
return sqrt((((512+rmean)*r*r)>>8) + 4*g*g + (((767-rmean)*b*b)>>8));
}