A color value has more than one dimension, so there is no intrinsic way to compare two colors. You have to determine for your use case the meaning of the colors and thereby how to best compare them.
Most likely you want to compare the hue, saturation and/or lightness properties of the colors as oppposed to the red/green/blue components. If you are having trouble figuring out how you want to compare them, take some pairs of sample colors and compare them mentally, then try to justify/explain to yourself why they are similar/different.
Once you know which properties/components of the colors you want to compare, then you need to figure out how to extract that information from a color.
Most likely you will just need to convert the color from the common RedGreenBlue representation to HueSaturationLightness, and then calculate something like
avghue = (color1.hue + color2.hue)/2
distance = abs(color1.hue-avghue)
This example would give you a simple scalar value indicating how far the gradient/hue of the colors are from each other.