I'm no expert in map-reading / navigation and so on but surely 'directions' are absolute and not relative or in reality, they are relative to N or S which themselves are fixed/absolute.
Example: Suppose an imaginary line drawn between you and your destination corresponds with 'absolute' SE (a bearing of 135 degrees relative to magnetic N). Now suppose your phone is pointing NW - if you draw an imaginary line from an imaginary object on the horizon to your destination, it will pass through your location and have an angle of 180 degrees. Now 180 degrees in the sense of a compass actually refers to S but the destination is not 'due S' of the imaginary object your phone is pointing at and, moreover, if you travelled to that imaginary point, your destination would still be SE of where you moved to.
In reality, the 180 degree line actually tells you the destination is 'behind you' relative to the way the phone (and presumably you) are pointing.
Having said that, however, if calculating the angle of a line from the imaginary point to your destination (passing through your location) in order to draw a pointer towards your destination is what you want...simply subtract the (absolute) bearing of the destination from the absolute bearing of the imaginary object and ignore a negation (if present). e.g., NW - SE is 315 - 135 = 180 so draw the pointer to point at the bottom of the screen indicating 'behind you'.
EDIT: I got the Maths slightly wrong...subtract the smaller of the bearings from the larger then subtract the result from 360 to get the angle in which to draw the pointer on the screen.