We have only single option left after reading all the fake referrer problems: i.e. The page we desire to track as referrer should be kept in session, and as ajax called then checking in session if it has referrer page value and doing the action other wise no action.
While on the other hand as he request any different page then make the referrer session value to null.
Remember that session variable is set on desire page request only.