When indicating HTTP Basic Authentication we return something like:
WWW-Authenticate: Basic realm="myRealm"
Whereas Basic is the scheme and the remainder is very much dependent on that scheme. In this case realm just provides the browser a literal that can be displayed to the user when prompting for the user id and password.
You're obviously not using Basic however since there is no point having session expiry when Basic Auth is used. I assume you're using some form of Forms based authentication.
From recollection, Windows Challenge Response uses a different scheme and different arguments.
The trick is that it's up to the browser to determine what schemes it supports and how it responds to them.
My gut feel if you are using forms based authentication is to stay with the 200 + relogin page but add a custom header that the browser will ignore but your AJAX can identify.
For a really good User + AJAX experience, get the script to hang on to the AJAX request that found the session expired, fire off a relogin request via a popup, and on success, resubmit the original AJAX request and carry on as normal.
Avoid the cheat that just gets the script to hit the site every 5 mins to keep the session alive cause that just defeats the point of session expiry.
The other alternative is burn the AJAX request but that's a poor user experience.