You just can't be sure on a Unix system, that a .txt file truly is a text file. Your best bet is to use "file". Maybe try using:
file -ib "$file"
Then you can use a list of MIME types to match against or parse the first part of the MIME where you get stuff like "text", "application", etc.