With a character class this big it is easier to say what you want to keep. A caret in the first position of a character class inverts its sense, so you can write
$varTemp =~ s/[^"%'+\-0-9<=>a-z_{|}]+//gi
or, using the more efficient tr
$varTemp =~ tr/"%'+\-0-9<=>A-Z_a-z{|}//cd