Our version of Oracle is running on Red Hat Enterprise Linux. We experimented with several different types of group permissions to no avail. The /defaultdir directory had a group that was a secondary group for the oracle user. When we updated the /defaultdir directory to have a group of "oinstall" (oracle's primary group), I was able to select from the external tables underneath that directory with no problem.
So, for others that come along and might have this issue, make the directory have oracle's primary group as the group and it might resolve it for you as it did us. We were able to set the permissions to 770 on the directory and files and selecting on the external tables works fine now.