Don't know if this will be everybody's answer, but after some digging, here's what we came up with.
The error is obviously caused by the fact that the listener was not accepting connections, but why would we get that error when other tests could connect fine (we could also connect no problem through sqlplus)? The key to the issue wasn't that we couldn't connect, but that it was intermittent
After some investigation, we found that there was some static data created during the class setup that would keep open connections for the life of the test class, creating new ones as it went. Now, even though all of the resources were properly released when this class went out of scope (via a finally{} block, of course), there were some cases during the run when this class would swallow up all available connections (okay, bad practice alert - this was unit test code that connected directly rather than using a pool, so the same problem could not happen in production).
The fix was to not make that class static and run in the class setup, but instead use it in the per method setUp and tearDown methods.
So if you get this error in your own apps, slap a profiler on that bad boy and see if you might have a connection leak. Hope that helps.