USB 5v power is always on (even when the computer is turned off, on some computers and on some ports.) You will probably need to program an Arduino with some sort of switch, and control it via Serial library from USB plugged in to the computer.
In other words, a combination of this switch tutorial and this tutorial on communicating via Serial libary to Arduino plugged in via USB.