The explanation from Scott Meyers in Effective C++ might help understand when to use them:
Public inheritance should model "is-a relationship," whereas private inheritance should be used for "is-implemented-in-terms-of" - so you don't have to adhere to the interface of the superclass, you're just reusing the implementation.