It has to do with how the public members of the base class are exposed from the derived class.
As litb points out, public inheritance is traditional inheritance that you'll see in most programming languages. That is it models an "IS-A" relationship. Private inheritance, something AFAIK peculiar to C++, is an "IMPLEMENTED IN TERMS OF" relationship. That is you want to use the public interface in the derived class, but don't want the user of the derived class to have access to that interface. Many argue that in this case you should aggregate the base class, that is instead of having the base class as a private base, make in a member of derived in order to reuse base class's functionality.