Configuration files are application-scoped and not assembly-scoped. So you'll need to put your library's configuration sections in every application's configuration file that is using your library.
That said, it is not a good practice to get configuration from the application's configuration file, specially the appSettings
section, in a class library. If your library needs parameters, they should probably be passed as method arguments in constructors, factory methods, etc. by whoever is calling your library. This prevents calling applications from accidentally reusing configuration entries that were expected by the class library.
That said, XML configuration files are extremely handy, so the best compromise that I've found is using custom configuration sections. You get to put your library's configuration in an XML file that is automatically read and parsed by the framework and you avoid potential accidents.
You can learn more about custom configuration sections on MSDN and also Phil Haack has a nice article on them.