I like the error as return-value way. If you're designing the api and you want to make use of your library as painless as possible think about these additions:
store all possible error-states in one typedef'ed enum and use it in your lib. Don't just return ints or even worse, mix ints or different enumerations with return-codes.
provide a function that converts errors into something human readable. Can be simple. Just error-enum in, const char* out.
I know this idea makes multithreaded use a bit difficult, but it would be nice if application programmer can set an global error-callback. That way they will be able to put a breakpoint into the callback during bug-hunt sessions.
Hope it helps.