You had better use find_library command instead of link_directories. Concretely speaking there are two ways:
designate the path within the command
find_library(NAMES gtest PATHS path1 path2 ... pathN)
set the variable CMAKE_LIBRARY_PATH
set(CMAKE_LIBRARY_PATH path1 path2)
find_library(NAMES gtest)
the reason is as flowings:
Note This command is rarely necessary and should be avoided where there are other choices. Prefer to pass full absolute paths to libraries where possible, since this ensures the correct library will always be linked. The find_library() command provides the full path, which can generally be used directly in calls to target_link_libraries(). Situations where a library search path may be needed include: Project generators like Xcode where the user can switch target architecture at build time, but a full path to a library cannot be used because it only provides one architecture (i.e. it is not a universal binary).
Libraries may themselves have other private library dependencies that expect to be found via RPATH mechanisms, but some linkers are not able to fully decode those paths (e.g. due to the presence of things like $ORIGIN).
If a library search path must be provided, prefer to localize the effect where possible by using the target_link_directories() command rather than link_directories(). The target-specific command can also control how the search directories propagate to other dependent targets.