To make it more clear(and to put it together) I had to do Two things mentioned above.
1- Create a file /etc/ld.so.conf.d/opencv.conf
and write to it the paths of folder where your opencv libraries are stored.(Answer by Cookyt)
2- Include the path of your opencv's .so
files in LD_LIBRARY_PATH ()
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/opencv/lib