If you are on a 64bit build of ubuntu or debian (see e.g. 'cat /proc/version') you should simply use the 64bit cross compilers, if you cloned
git clone https://github.com/raspberrypi/tools
then the 64bit tools are in
tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64
use that directory for the gcc-toolchain. A useful tutorial for compiling that I followed is available here Building and compiling Raspberry PI Kernel (use the -x64 path from above as ${CCPREFIX})