If you download Java Development Kit(JDK) then there is a difference as it contains native libraries which differ for different architectures:
In addition you can use 32-bit JDK(x86) on 64-bit OS. But you can not use 64-bit JDK on 32-bit OS.
At the same time you can run compiled Java classes on any JVM. It does not matter whether it 32 or 64-bit.