The x86 instruction set includes an unsigned multiply instruction that stores the result to two registers. To use that instruction from C, one can write the following code in a 64-bit program (GCC):
unsigned long checked_imul(unsigned long a, unsigned long b) {
unsigned __int128 res = (unsigned __int128)a * b;
if ((unsigned long)(res >> 64))
printf("overflow in integer multiply");
return (unsigned long)res;
}
For a 32-bit program, one needs to make the result 64 bit and parameters 32 bit.
An alternative is to use compiler-dependent intrinsic to check the flag register. GCC documentation for overflow intrinsic can be found from 6.56 Built-in Functions to Perform Arithmetic with Overflow Checking.