Starting from a byte array you can use the binary package to do the conversions.
For example if you want to read ints :
buf := bytes.NewBuffer(b) // b is []byte
myfirstint, err := binary.ReadVarint(buf)
anotherint, err := binary.ReadVarint(buf)
The same package allows the reading of unsigned int or floats, with the desired byte orders, using the general Read function.