[synchronization] When should one use a spinlock instead of mutex?

The rule for using spinlocks is simple: use a spinlock if and only if the real time the lock is held is bounded and sufficiently small.

Note that usually user implemented spinlocks DO NOT satisfy this requirement because they do not disable interrupts. Unless pre-emptions are disabled, a pre-emption whilst a spinlock is held violates the bounded time requirement.

Sufficiently small is a judgement call and depends on the context.

Exception: some kernel programming must use a spinlock even when the time is not bounded. In particular if a CPU has no work to do, it has no choice but to spin until some more work turns up.

Special danger: in low level programming take great care when multiple interrupt priorities exist (usually there is at least one non-maskable interrupt). In this higher priority pre-emptions can run even if interrupts at the thread priority are disabled (such as priority hardware services, often related to the virtual memory management). Provided a strict priority separation is maintained, the condition for bounded real time must be relaxed and replaced with bounded system time at that priority level. Note in this case not only can the lock holder be pre-empted but the spinner can also be interrupted; this is generally not a problem because there's nothing you can do about it.