I did manage to figure it out:
$order_meta = get_post_meta($order_id);
$email = $order_meta["_shipping_email"][0] ?: $order_meta["_billing_email"][0];
I do know know for sure if the shipping email is part of the metadata, but if so I would rather have it than the billing email - at least for my purposes.