I figured it out. Read the MSDN documentation and it says to use .Load instead of LoadXml when reading from strings. Found out this works 100% of time. Oddly enough using StringReader causes problems. I think the main reason is that this is a Unicode encoded string and that could cause problems because StringReader is UTF-8 only.
MemoryStream stream = new MemoryStream();
byte[] data = body.PayloadEncoding.GetBytes(body.Payload);
stream.Write(data, 0, data.Length);
stream.Seek(0, SeekOrigin.Begin);
XmlTextReader reader = new XmlTextReader(stream);
// MSDN reccomends we use Load instead of LoadXml when using in memory XML payloads
bodyDoc.Load(reader);