At first I thought this was a coercion bug where null was getting coerced to "null" and a test of "null" == null was passing. It's not. I was close, but so very, very wrong. Sorry about that!
I've since done lots of fiddling on wonderfl.net and tracing through the code in mx.rpc.xml.*. At line 1795 of XMLEncoder (in the 3.5 source), in setValue, all of the XMLEncoding boils down to
currentChild.appendChild(xmlSpecialCharsFilter(Object(value)));
which is essentially the same as:
currentChild.appendChild("null");
This code, according to my original fiddle, returns an empty XML element. But why?
According to commenter Justin Mclean on bug report FLEX-33664, the following is the culprit (see last two tests in my fiddle which verify this):
var thisIsNotNull:XML = <root>null</root>;
if(thisIsNotNull == null){
// always branches here, as (thisIsNotNull == null) strangely returns true
// despite the fact that thisIsNotNull is a valid instance of type XML
}
When currentChild.appendChild is passed the string "null", it first converts it to a root XML element with text null, and then tests that element against the null literal. This is a weak equality test, so either the XML containing null is coerced to the null type, or the null type is coerced to a root xml element containing the string "null", and the test passes where it arguably should fail. One fix might be to always use strict equality tests when checking XML (or anything, really) for "nullness."
CDATA values are the most appropriate way to mutate an entire text value that would otherwise cause encoding/decoding problems. Hex encoding, for instance, is meant for individual characters. CDATA values are preferred when you're escaping the entire text of an element. The biggest reason for this is that it maintains human readability.