Best way to encode text data for XML

Question

I was looking for a generic method in  Net to encode a string for use in an Xml element or attribute  and was surprised when I didn t immediately find one   So  before I go too much further  could I just be missing the built-in function     Assuming for a moment that it really doesn t exist  I m putting together my own generic EncodeForXml string data  method  and I m thinking about the best way to do this     The data I m using that prompted this whole thing could contain bad characters like  amp    lt    quot   etc   It could also contains on occasion the properly escaped entities   amp amp    amp lt   and  amp quot   which means just using a CDATA section may not be the best idea   That seems kinda klunky anyay  I d much rather end up with a nice string value that can be used directly in the xml   I ve used a regular expression in the past to just catch bad ampersands  and I m thinking of using it to catch them in this case as well as the first step  and then doing a simple replace for other characters     So  could this be optimized further without making it too complex  and is there anything I m missing     Function EncodeForXml ByVal data As String  As String     Static badAmpersand As new Regex   amp     a-zA-Z  2 6     0-9  2 4           data   badAmpersand Replace data    amp amp         return data Replace   lt      amp lt    Replace         amp quot    Replace   gt     gt    End Function   Sorry for all you C  -only folks-- I don t really care which language I use  but I wanted to make the Regex static and you can t do that in C  without declaring it outside the method  so this will be VB Net  Finally  we re still on  Net 2 0 where I work  but if someone could take the final product and turn it into an extension method for the string class  that d be pretty cool too   Update The first few responses indicate that  Net does indeed have built-in ways of doing this   But now that I ve started  I kind of want to finish my EncodeForXml   method just for the fun of it  so I m still looking for ideas for improvement   Notably  a more complete list of characters that should be encoded as entities  perhaps stored in a list map   and something that gets better performance than doing a  Replace   on immutable strings in serial

User · Answer

Depending on how much you know about the input, you may have to take into account that not all Unicode characters are valid XML characters.

Both Server.HtmlEncode and System.Security.SecurityElement.Escape seem to ignore illegal XML characters, while System.XML.XmlWriter.WriteString throws an ArgumentException when it encounters illegal characters (unless you disable that check in which case it ignores them). An overview of library functions is available here.

Edit 2011/8/14: seeing that at least a few people have consulted this answer in the last couple years, I decided to completely rewrite the original code, which had numerous issues, including horribly mishandling UTF-16.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;

/// <summary>
/// Encodes data so that it can be safely embedded as text in XML documents.
/// </summary>
public class XmlTextEncoder : TextReader {
    public static string Encode(string s) {
        using (var stream = new StringReader(s))
        using (var encoder = new XmlTextEncoder(stream)) {
            return encoder.ReadToEnd();
        }
    }

    /// <param name="source">The data to be encoded in UTF-16 format.</param>
    /// <param name="filterIllegalChars">It is illegal to encode certain
    /// characters in XML. If true, silently omit these characters from the
    /// output; if false, throw an error when encountered.</param>
    public XmlTextEncoder(TextReader source, bool filterIllegalChars=true) {
        _source = source;
        _filterIllegalChars = filterIllegalChars;
    }

    readonly Queue<char> _buf = new Queue<char>();
    readonly bool _filterIllegalChars;
    readonly TextReader _source;

    public override int Peek() {
        PopulateBuffer();
        if (_buf.Count == 0) return -1;
        return _buf.Peek();
    }

    public override int Read() {
        PopulateBuffer();
        if (_buf.Count == 0) return -1;
        return _buf.Dequeue();
    }

    void PopulateBuffer() {
        const int endSentinel = -1;
        while (_buf.Count == 0 && _source.Peek() != endSentinel) {
            // Strings in .NET are assumed to be UTF-16 encoded [1].
            var c = (char) _source.Read();
            if (Entities.ContainsKey(c)) {
                // Encode all entities defined in the XML spec [2].
                foreach (var i in Entities[c]) _buf.Enqueue(i);
            } else if (!(0x0 <= c && c <= 0x8) &&
                       !new[] { 0xB, 0xC }.Contains(c) &&
                       !(0xE <= c && c <= 0x1F) &&
                       !(0x7F <= c && c <= 0x84) &&
                       !(0x86 <= c && c <= 0x9F) &&
                       !(0xD800 <= c && c <= 0xDFFF) &&
                       !new[] { 0xFFFE, 0xFFFF }.Contains(c)) {
                // Allow if the Unicode codepoint is legal in XML [3].
                _buf.Enqueue(c);
            } else if (char.IsHighSurrogate(c) &&
                       _source.Peek() != endSentinel &&
                       char.IsLowSurrogate((char) _source.Peek())) {
                // Allow well-formed surrogate pairs [1].
                _buf.Enqueue(c);
                _buf.Enqueue((char) _source.Read());
            } else if (!_filterIllegalChars) {
                // Note that we cannot encode illegal characters as entity
                // references due to the "Legal Character" constraint of
                // XML [4]. Nor are they allowed in CDATA sections [5].
                throw new ArgumentException(
                    String.Format("Illegal character: '{0:X}'", (int) c));
            }
        }
    }

    static readonly Dictionary<char,string> Entities =
        new Dictionary<char,string> {
            { '"', "&quot;" }, { '&', "&amp;"}, { '\'', "&apos;" },
            { '<', "&lt;" }, { '>', "&gt;" },
        };

    // References:
    // [1] http://en.wikipedia.org/wiki/UTF-16/UCS-2
    // [2] http://www.w3.org/TR/xml11/#sec-predefined-ent
    // [3] http://www.w3.org/TR/xml11/#charsets
    // [4] http://www.w3.org/TR/xml11/#sec-references
    // [5] http://www.w3.org/TR/xml11/#sec-cdata-sect
}

Unit tests and full code can be found here.

User · Answer

You can use the built-in class XAttribute  which handles the encoding automatically   using System Xml Linq   XDocument doc   new XDocument     List lt XAttribute gt  attributes   new List lt XAttribute gt     attributes Add new XAttribute  key1    val1 amp val11     attributes Add new XAttribute  key2    val2      XElement elem   new XElement  test   attributes ToArray      doc Add elem    string xmlStr   doc ToString

User · Answer

If you re serious about handling all of the invalid characters  not just the few  html  ones   and you have access to System Xml  here s the simplest way to do proper Xml encoding of value data   string theTextToEscape    Something  x1d else  x1D  lt script gt alert  123    lt  script gt    var x   new XmlDocument    x LoadXml   lt r  gt        simple  empty root element x DocumentElement InnerText   theTextToEscape     put in raw string string escapedText   x DocumentElement InnerXml     Returns   Something  amp  x1D  else  amp  x1D   amp lt script amp gt alert  123    amp lt  script amp gt      Repeat the last 2 lines to escape additional strings    It s important to know that XmlConvert EncodeName   is not appropriate  because that s for entity tag names  not values  Using that would be like Url-encoding when you needed to Html-encode

User · Answer

This might be the case where you could benefit from using the WriteCData method   public override void WriteCData string text      Member of System Xml XmlTextWriter  Summary  Writes out a  lt   CDATA       gt  block containing the specified text   Parameters  text  Text to place inside the CDATA block    A simple example would look like the following   writer WriteStartElement  name    writer WriteCData   lt unsafe characters gt     writer WriteFullEndElement      The result looks like    lt name gt  lt   CDATA  lt unsafe characters gt    gt  lt  name gt    When reading the node values the XMLReader automatically strips out the CData part of the innertext so you don t have to worry about it   The only catch is that you have to store the data as an innerText value to an XML node   In other words  you can t insert CData content into an attribute value

User · Answer

System XML handles the encoding for you  so you don t need a method like this

User · Answer

Microsoft s AntiXss library AntiXssEncoder Class in System Web dll has methods for this   AntiXss XmlEncode string s  AntiXss XmlAttributeEncode string s    it has HTML as well   AntiXss HtmlEncode string s  AntiXss HtmlAttributeEncode string s

User · Answer

Here is a single line solution using the XElements  I use it in a very small tool  I don t need it a second time so I keep it this way   Its dirdy doug   StrVal     lt x a  lt    StrVal   gt  gt END lt  x gt   ToString   Replace   lt x a          Replace   gt END lt  x gt          Oh and it only works in VB not in C

User · Answer

Brilliant  That s all I can say   Here is a VB variant of the updated code  not in a class  just a function  that will clean up and also sanitize the xml  Function cXML ByVal  buf As String  As String     Dim textOut As New StringBuilder     Dim c As Char     If  buf Trim Is Nothing OrElse  buf   String Empty Then Return String Empty     For i As Integer   0 To  buf Length - 1         c    buf i          If Entities ContainsKey c  Then             textOut Append Entities Item c           ElseIf  AscW c     amp H9 OrElse AscW c     amp HA OrElse AscW c     amp HD  OrElse   AscW c   gt    amp H20  AndAlso  AscW c   lt    amp HD7FF                 OrElse   AscW c   gt    amp HE000  AndAlso  AscW c   lt    amp HFFFD   OrElse   AscW c   gt    amp H10000  AndAlso  AscW c   lt    amp H10FFFF   Then             textOut Append c          End If     Next     Return textOut ToString  End Function  Shared ReadOnly Entities As New Dictionary Of Char  String    From       c    amp quot        amp  c    amp amp         c    amp apos        lt  c    amp lt        gt  c    amp gt

User · Answer

If this is an ASP NET app why not use Server HtmlEncode

User · Answer

In the past I have used HttpUtility HtmlEncode to encode text for xml  It performs the same task  really  I havent ran into any issues with it yet  but that s not to say I won t in the future  As the name implies  it was made for HTML  not XML   You ve probably already read it  but here is an article on xml encoding and decoding   EDIT  Of course  if you use an xmlwriter or one of the new XElement classes  this encoding is done for you  In fact  you could just take the text  place it in a new XElement instance  then return the string   tostring  version of the element  I ve heard that SecurityElement Escape will perform the same task as your utility method as well  but havent read much about it or used it   EDIT2  Disregard my comment about XElement  since you re still on 2 0

User · Answer

In  net 3 5   new XText  I  lt want gt  to  amp  encode this for XML   ToString      Gives you   I  amp lt want amp gt  to  amp amp  encode this for XML   Turns out that this method doesn t encode some things that it should  like quotes    SecurityElement Escape  workmad3 s answer  seems to do a better job with this and it s included in earlier versions of  net   If you don t mind 3rd party code and want to ensure no illegal characters make it into your XML  I would recommend Michael Kropat s answer

User · Answer

This might be the case where you could benefit from using the WriteCData method   public override void WriteCData string text      Member of System Xml XmlTextWriter  Summary  Writes out a  lt   CDATA       gt  block containing the specified text   Parameters  text  Text to place inside the CDATA block    A simple example would look like the following   writer WriteStartElement  name    writer WriteCData   lt unsafe characters gt     writer WriteFullEndElement      The result looks like    lt name gt  lt   CDATA  lt unsafe characters gt    gt  lt  name gt    When reading the node values the XMLReader automatically strips out the CData part of the innertext so you don t have to worry about it   The only catch is that you have to store the data as an innerText value to an XML node   In other words  you can t insert CData content into an attribute value

User · Answer

SecurityElement Escape  documented here

User · Answer

XmlTextWriter WriteString   does the escaping

[.net] Best way to encode text data for XML

Examples related to .net

Examples related to xml

Examples related to encoding

Examples related to .net-2.0