Use empty (it checks both nullness and emptiness) and group the nested ternary expression by parentheses (EL is in certain implementations/versions namely somewhat problematic with nested ternary expressions). Thus, so:
styleClass="#{empty obj.validationErrorMap ? ' ' :
(obj.validationErrorMap.contains('key') ? 'highlight_field' : 'highlight_row')}"
If still in vain (I would then check JBoss EL configs), use the "normal" EL approach:
styleClass="#{empty obj.validationErrorMap ? ' ' :
(obj.validationErrorMap['key'] ne null ? 'highlight_field' : 'highlight_row')}"
Update: as per the comments, the Map turns out to actually be a List (please work on your naming conventions). To check if a List contains an item the "normal" EL way, use JSTL fn:contains (although not explicitly documented, it works for List as well).
styleClass="#{empty obj.validationErrorMap ? ' ' :
(fn:contains(obj.validationErrorMap, 'key') ? 'highlight_field' : 'highlight_row')}"