I'm processing some Java source code using Java. I'm extracting the string literals and feeding them to a function taking a String. The problem is that I need to pass the unescaped version of the String to the function (i.e. this means converting \n to a newline, and \\ to a single \, etc).
Is there a function inside the Java API that does this? If not, can I obtain such functionality from some library? Obviously the Java compiler has to do this conversion.
In case anyone wants to know, I'm trying to un-obfuscate string literals in decompiled obfuscated Java files.
org.apache.commons.lang3.StringEscapeUtils from commons-lang3 is marked deprecated now. You can use org.apache.commons.text.StringEscapeUtils#unescapeJava(String) instead. It requires an additional Maven dependency:
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-text</artifactId>
<version>1.4</version>
</dependency>
and seems to handle some more special cases, it e.g. unescapes:
\\b, \\n, \\t, \\f, \\rSee this from http://commons.apache.org/lang/:
StringEscapeUtils.unescapeJava(String str)
You can use String unescapeJava(String) method of StringEscapeUtils from Apache Commons Lang.
Here's an example snippet:
String in = "a\\tb\\n\\\"c\\\"";
System.out.println(in);
// a\tb\n\"c\"
String out = StringEscapeUtils.unescapeJava(in);
System.out.println(out);
// a b
// "c"
The utility class has methods to escapes and unescape strings for Java, Java Script, HTML, XML, and SQL. It also has overloads that writes directly to a java.io.Writer.
It looks like StringEscapeUtils handles Unicode escapes with one u, but not octal escapes, or Unicode escapes with extraneous us.
/* Unicode escape test #1: PASS */
System.out.println(
"\u0030"
); // 0
System.out.println(
StringEscapeUtils.unescapeJava("\\u0030")
); // 0
System.out.println(
"\u0030".equals(StringEscapeUtils.unescapeJava("\\u0030"))
); // true
/* Octal escape test: FAIL */
System.out.println(
"\45"
); // %
System.out.println(
StringEscapeUtils.unescapeJava("\\45")
); // 45
System.out.println(
"\45".equals(StringEscapeUtils.unescapeJava("\\45"))
); // false
/* Unicode escape test #2: FAIL */
System.out.println(
"\uu0030"
); // 0
System.out.println(
StringEscapeUtils.unescapeJava("\\uu0030")
); // throws NestableRuntimeException:
// Unable to parse unicode value: u003
A quote from the JLS:
Octal escapes are provided for compatibility with C, but can express only Unicode values
\u0000through\u00FF, so Unicode escapes are usually preferred.
If your string can contain octal escapes, you may want to convert them to Unicode escapes first, or use another approach.
The extraneous u is also documented as follows:
The Java programming language specifies a standard way of transforming a program written in Unicode into ASCII that changes a program into a form that can be processed by ASCII-based tools. The transformation involves converting any Unicode escapes in the source text of the program to ASCII by adding an extra
u-for example,\uxxxxbecomes\uuxxxx-while simultaneously converting non-ASCII characters in the source text to Unicode escapes containing a single u each.This transformed version is equally acceptable to a compiler for the Java programming language and represents the exact same program. The exact Unicode source can later be restored from this ASCII form by converting each escape sequence where multiple
u's are present to a sequence of Unicode characters with one feweru, while simultaneously converting each escape sequence with a singleuto the corresponding single Unicode character.
If your string can contain Unicode escapes with extraneous u, then you may also need to preprocess this before using StringEscapeUtils.
Alternatively you can try to write your own Java string literal unescaper from scratch, making sure to follow the exact JLS specifications.
Came across a similar problem, wasn't also satisfied with the presented solutions and implemented this one myself.
Also available as a Gist on Github:
/**
* Unescapes a string that contains standard Java escape sequences.
* <ul>
* <li><strong>\b \f \n \r \t \" \'</strong> :
* BS, FF, NL, CR, TAB, double and single quote.</li>
* <li><strong>\X \XX \XXX</strong> : Octal character
* specification (0 - 377, 0x00 - 0xFF).</li>
* <li><strong>\uXXXX</strong> : Hexadecimal based Unicode character.</li>
* </ul>
*
* @param st
* A string optionally containing standard java escape sequences.
* @return The translated string.
*/
public String unescapeJavaString(String st) {
StringBuilder sb = new StringBuilder(st.length());
for (int i = 0; i < st.length(); i++) {
char ch = st.charAt(i);
if (ch == '\\') {
char nextChar = (i == st.length() - 1) ? '\\' : st
.charAt(i + 1);
// Octal escape?
if (nextChar >= '0' && nextChar <= '7') {
String code = "" + nextChar;
i++;
if ((i < st.length() - 1) && st.charAt(i + 1) >= '0'
&& st.charAt(i + 1) <= '7') {
code += st.charAt(i + 1);
i++;
if ((i < st.length() - 1) && st.charAt(i + 1) >= '0'
&& st.charAt(i + 1) <= '7') {
code += st.charAt(i + 1);
i++;
}
}
sb.append((char) Integer.parseInt(code, 8));
continue;
}
switch (nextChar) {
case '\\':
ch = '\\';
break;
case 'b':
ch = '\b';
break;
case 'f':
ch = '\f';
break;
case 'n':
ch = '\n';
break;
case 'r':
ch = '\r';
break;
case 't':
ch = '\t';
break;
case '\"':
ch = '\"';
break;
case '\'':
ch = '\'';
break;
// Hex Unicode: u????
case 'u':
if (i >= st.length() - 5) {
ch = 'u';
break;
}
int code = Integer.parseInt(
"" + st.charAt(i + 2) + st.charAt(i + 3)
+ st.charAt(i + 4) + st.charAt(i + 5), 16);
sb.append(Character.toChars(code));
i += 5;
continue;
}
i++;
}
sb.append(ch);
}
return sb.toString();
}
I'm a little late on this, but I thought I'd provide my solution since I needed the same functionality. I decided to use the Java Compiler API which makes it slower, but makes the results accurate. Basically I live create a class then return the results. Here is the method:
public static String[] unescapeJavaStrings(String... escaped) {
//class name
final String className = "Temp" + System.currentTimeMillis();
//build the source
final StringBuilder source = new StringBuilder(100 + escaped.length * 20).
append("public class ").append(className).append("{\n").
append("\tpublic static String[] getStrings() {\n").
append("\t\treturn new String[] {\n");
for (String string : escaped) {
source.append("\t\t\t\"");
//we escape non-escaped quotes here to be safe
// (but something like \\" will fail, oh well for now)
for (int i = 0; i < string.length(); i++) {
char chr = string.charAt(i);
if (chr == '"' && i > 0 && string.charAt(i - 1) != '\\') {
source.append('\\');
}
source.append(chr);
}
source.append("\",\n");
}
source.append("\t\t};\n\t}\n}\n");
//obtain compiler
final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
//local stream for output
final ByteArrayOutputStream out = new ByteArrayOutputStream();
//local stream for error
ByteArrayOutputStream err = new ByteArrayOutputStream();
//source file
JavaFileObject sourceFile = new SimpleJavaFileObject(
URI.create("string:///" + className + Kind.SOURCE.extension), Kind.SOURCE) {
@Override
public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException {
return source;
}
};
//target file
final JavaFileObject targetFile = new SimpleJavaFileObject(
URI.create("string:///" + className + Kind.CLASS.extension), Kind.CLASS) {
@Override
public OutputStream openOutputStream() throws IOException {
return out;
}
};
//file manager proxy, with most parts delegated to the standard one
JavaFileManager fileManagerProxy = (JavaFileManager) Proxy.newProxyInstance(
StringUtils.class.getClassLoader(), new Class[] { JavaFileManager.class },
new InvocationHandler() {
//standard file manager to delegate to
private final JavaFileManager standard =
compiler.getStandardFileManager(null, null, null);
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
if ("getJavaFileForOutput".equals(method.getName())) {
//return the target file when it's asking for output
return targetFile;
} else {
return method.invoke(standard, args);
}
}
});
//create the task
CompilationTask task = compiler.getTask(new OutputStreamWriter(err),
fileManagerProxy, null, null, null, Collections.singleton(sourceFile));
//call it
if (!task.call()) {
throw new RuntimeException("Compilation failed, output:\n" +
new String(err.toByteArray()));
}
//get the result
final byte[] bytes = out.toByteArray();
//load class
Class<?> clazz;
try {
//custom class loader for garbage collection
clazz = new ClassLoader() {
protected Class<?> findClass(String name) throws ClassNotFoundException {
if (name.equals(className)) {
return defineClass(className, bytes, 0, bytes.length);
} else {
return super.findClass(name);
}
}
}.loadClass(className);
} catch (ClassNotFoundException e) {
throw new RuntimeException(e);
}
//reflectively call method
try {
return (String[]) clazz.getDeclaredMethod("getStrings").invoke(null);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
It takes an array so you can unescape in batches. So the following simple test succeeds:
public static void main(String[] meh) {
if ("1\02\03\n".equals(unescapeJavaStrings("1\\02\\03\\n")[0])) {
System.out.println("Success");
} else {
System.out.println("Failure");
}
}
If you are reading unicode escaped chars from a file, then you will have a tough time doing that because the string will be read literally along with an escape for the back slash:
my_file.txt
Blah blah...
Column delimiter=;
Word delimiter=\u0020 #This is just unicode for whitespace
.. more stuff
Here, when you read line 3 from the file the string/line will have:
"Word delimiter=\u0020 #This is just unicode for whitespace"
and the char[] in the string will show:
{...., '=', '\\', 'u', '0', '0', '2', '0', ' ', '#', 't', 'h', ...}
Commons StringUnescape will not unescape this for you (I tried unescapeXml()). You'll have to do it manually as described here.
So, the sub-string "\u0020" should become 1 single char '\u0020'
But if you are using this "\u0020" to do String.split("... ..... ..", columnDelimiterReadFromFile) which is really using regex internally, it will work directly because the string read from file was escaped and is perfect to use in the regex pattern!! (Confused?)
For the record, if you use Scala, you can do:
StringContext.treatEscapes(escaped)
Java 13 added a method which does this: String#translateEscapes.
It was a preview feature in Java 13 and 14, but was promoted to a full feature in Java 15.
I know this question was old, but I wanted a solution that doesn't involve libraries outside those included JRE6 (i.e. Apache Commons is not acceptable), and I came up with a simple solution using the built-in java.io.StreamTokenizer:
import java.io.*;
// ...
String literal = "\"Has \\\"\\\\\\\t\\\" & isn\\\'t \\\r\\\n on 1 line.\"";
StreamTokenizer parser = new StreamTokenizer(new StringReader(literal));
String result;
try {
parser.nextToken();
if (parser.ttype == '"') {
result = parser.sval;
}
else {
result = "ERROR!";
}
}
catch (IOException e) {
result = e.toString();
}
System.out.println(result);
Output:
Has "\ " & isn't
on 1 line.
I came across the same problem, but I wasn't enamoured by any of the solutions I found here. So, I wrote one that iterates over the characters of the string using a matcher to find and replace the escape sequences. This solution assumes properly formatted input. That is, it happily skips over nonsensical escapes, and it decodes Unicode escapes for line feed and carriage return (which otherwise cannot appear in a character literal or a string literal, due to the definition of such literals and the order of translation phases for Java source). Apologies, the code is a bit packed for brevity.
import java.util.Arrays;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class Decoder {
// The encoded character of each character escape.
// This array functions as the keys of a sorted map, from encoded characters to decoded characters.
static final char[] ENCODED_ESCAPES = { '\"', '\'', '\\', 'b', 'f', 'n', 'r', 't' };
// The decoded character of each character escape.
// This array functions as the values of a sorted map, from encoded characters to decoded characters.
static final char[] DECODED_ESCAPES = { '\"', '\'', '\\', '\b', '\f', '\n', '\r', '\t' };
// A pattern that matches an escape.
// What follows the escape indicator is captured by group 1=character 2=octal 3=Unicode.
static final Pattern PATTERN = Pattern.compile("\\\\(?:(b|t|n|f|r|\\\"|\\\'|\\\\)|((?:[0-3]?[0-7])?[0-7])|u+(\\p{XDigit}{4}))");
public static CharSequence decodeString(CharSequence encodedString) {
Matcher matcher = PATTERN.matcher(encodedString);
StringBuffer decodedString = new StringBuffer();
// Find each escape of the encoded string in succession.
while (matcher.find()) {
char ch;
if (matcher.start(1) >= 0) {
// Decode a character escape.
ch = DECODED_ESCAPES[Arrays.binarySearch(ENCODED_ESCAPES, matcher.group(1).charAt(0))];
} else if (matcher.start(2) >= 0) {
// Decode an octal escape.
ch = (char)(Integer.parseInt(matcher.group(2), 8));
} else /* if (matcher.start(3) >= 0) */ {
// Decode a Unicode escape.
ch = (char)(Integer.parseInt(matcher.group(3), 16));
}
// Replace the escape with the decoded character.
matcher.appendReplacement(decodedString, Matcher.quoteReplacement(String.valueOf(ch)));
}
// Append the remainder of the encoded string to the decoded string.
// The remainder is the longest suffix of the encoded string such that the suffix contains no escapes.
matcher.appendTail(decodedString);
return decodedString;
}
public static void main(String... args) {
System.out.println(decodeString(args[0]));
}
}
I should note that Apache Commons Lang3 doesn't seem to suffer the weaknesses indicated in the accepted solution. That is, StringEscapeUtils seems to handle octal escapes and multiple u characters of Unicode escapes. That means unless you have some burning reason to avoid Apache Commons, you should probably use it rather than my solution (or any other solution here).
Source: Stackoverflow.com