We have the following string which is a valid JSON written to a file on HDFS.
{
"id":"tag:search.twitter.com,2005:564407444843950080",
"objectType":"activity",
"actor":{
"objectType":"person",
"id":"id:twitter.com:2302910022",
"link":"http%3A%2F%2Fwww.twitter.com%2Fme7me4610012",
"displayName":"",
"postedTime":"2014-01-21T11:06:06.000Z",
"image":"https%3A%2F%2Fpbs.twimg.com%2Fprofile_images%2F563125491159162881%2FfypkHK3M_normal.jpeg",
"summary":"???????????????????? ????????????? ??????????????? ???? ???????????????? ?????? ????? ??????????? lloooo_20",
"links":[
{
"href":null,
"rel":"me"
}
],
"friendsCount":10503,
"followersCount":10325,
"listedCount":12,
"statusesCount":84957,
"twitterTimeZone":null,
"verified":false,
"utcOffset":null,
"preferredUsername":"me7me4610012",
"languages":[
"ar"
],
"favoritesCount":17
},
"verb":"share",
"postedTime":"2015-02-08T12:56:35.000Z",
"generator":{
"displayName":"Twitter for Android",
"link":"http%3A%2F%2Ftwitter.com%2Fdownload%2Fandroid"
},
"provider":{
"objectType":"service",
"displayName":"Twitter",
"link":"http%3A%2F%2Fwww.twitter.com"
},
"link":"http%3A%2F%2Ftwitter.com%2Fme7me4610012%2Fstatuses%2F564407444843950080",
"body":"RT @sckud1: ?????: ???? ???? ???? ?????? ??? ??? ???? ??? ???? ?? ????? ???? ????? ?????: ????? http%3A%2F%2Ft.co%2FC55SaQKmUV http%3A%2F%2Ft.co%2Ft5TjIln…",
"object":{
"id":"tag:search.twitter.com,2005:564407126526013440",
"objectType":"activity",
"actor":{
"objectType":"person",
"id":"id:twitter.com:462268717",
"link":"http%3A%2F%2Fwww.twitter.com/sckud1",
"displayName":"??? ?????",
"postedTime":"2012-01-12T19:24:17.000Z",
"image":"https%3A%2F%2Fpbs.twimg.com%2Fprofile_images%2F508424482885615616%2FmPBGZBPx_normal.jpeg",
"summary":"?????? ?? ??? ?????? ???? ?? ?????? ??? ???? ?? ????? ????? ????? http%3A%2F%2Fmarketgulf.com",
"links":[
{
"href":"http%3A%2F%2Fmarketgulf.com",
"rel":"me"
}
],
"friendsCount":435237,
"followersCount":464951,
"listedCount":708,
"statusesCount":1071685,
"twitterTimeZone":"Riyadh",
"verified":false,
"utcOffset":"10800",
"preferredUsername":"sckud1",
"languages":[
"ar"
],
"location":{
"objectType":"place",
"displayName":"Made in K S A"
},
"favoritesCount":77
},
"verb":"post",
"postedTime":"2015-02-08T12:55:19.000Z",
"generator":{
"displayName":"Tweet Old Post",
"link":"http%3A%2F%2Fwww.ajaymatharu.com%2F"
},
"provider":{
"objectType":"service",
"displayName":"Twitter",
"link":"http%3A%2F%2Fwww.twitter.com"
},
"link":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatuses%2F564407126526013440",
"body":"?????: ???? ???? ???? ?????? ??? ??? ???? ??? ???? ?? ????? ???? ????? ?????: ????? http%3A%2F%2Ft.co%2FC55SaQKmUV http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"object":{
"objectType":"note",
"id":"object:search.twitter.com,2005:564407126526013440",
"summary":"?????: ???? ???? ???? ?????? ??? ??? ???? ??? ???? ?? ????? ???? ????? ?????: ????? http%3A%2F%2Ft.co%2FC55SaQKmUV http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"link":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatuses%2F564407126526013440",
"postedTime":"2015-02-08T12:55:19.000Z"
},
"favoritesCount":0,
"twitter_entities":{
"hashtags":[
],
"trends":[
],
"urls":[
{
"url":"http%3A%2F%2Ft.co%2FC55SaQKmUV",
"expanded_url":"http%3A%2F%2Fwww.hasterya.com%2Farchives%2F34688utm_source%3DReviveOldPost%26utm_medium%3Dsocial%26utm_campaign%3DReviveOldPost",
"display_url":"hasterya.com/archives/34688…",
"indices":[
85,
107
]
}
],
"user_mentions":[
],
"symbols":[
],
"media":[
{
"id":564407126341468160,
"id_str":"564407126341468160",
"indices":[
108,
130
],
"media_url":"http%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"media_url_https":"https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"url":"http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"display_url":"pic.twitter.com/t5TjIlnZgN",
"expanded_url":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatus%2F564407126526013440%2Fphoto%2F1",
"type":"photo",
"sizes":{
"large":{
"w":320,
"h":180,
"resize":"fit"
},
"thumb":{
"w":150,
"h":150,
"resize":"crop"
},
"small":{
"w":320,
"h":180,
"resize":"fit"
},
"medium":{
"w":320,
"h":180,
"resize":"fit"
}
}
}
]
},
"twitter_extended_entities":{
"media":[
{
"id":564407126341468160,
"id_str":"564407126341468160",
"indices":[
108,
130
],
"media_url":"http%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"media_url_https":"https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"url":"http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"display_url":"pic.twitter.com/t5TjIlnZgN",
"expanded_url":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatus%2F564407126526013440%2Fphoto%2F1",
"type":"photo",
"sizes":{
"large":{
"w":320,
"h":180,
"resize":"fit"
},
"thumb":{
"w":150,
"h":150,
"resize":"crop"
},
"small":{
"w":320,
"h":180,
"resize":"fit"
},
"medium":{
"w":320,
"h":180,
"resize":"fit"
}
}
}
]
},
"twitter_filter_level":"low",
"twitter_lang":"ar"
},
"favoritesCount":0,
"twitter_entities":{
"hashtags":[
],
"trends":[
],
"urls":[
{
"url":"http%3A%2F%2Ft.co%2FC55SaQKmUV",
"expanded_url":"http%3A%2F%2Fwww.hasterya.com%2Farchives%2F34688utm_source%3DReviveOldPost%26utm_medium%3Dsocial%26utm_campaign%3DReviveOldPost",
"display_url":"hasterya.com/archives/34688…",
"indices":[
97,
119
]
}
],
"user_mentions":[
{
"screen_name":"sckud1",
"name":"??? ?????",
"id":462268717,
"id_str":"462268717",
"indices":[
3,
10
]
}
],
"symbols":[
],
"media":[
{
"id":564407126341468160,
"id_str":"564407126341468160",
"indices":[
139,
140
],
"media_url":"http%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"media_url_https":"https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"url":"http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"display_url":"pic.twitter.com/t5TjIlnZgN",
"expanded_url":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatus%2F564407126526013440%2Fphoto%2F1",
"type":"photo",
"sizes":{
"large":{
"w":320,
"h":180,
"resize":"fit"
},
"thumb":{
"w":150,
"h":150,
"resize":"crop"
},
"small":{
"w":320,
"h":180,
"resize":"fit"
},
"medium":{
"w":320,
"h":180,
"resize":"fit"
}
},
"source_status_id":564407126526013440,
"source_status_id_str":"564407126526013440"
}
]
},
"twitter_extended_entities":{
"media":[
{
"id":564407126341468160,
"id_str":"564407126341468160",
"indices":[
139,
140
],
"media_url":"http%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"media_url_https":"https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FB9UtSoJIQAA07-r.jpg",
"url":"http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"display_url":"pic.twitter.com/t5TjIlnZgN",
"expanded_url":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatus%2F564407126526013440%2Fphoto%2F1",
"type":"photo",
"sizes":{
"large":{
"w":320,
"h":180,
"resize":"fit"
},
"thumb":{
"w":150,
"h":150,
"resize":"crop"
},
"small":{
"w":320,
"h":180,
"resize":"fit"
},
"medium":{
"w":320,
"h":180,
"resize":"fit"
}
},
"source_status_id":564407126526013440,
"source_status_id_str":"564407126526013440"
}
]
},
"twitter_filter_level":"low",
"twitter_lang":"ar",
"retweetCount":1,
"gnip":{
"matching_rules":[
{
"tag":"ISIS66"
}
],
"urls":[
{
"url":"http%3A%2F%2Ft.co%2Ft5TjIlnZgN",
"expanded_url":"http%3A%2F%2Ftwitter.com%2Fsckud1%2Fstatus%2F564407126526013440%2Fphoto%2F1",
"expanded_status":200
},
{
"url":"http%3A%2F%2Ft.co%2FC55SaQKmUV",
"expanded_url":"http%3A%2F%2Fwww.hasterya.com%2Farchives%2F34688utm_source%3DReviveOldPost%26utm_medium%3Dsocial%26utm_campaign%3DReviveOldPost",
"expanded_status":200
}
],
"klout_score":50,
"language":{
"value":"ar"
}
}
}
EDIT
We configure a flume agent that reads the data from that file and pass it to Solr sink but unfortunately this exception in the title is throw.
and here is the stack trace
org.kitesdk.morphline.api.MorphlineRuntimeException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'http': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@20d7aa52; line: 1, column: 9]
at org.kitesdk.morphline.stdio.AbstractParser.doProcess(AbstractParser.java:98)
at org.kitesdk.morphline.base.AbstractCommand.process(AbstractCommand.java:156)
at org.kitesdk.morphline.stdlib.TryRulesBuilder$TryRules.doProcess(TryRulesBuilder.java:120)
at org.kitesdk.morphline.base.AbstractCommand.process(AbstractCommand.java:156)
at org.kitesdk.morphline.base.AbstractCommand.doProcess(AbstractCommand.java:181)
at org.kitesdk.morphline.base.AbstractCommand.process(AbstractCommand.java:156)
at org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.process(MorphlineHandlerImpl.java:128)
at org.apache.flume.sink.solr.morphline.MorphlineSink.process(MorphlineSink.java:141)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:744)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'http': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@20d7aa52; line: 1, column: 9]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1524)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:557)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3095)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2340)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:818)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:698)
at com.fasterxml.jackson.databind.MappingIterator.hasNextValue(MappingIterator.java:159)
at org.kitesdk.morphline.json.ReadJsonBuilder$ReadJson.doProcess(ReadJsonBuilder.java:109)
at org.kitesdk.morphline.stdio.AbstractParser.doProcess(AbstractParser.java:96)
... 10 more
I faced this exception for a long time and was not able to pinpoint the problem. The exception says line 1 column 9. The mistake I did is to get the first line of the file which flume is processing.
Apache flume process the content of the file in patches. So, when flume throws this exception and says line 1, it means the first line in the current patch.
If your flume agent is configured to use batch size = 100, and (for example) the file contains 400 lines, this means the exception is thrown in one of the following lines 1, 101, 201,301.
How to discover the line which causes the problem?
You have three ways to do that.
1- pull the source code and run the agent in debug mode. If you are an average developer like me and do not know how to make this, check the other two options.
2- Try to split the file based on the batch size and run the flume agent again. If you split the file into 4 files, and the invalid json exists between lines 301 and 400, the flume agent will process the first 3 files and stop at the fourth file. Take the fourth file and again split it into more smaller files. continue the process until you reach a file with only one line and flume fails while processing it.
3- Reduce the batch size of the flume agent to only one and compare the number of processed events in the output of the sink you are using. For example, in my case I am using Solr sink. The file contains 400 lines. The flume agent is configured with batch size=100. When I run the flume agent, it fails at some point and throw that exception. At this point check how many documents are ingested in Solr. If the invalid json exists at line 346, the number of documents indexed into Solr will be 345, so the next line is the line which causes the problem.
In my case I followed the third option and fortunately I pinpoint the line which causes the problem.
This is a long answer but it actually does not solve the exception. How I overcome this exception?
I have no idea why Jackson library complain while parsing a json string contains escaped characters \n \r \t. I think (but I am not sure) the Jackson parser is by default escaping these characters which cases the json string to be split into two lines (in case of \n) and then it deals each line as a separate json string.
In my case we used a customized interceptor to remove these characters before being processed by the flume agent. This is the way we solved this problem.
It might be obvious, but make sure that you are sending to the parser URL object not a String containing www adress. This will not work:
ObjectMapper mapper = new ObjectMapper();
String www = "www.sample.pl";
Weather weather = mapper.readValue(www, Weather.class);
But this will:
ObjectMapper mapper = new ObjectMapper();
URL www = new URL("http://www.oracle.com/");
Weather weather = mapper.readValue(www, Weather.class);
Add produces = "application/json" in @RequestMapping
Source: Stackoverflow.com