[python] Python NLTK: SyntaxError: Non-ASCII character '\xc3' in file (Sentiment Analysis -NLP)

I am playing around with NLTK to do an assignment on sentiment analysis. I am using Python 2.7. NLTK 3.0 and NumPy1.9.1 version.

This is the code :

__author__ = 'karan'
import nltk
import re
import sys



def main():
    print("Start");
    # getting the stop words
    stopWords = open("english.txt","r");
    stop_word = stopWords.read().split();
    AllStopWrd = []
    for wd in stop_word:
        AllStopWrd.append(wd);
    print("stop words-> ",AllStopWrd);

    # sample and also cleaning it
    tweet1= 'Love, my new toyí ½í¸í ½í¸#iPhone6. Its good https://twitter.com/Sandra_Ortega/status/513807261769424897/photo/1'
    print("old tweet-> ",tweet1)
    tweet1 = tweet1.lower()
    tweet1 = ' '.join(re.sub("(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)"," ",tweet1).split())
    print(tweet1);
    tw = tweet1.split()
    print(tw)


    #tokenize
    sentences = nltk.word_tokenize(tweet1)
    print("tokenized ->", sentences)


    #remove stop words
    Otweet =[]
    for w in tw:
        if w not in AllStopWrd:
            Otweet.append(w);
    print("sans stop word-> ",Otweet)


    # get taggers for neg/pos/inc/dec/inv words
    taggers ={}
    negWords = open("neg.txt","r");
    neg_word = negWords.read().split();
    print("ned words-> ",neg_word)
    posWords = open("pos.txt","r");
    pos_word = posWords.read().split();
    print("pos words-> ",pos_word)
    incrWords = open("incr.txt","r");
    inc_word = incrWords.read().split();
    print("incr words-> ",inc_word)
    decrWords = open("decr.txt","r");
    dec_word = decrWords.read().split();
    print("dec wrds-> ",dec_word)
    invWords = open("inverse.txt","r");
    inv_word = invWords.read().split();
    print("inverse words-> ",inv_word)
    for nw in neg_word:
        taggers.update({nw:'negative'});
    for pw in pos_word:
        taggers.update({pw:'positive'});
    for iw in inc_word:
        taggers.update({iw:'inc'});
    for dw in dec_word:
        taggers.update({dw:'dec'});
    for ivw in inv_word:
        taggers.update({ivw:'inv'});
    print("tagger-> ",taggers)
    print(taggers.get('little'))

    # get parts of speech
    posTagger = [nltk.pos_tag(tw)]
    print("posTagger-> ",posTagger)

main();

This is the error that I am getting when running my code:

SyntaxError: Non-ASCII character '\xc3' in file C:/Users/karan/PycharmProjects/mainProject/sentiment.py on line 19, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

How do I fix this error?

I also tried the code using Python 3.4.2 and with NLTK 3.0 and NumPy 1.9.1 but then I get the error:

Traceback (most recent call last):
  File "C:/Users/karan/PycharmProjects/mainProject/sentiment.py", line 80, in <module>
    main();
  File "C:/Users/karan/PycharmProjects/mainProject/sentiment.py", line 72, in main
    posTagger = [nltk.pos_tag(tw)]
  File "C:\Python34\lib\site-packages\nltk\tag\__init__.py", line 100, in pos_tag
    tagger = load(_POS_TAGGER)
  File "C:\Python34\lib\site-packages\nltk\data.py", line 779, in load
    resource_val = pickle.load(opened_resource)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcb in position 0: ordinal not in range(128)

This question is related to python unicode nlp nltk

The answer is


Add the following to the top of your file # coding=utf-8

If you go to the link in the error you can seen the reason why:

Defining the Encoding

Python will default to ASCII as standard encoding if no other encoding hints are given. To define a source code encoding, a magic comment must be placed into the source files either as first or second line in the file, such as: # coding=


Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to unicode

How to resolve TypeError: can only concatenate str (not "int") to str (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape UnicodeEncodeError: 'ascii' codec can't encode character at special name Python NLTK: SyntaxError: Non-ASCII character '\xc3' in file (Sentiment Analysis -NLP) HTML for the Pause symbol in audio and video control Javascript: Unicode string to hex Concrete Javascript Regex for Accented Characters (Diacritics) Replace non-ASCII characters with a single space UTF-8 in Windows 7 CMD NameError: global name 'unicode' is not defined - in Python 3

Examples related to nlp

Replace specific text with a redacted version using Python How to return history of validation loss in Keras How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn? Python NLTK: SyntaxError: Non-ASCII character '\xc3' in file (Sentiment Analysis -NLP) Stopword removal with NLTK How to get rid of punctuation using NLTK tokenizer? Calculate cosine similarity given 2 sentence strings How do I tokenize a string sentence in NLTK? How to compute the similarity between two text documents? How do I do word Stemming or Lemmatization?

Examples related to nltk

re.sub erroring with "Expected string or bytes-like object" how to check which version of nltk, scikit learn installed? Python NLTK: SyntaxError: Non-ASCII character '\xc3' in file (Sentiment Analysis -NLP) NLTK and Stopwords Fail #lookuperror Resource u'tokenizers/punkt/english.pickle' not found How do I download NLTK data? Stopword removal with NLTK n-grams in python, four, five, six grams? pip issue installing almost any library How to get rid of punctuation using NLTK tokenizer?