Replace non-ASCII characters with a single space

Question

I need to replace all non-ASCII   x00- x7F  characters with a space  I m surprised that this is not dead-easy in Python  unless I m missing something  The following function simply removes all non-ASCII characters   def remove non ascii 1 text        return    join i for i in text if ord i  lt 128    And this one replaces non-ASCII characters with the amount of spaces as per the amount of bytes in the character code point  i e  the     character is replaced with 3 spaces    def remove non ascii 2 text        return re sub r    x00- x7F        text    How can I replace all non-ASCII characters with a single space   Of the myriad of similar SO questions  none address character replacement as opposed to stripping  and  additionally address all non-ascii characters not  a  specific  character

User · Answer

As a native and efficient approach  you don t need to use ord or any loop over the characters  Just encode with ascii and ignore the errors   The following will just remove the non-ascii characters   new string   old string encode  ascii  errors  ignore     Now if you want to replace the deleted characters just do the following   final string   new string   b       len old string  - len new string

User · Answer

For you the get the most alike representation of your original string I recommend the unidecode module   from unidecode import unidecode def remove non ascii text       return unidecode unicode text  encoding    utf-8      Then you can use it in a string   remove non ascii  Ce    a   Cenia

User · Answer

If the replacement character can be     instead of a space  then I d suggest result   text encode  ascii    replace   decode        Test the performance of different non-ASCII replacement methods       import re from timeit import timeit     10 000 is typical in the project that I m working on and most of the text   is going to be non-ASCII  text          10 000   print timeit          result      join  c if ord c   lt  128 else     for c in text                number 1000      globals globals        print timeit          result   text encode  ascii    replace   decode                number 1000      globals globals         Results   0 7208260721400134 0 009975979187503592

User · Answer

For character processing  use Unicode strings   PythonWin 3 3 0  v3 3 0 bd8afb90ebf2  Sep 29 2012  10 57 17   MSC v 1600 64 bit  AMD64   on win32   gt  gt  gt  s  ABC  def   gt  gt  gt  import re  gt  gt  gt  re sub r    x00- x7f   r    s      Each char is a Unicode codepoint   ABC  def   gt  gt  gt  b   s encode  utf8    gt  gt  gt  re sub rb    x00- x7f   rb    b    Each char is a 3-byte UTF-8 sequence  b ABC      def    But note you will still have a problem if your string contains decomposed Unicode characters  separate character and combining accent marks  for example     gt  gt  gt  s    ma  ana   gt  gt  gt  len s  6  gt  gt  gt  import unicodedata as ud  gt  gt  gt  n ud normalize  NFD  s   gt  gt  gt  n  man ana   gt  gt  gt  len n  7  gt  gt  gt  re sub r    x00- x7f   r    s    single codepoint  ma ana   gt  gt  gt  re sub r    x00- x7f   r    n    only combining mark replaced  man ana

User · Answer

Your    join   expression is filtering  removing anything non-ASCII  you could use a conditional expression instead   return    join  i if ord i   lt  128 else     for i in text     This handles characters one by one and would still use one space per character replaced   Your regular expression should just replace consecutive non-ASCII characters with a space   re sub r    x00- x7F         text    Note the   there

User · Answer

When we use the ascii   it escapes the non-ascii characters and it doesn t change ascii characters correctly  So my main thought is  it doesn t change the ASCII characters  so I am iterating through the string and checking if the character is changed  If it changed then replacing it with the replacer  what you give  For example      a single space  or      with a question mark   def remove x  replacer         for i in x          if f quot   i   quot     ascii i               pass         else              x x replace i replacer       return x remove  h  i        Result   quot h i quot   with single space between   Syntax    remove str non ascii replacer  str   Here you will give the string you want to work with  non ascii replacer   Here you will give the replacer which you want to replace all the non ASCII characters with

User · Answer

Potentially for a different question  but I m providing my version of  Alvero s answer  using unidecode    I want to do a  regular  strip on my strings  i e  the beginning and end of my string for whitespace characters  and then replace only other whitespace characters with a  regular  space  i e    Ce    a ma  ana        to   Ce    a ma  ana       def safely stripped s  str       return     join          stripped for stripped in          bit strip   for bit in             join  c if unidecode c  else      for c in s  strip   split            if stripped    We first replace all non-unicode spaces with a regular space  and join it back again       join  c if unidecode c  else      for c in s    And then we split that again  with python s normal split  and strip each  bit     bit strip   for bit in s split      And lastly join those back again  but only if the string passes an if test       join stripped for stripped in s if stripped    And with that  safely stripped      Ce    a ma  ana       correctly returns  Ce    a ma  ana

User · Answer

What about this one   def replace trash unicode string        for i in range 0  len unicode string             try               unicode string i  encode  ascii            except                 means it s non-ASCII               unicode string unicode string i  replace       replacing it with a single space      return unicode string

[python] Replace non-ASCII characters with a single space

Examples related to python

Examples related to unicode

Examples related to encoding

Examples related to ascii