Iterate over the lines of a string

Question

I have a multi-line string defined like this   foo       this is  a multi-line string        This string we used as test-input for a parser I am writing  The parser-function receives a file-object as input and iterates over it  It does also call the next   method directly to skip lines  so I really need an iterator as input  not an iterable  I need an iterator that iterates over the individual lines of that string like a file-object would over the lines of a text-file  I could of course do it like this   lineiterator   iter foo splitlines      Is there a more direct way of doing this  In this scenario the string has to traversed once for the splitting  and then again by the parser  It doesn t matter in my test-case  since the string is very short there  I am just asking out of curiosity  Python has so many useful and efficient built-ins for such stuff  but I could find nothing that suits this need

User · Answer

If I read Modules cStringIO c correctly  this should be quite efficient  although somewhat verbose    from cStringIO import StringIO  def iterbuf buf       stri   StringIO buf      while True          nl   stri readline           if nl                    yield nl strip           else              raise StopIteration

User · Answer

You can iterate over  a file   which produces lines  including the trailing newline character  To make a  virtual file  out of a string  you can use StringIO   import io    for Py2 7 that would be import cStringIO as io  for line in io StringIO foo       print repr line

User · Answer

I m not sure what you mean by  then again by the parser    After the splitting has been done  there s no further traversal of the string  only a traversal of the list of split strings   This will probably actually be the fastest way to accomplish this  so long as the size of your string isn t absolutely huge   The fact that python uses immutable strings means that you must always create a new string  so this has to be done at some point anyway   If your string is very large  the disadvantage is in memory usage  you ll have the original string and a list of split strings in memory at the same time  doubling the memory required   An iterator approach can save you this  building a string as needed  though it still pays the  splitting  penalty   However  if your string is that large  you generally want to avoid even the unsplit string being in memory   It would be better just to read the string from a file  which already allows you to iterate through it as lines   However if you do have a huge string in memory already  one approach would be to use StringIO  which presents a file-like interface to a string  including allowing iterating by line  internally using  find to find the next newline    You then get   import StringIO s   StringIO StringIO myString  for line in s      do something with line

User · Answer

Regex-based searching is sometimes faster than generator approach   RRR   re compile r      n   def f4 arg       return  i group 1  for i in RRR finditer arg

User · Answer

I suppose you could roll your own   def parse string       retval          for char in string          retval    char if not char      n  else            if char      n               yield retval             retval          if retval          yield retval   I m not sure how efficient this implementation is  but that will only iterate over your string once   Mmm  generators   Edit   Of course you ll also want to add in whatever type of parsing actions you want to take  but that s pretty simple

User · Answer

Here are three possibilities   foo       this is  a multi-line string       def f1 foo foo   return iter foo splitlines     def f2 foo foo       retval          for char in foo          retval    char if not char      n  else            if char      n               yield retval             retval          if retval          yield retval  def f3 foo foo       prevnl   -1     while True        nextnl   foo find   n   prevnl   1        if nextnl  lt  0  break       yield foo prevnl   1 nextnl        prevnl   nextnl  if   name         main       for f in f1  f2  f3      print list f      Running this as the main script confirms the three functions are equivalent  With timeit  and a   100 for foo to get substantial strings for more precise measurement      python -mtimeit -s import asp   list asp f3     1000 loops  best of 3  370 usec per loop   python -mtimeit -s import asp   list asp f2     1000 loops  best of 3  1 36 msec per loop   python -mtimeit -s import asp   list asp f1     10000 loops  best of 3  61 5 usec per loop   Note we need the list   call to ensure the iterators are traversed  not just built   IOW  the naive implementation is so much faster it isn t even funny  6 times faster than my attempt with find calls  which in turn is 4 times faster than a lower-level approach   Lessons to retain  measurement is always a good thing  but must be accurate   string methods like splitlines are implemented in very fast ways  putting strings together by programming at a very low level  esp  by loops of    of very small pieces  can be quite slow   Edit  added  Jacob s proposal  slightly modified to give the same results as the others  trailing blanks on a line are kept   i e    from cStringIO import StringIO  def f4 foo foo       stri   StringIO foo      while True          nl   stri readline           if nl                    yield nl strip   n           else              raise StopIteration   Measuring gives     python -mtimeit -s import asp   list asp f4     1000 loops  best of 3  406 usec per loop   not quite as good as the  find based approach -- still  worth keeping in mind because it might be less prone to small off-by-one bugs  any loop where you see occurrences of  1 and -1  like my f3 above  should automatically trigger off-by-one suspicions -- and so should many loops which lack such tweaks and should have them -- though I believe my code is also right since I was able to check its output with other functions     But the split-based approach still rules   An aside  possibly better style for f4 would be   from cStringIO import StringIO  def f4 foo foo       stri   StringIO foo      while True          nl   stri readline           if nl        break         yield nl strip   n     at least  it s a bit less verbose   The need to strip trailing  ns unfortunately prohibits the clearer and faster replacement of the while loop with return iter stri   the iter part whereof is redundant in modern versions of Python  I believe since 2 3 or 2 4  but it s also innocuous    Maybe worth trying  also       return itertools imap lambda s  s strip   n    stri    or variations thereof -- but I m stopping here since it s pretty much a theoretical exercise wrt the strip based  simplest and fastest  one

[python] Iterate over the lines of a string

Examples related to python

Examples related to string

Examples related to iterator