Reading a file line by line in Go

Question

I m unable to find file ReadLine function in Go  I can figure out how to quickly write one  but I am just wondering if I m overlooking something here  How does one read a file line by line

User · Answer

bufio Reader ReadLine   works well  But if you want to read each line by a string  try to use ReadString   n    It doesn t need to reinvent the wheel

User · Answer

In Go 1 1 and newer the most simple way to do this is with a bufio Scanner  Here is a simple example that reads lines from a file   package main  import        bufio       fmt       log       os     func main         file  err    os Open   path to file txt       if err    nil           log Fatal err            defer file Close        scanner    bufio NewScanner file      for scanner Scan             fmt Println scanner Text               if err    scanner Err    err    nil           log Fatal err            This is the cleanest way to read from a Reader line by line   There is one caveat  Scanner does not deal well with lines longer than 65536 characters  If that is an issue for you then then you should probably roll your own on top of Reader Read

User · Answer

Another method is to use the io ioutil and strings libraries to read the entire file s bytes  convert them into a string and split them using a   n   newline  character as the delimiter  for example   import        io ioutil       strings     func main         bytesRead       ioutil ReadFile  something txt       file content    string bytesRead      lines    strings Split file content    n       Technically you re not reading the file line-by-line  however you are able to parse each line using this technique  This method is applicable to smaller files  If you re attempting to parse a massive file use one of the techniques that reads line-by-line

User · Answer

Example from this gist  func readLine path string      inFile  err    os Open path    if err    nil        fmt Println err Error            path       return       defer inFile Close      scanner    bufio NewScanner inFile    for scanner Scan         fmt Println scanner Text       the line         but this gives an error when there is a line that larger than Scanner s buffer   When that happened  what I do is use reader    bufio NewReader inFile  create and concat my own buffer either using ch  err    reader ReadByte   or len  err    reader Read myBuffer   Another way that I use  replace os Stdin with file like above   this one concats when lines are long  isPrefix  and ignores empty lines    func readLines     string     r    bufio NewReader os Stdin    bytes      byte     lines      string     for       line  isPrefix  err    r ReadLine       if err    nil         break           bytes   append bytes  line         if  isPrefix         str    strings TrimSpace string bytes         if len str   gt  0           lines   append lines  str          bytes     byte                       if len bytes   gt  0       lines   append lines  string bytes         return lines

User · Answer

import         bufio        os     var       reader   bufio NewReader os Stdin     func ReadFromStdin   string      result       reader ReadString   n       witl    result  len result -1      return witl     Here is an example with function ReadFromStdin   it s like fmt Scan  amp name  but its takes all strings with blank spaces like   Hello My Name Is       var name string   ReadFromStdin    println name

User · Answer

NOTE  The accepted answer was correct in early versions of Go   See the highest voted answer contains the more recent idiomatic way to achieve this   There is function ReadLine in package bufio   Please note that if the line does not fit into the read buffer  the function will return an incomplete line  If you want to always read a whole line in your program by a single call to a function  you will need to encapsulate the ReadLine function into your own function which calls ReadLine in a for-loop   bufio ReadString   n   isn t fully equivalent to ReadLine because ReadString is unable to handle the case when the last line of a file does not end with the newline character

User · Answer

strip   n  or read until EOF  return error if read error   func readline reader io Reader   line   byte  err error           line   make   byte  0  100                                    for                                                               b    make   byte  1                                           n  er    reader Read b                                        if n  gt  0                                                          c    b 0                                                      if c      n       end of line                                     break                                                                                                                   line   append line  c                                                                                                   if er    nil                                                      err   er                                                      return                                                                                                                                                                            return

User · Answer

There two common way to read file line by line    Use bufio Scanner Use ReadString ReadBytes     in bufio Reader   In my testcase   250MB   2 500 000 lines  bufio Scanner time used  0 395491384s  is faster than bufio Reader ReadString time used  0 446867622s    Source code  https   github com xpzouying go-practice tree master read file line by line  Read file use bufio Scanner   func scanFile         f  err    os OpenFile logfile  os O RDONLY  os ModePerm      if err    nil           log Fatalf  open file error   v   err          return           defer f Close        sc    bufio NewScanner f      for sc Scan                 sc Text       GET the line string           if err    sc Err    err    nil           log Fatalf  scan file error   v   err          return           Read file use bufio Reader   func readFileLines         f  err    os OpenFile logfile  os O RDONLY  os ModePerm      if err    nil           log Fatalf  open file error   v   err          return           defer f Close        rd    bufio NewReader f      for           line  err    rd ReadString   n           if err    nil               if err    io EOF                   break                            log Fatalf  read file line error   v   err              return                       line     GET the line string

User · Answer

EDIT  As of go1 1  the idiomatic solution is to use bufio Scanner  I wrote up a way to easily read each line from a file  The Readln  bufio Reader  function returns a line  sans  n  from the underlying bufio Reader struct      Readln returns a single line  without the ending  n     from the input buffered reader     An error is returned iff there is an error with the    buffered reader  func Readln r  bufio Reader   string  error      var  isPrefix bool   true        err error   nil        line  ln   byte           for isPrefix  amp  amp  err    nil         line  isPrefix  err   r ReadLine         ln   append ln  line           return string ln  err     You can use Readln to read every line from a file  The following code reads every line in a file and outputs each line to stdout   f  err    os Open fi  if err    nil       fmt Printf  error opening file   v n  err      os Exit 1    r    bufio NewReader f  s  e    Readln r  for e    nil       fmt Println s      s e   Readln r      Cheers

User · Answer

You can also use ReadString with  n as a separator     f  err    os Open filename    if err    nil       fmt Println  error opening file    err      os Exit 1        defer f Close     r    bufio NewReader f    for       path  err    r ReadString 10     0x0A separator   newline     if err    io EOF            do something here       break       else if err    nil         return err    if you return error

User · Answer

In the code bellow  I read the interests from the CLI until the user hits enter and I m using Readline   interests    make   string  1  r    bufio NewReader os Stdin  for true       fmt Print  Give me an interest        t          r ReadLine       interests   append interests  string t       if len t     0           break          fmt Println interests

User · Answer

In the new version of Go 1 16 we can use package embed to read the file contents as shown below  package main  import   quot embed quot    func main           go embed  quot hello txt quot      var s string     print s         go embed  quot hello txt quot      var b   byte     print string b          go embed hello txt     var f embed FS     data       f ReadFile  quot hello txt quot       print string data      For more details go through https   tip golang org pkg embed  And https   golangtutorial dev tips embed-files-in-go

User · Answer

Use   reader ReadString   n    If you don t mind that the line could be very long  i e  use a lot of RAM   It keeps the  n at the end of the string returned    reader ReadLine    If you care about limiting RAM consumption and don t mind the extra work of handling the case where the line is greater than the reader s buffer size     I tested the various solutions suggested by writing a program to test the scenarios which are identified as problems in other answers   A file with a 4MB line  A file which doesn t end with a line break   I found that   The Scanner solution does not handle long lines  The ReadLine solution is complex to implement  The ReadString solution is the simplest and works for long lines   Here is code which demonstrates each solution  it can be run via go run main go  or at https   play golang org p RAW3sGblbas package main  import        quot bufio quot       quot bytes quot       quot fmt quot       quot io quot       quot os quot     func readFileWithReadString fn string   err error        fmt Println  quot readFileWithReadString quot        file  err    os Open fn      if err    nil           return err           defer file Close           Start reading from the file with a reader      reader    bufio NewReader file      var line string     for           line  err   reader ReadString   n           if err    nil  amp  amp  err    io EOF               break                       Process the line here          fmt Printf  quot   gt  Read  d characters n quot   len line           fmt Printf  quot   gt   gt   s n quot   limitLength line  50            if err    nil               break                     if err    io EOF           fmt Printf  quot   gt  Failed with error   v n quot   err          return err           return    func readFileWithScanner fn string   err error        fmt Println  quot readFileWithScanner  scanner fails with long lines  quot           Don t use this  it doesn t work with long lines         file  err    os Open fn      if err    nil           return err           defer file Close           Start reading from the file using a scanner      scanner    bufio NewScanner file      for scanner Scan             line    scanner Text               Process the line here          fmt Printf  quot   gt  Read  d characters n quot   len line           fmt Printf  quot   gt   gt   s n quot   limitLength line  50             if scanner Err      nil           fmt Printf  quot   gt  Failed with error  v n quot   scanner Err            return scanner Err             return    func readFileWithReadLine fn string   err error        fmt Println  quot readFileWithReadLine quot        file  err    os Open fn      if err    nil           return err           defer file Close           Start reading from the file with a reader      reader    bufio NewReader file      for           var buffer bytes Buffer          var l   byte         var isPrefix bool         for               l  isPrefix  err   reader ReadLine               buffer Write l                 If we ve reached the end of the line  stop reading              if  isPrefix                   break                              If we re at the EOF  break              if err    nil                   if err    io EOF                       return err                                   break                                 line    buffer String               Process the line here          fmt Printf  quot   gt  Read  d characters n quot   len line           fmt Printf  quot   gt   gt   s n quot   limitLength line  50            if err    io EOF               break                     if err    io EOF           fmt Printf  quot   gt  Failed with error   v n quot   err          return err           return    func main         testLongLines       testLinesThatDoNotFinishWithALinebreak      func testLongLines         fmt Println  quot Long lines quot       fmt Println        createFileWithLongLine  quot longline txt quot       readFileWithReadString  quot longline txt quot       fmt Println       readFileWithScanner  quot longline txt quot       fmt Println       readFileWithReadLine  quot longline txt quot       fmt Println      func testLinesThatDoNotFinishWithALinebreak         fmt Println  quot No linebreak quot       fmt Println        createFileThatDoesNotEndWithALineBreak  quot nolinebreak txt quot       readFileWithReadString  quot nolinebreak txt quot       fmt Println       readFileWithScanner  quot nolinebreak txt quot       fmt Println       readFileWithReadLine  quot nolinebreak txt quot       fmt Println      func createFileThatDoesNotEndWithALineBreak fn string   err error        file  err    os Create fn      if err    nil           return err           defer file Close        w    bufio NewWriter file      w WriteString  quot Does not end with linebreak  quot       w Flush       return    func createFileWithLongLine fn string   err error        file  err    os Create fn      if err    nil           return err           defer file Close        w    bufio NewWriter file      fs    1024   1024   4    4MB        Create a 4MB long line consisting of the letter a      for i    0  i  lt  fs  i             w WriteRune  a                Terminate the line with a break      w WriteRune   n           Put in a second line  which doesn t have a linebreak      w WriteString  quot Second line  quot       w Flush       return    func limitLength s string  length int  string       if len s   lt  length           return s           return s  length     I tested on   go version go1 15 darwin amd64 go version go1 7 windows amd64 go version go1 6 3 linux amd64 go version go1 7 4 darwin amd64  The test program outputs  Long lines  readFileWithReadString   gt  Read 4194305 characters   gt   gt  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa   gt  Read 12 characters   gt   gt  Second line   readFileWithScanner  scanner fails with long lines    gt  Failed with error bufio Scanner  token too long  readFileWithReadLine   gt  Read 4194304 characters   gt   gt  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa   gt  Read 12 characters   gt   gt  Second line    gt  Read 0 characters   gt   gt    No linebreak  readFileWithReadString   gt  Read 28 characters   gt   gt  Does not end with linebreak   readFileWithScanner  scanner fails with long lines    gt  Read 28 characters   gt   gt  Does not end with linebreak   readFileWithReadLine   gt  Read 28 characters   gt   gt  Does not end with linebreak    gt  Read 0 characters   gt   gt

[string] Reading a file line by line in Go

Examples related to string

Examples related to file

Examples related to parsing

Examples related to go

Examples related to line