How to efficiently concatenate strings in go

Question

In Go  a string is a primitive type  which means it is read-only  and every manipulation of it will create a new string  So if I want to concatenate strings many times without knowing the length of the resulting string  what s the best way to do it  The naive way would be  var s string for i    0  i  lt  1000  i         s    getShortStringFromSomewhere     return s  but that does not seem very efficient

User · Answer

This is the fastest solution that does not require you to know or calculate the overall buffer size first:

var data []byte
for i := 0; i < 1000; i++ {
    data = append(data, getShortStringFromSomewhere()...)
}
return string(data)

By my benchmark, it's 20% slower than the copy solution (8.1ns per append rather than 6.72ns) but still 55% faster than using bytes.Buffer.

User · Answer

New Way   From Go 1 10 there is a strings Builder type  please take a look at this answer for more detail   Old Way   Use the bytes package  It has a Buffer type which implements io Writer    package main  import        bytes       fmt     func main         var buffer bytes Buffer      for i    0  i  lt  1000  i             buffer WriteString  a              fmt Println buffer String        This does it in O n  time

User · Answer

package main  import      fmt     func main         var str1    string1      var str2    string2      out    fmt Sprintf   s  s   str1  str2      fmt Println out

User · Answer

goutils JoinBetween   func JoinBetween in   string  separator string  startIndex  endIndex int  string       if in    nil           return               noOfItems    endIndex - startIndex      if noOfItems  lt   0           return EMPTY            var builder strings Builder      for i    startIndex  i  lt  endIndex  i             if i  gt  startIndex               builder WriteString separator                    builder WriteString in i             return builder String

User · Answer

package main  import    fmt     func main         var str1    string1      var str2    string2      result    make   byte  0      result   append result    byte str1          result   append result    byte str2          result   append result    byte str1          result   append result    byte str2           fmt Println string result

User · Answer

Expanding on cd1 s answer  You might use append   instead of copy    append   makes ever bigger advance provisions  costing a little more memory  but saving time  I added two more benchmarks at the top of yours  Run locally with   go test -bench   -benchtime 100ms   On my thinkpad T400s it yields   BenchmarkAppendEmpty    50000000         5 0 ns op BenchmarkAppendPrealloc 50000000         3 5 ns op BenchmarkCopy           20000000        10 2 ns op

User · Answer

I do it using the following  -   package main  import        fmt       strings     func main         concatenation   strings Join   string  a   b   c         where second parameter is a separator       fmt Println concatenation    abc

User · Answer

If you have a string slice that you want to efficiently convert to a string then you can use this approach  Otherwise  take a look at the other answers  There is a library function in the strings package called Join  http   golang org pkg strings  Join A look at the code of Join shows a similar approach to Append function Kinopiko wrote  https   golang org src strings strings go L420 Usage  import        quot fmt quot        quot strings quot      func main         s      string  quot this quot    quot is quot    quot a quot    quot joined quot    quot string n quot        fmt Printf strings Join s   quot   quot            test bin this is a joined string

User · Answer

I just benchmarked the top answer posted above in my own code  a recursive tree walk  and the simple concat operator is actually faster than the BufferString   func  r  record  String   string       buffer    bytes NewBufferString          fmt Fprint buffer     r name          for i    0  i  lt  len r subs   i             fmt Fprint buffer   t  r subs i             fmt Fprint buffer     r size    n       return buffer String       This took 0 81 seconds  whereas the following code   func  r  record  String   string       s            r name              for i    0  i  lt  len r subs   i             s    r subs i  String             s           strconv FormatInt r size 10       n      return s      only took 0 61 seconds  This is probably due to the overhead of creating the new BufferString   Update  I also benchmarked the join function and it ran in 0 54 seconds   func  r  record  String   string       var parts   string     parts   append parts         r name               for i    0  i  lt  len r subs   i             parts   append parts  r subs i  String              parts   append parts  strconv FormatInt r size 10      n       return strings Join parts

User · Answer

You could create a big slice of bytes and copy the bytes of the short strings into it using string slices  There is a function given in  Effective Go    func Append slice  data  byte    byte       l    len slice       if l   len data   gt  cap slice       reallocate            Allocate double what s needed  for future growth          newSlice    make   byte   l len data   2              Copy data  could use bytes Copy             for i  c    range slice               newSlice i    c                   slice   newSlice            slice   slice 0 l len data        for i  c    range data           slice l i    c           return slice      Then when the operations are finished  use string     on the big slice of bytes to convert it into a string again

User · Answer

If you know the total length of the string that you re going to preallocate then the most efficient way to concatenate strings may be using the builtin function copy  If you don t know the total length before hand  do not use copy  and read the other answers instead  In my tests  that approach is  3x faster than using bytes Buffer and much  much faster   12 000x  than using the operator    Also  it uses less memory  I ve created a test case to prove this and here are the results   BenchmarkConcat  1000000    64497 ns op   502018 B op   0 allocs op BenchmarkBuffer  100000000  15 5  ns op   2 B op        0 allocs op BenchmarkCopy    500000000  5 39  ns op   0 B op        0 allocs op   Below is code for testing  package main  import        quot bytes quot       quot strings quot       quot testing quot     func BenchmarkConcat b  testing B        var str string     for n    0  n  lt  b N  n             str     quot x quot            b StopTimer        if s    strings Repeat  quot x quot   b N   str    s           b Errorf  quot unexpected result  got  s  want  s quot   str  s           func BenchmarkBuffer b  testing B        var buffer bytes Buffer     for n    0  n  lt  b N  n             buffer WriteString  quot x quot             b StopTimer        if s    strings Repeat  quot x quot   b N   buffer String      s           b Errorf  quot unexpected result  got  s  want  s quot   buffer String    s           func BenchmarkCopy b  testing B        bs    make   byte  b N      bl    0      b ResetTimer       for n    0  n  lt  b N  n             bl    copy bs bl     quot x quot             b StopTimer        if s    strings Repeat  quot x quot   b N   string bs     s           b Errorf  quot unexpected result  got  s  want  s quot   string bs   s              Go 1 10 func BenchmarkStringBuilder b  testing B        var strBuilder strings Builder      b ResetTimer       for n    0  n  lt  b N  n             strBuilder WriteString  quot x quot             b StopTimer        if s    strings Repeat  quot x quot   b N   strBuilder String      s           b Errorf  quot unexpected result  got  s  want  s quot   strBuilder String    s

User · Answer

benchmark result with memory allocation statistics  check benchmark code at github   use strings Builder to optimize performance   go test -bench   -benchmem goos  darwin goarch  amd64 pkg  github com hechen0 goexp exps BenchmarkConcat-8                1000000             60213 ns op          503992 B op          1 allocs op BenchmarkBuffer-8               100000000               11 3 ns op             2 B op          0 allocs op BenchmarkCopy-8                 300000000                4 76 ns op            0 B op          0 allocs op BenchmarkStringBuilder-8        1000000000               4 14 ns op            6 B op          0 allocs op PASS ok      github com hechen0 goexp exps   70 071s

User · Answer

In Go 1 10  there is strings Builder  here   A Builder is used to efficiently build a string using Write methods  It minimizes memory copying  The zero value is ready to use    Example It s almost the same with bytes Buffer  package main  import        quot strings quot       quot fmt quot     func main            ZERO-VALUE                It s ready to use from the get-go         You don t need to initialize it      var sb strings Builder      for i    0  i  lt  1000  i             sb WriteString  quot a quot              fmt Println sb String       Click to see this on the playground   Supported Interfaces StringBuilder s methods are being implemented with the existing interfaces in mind  So that you can switch to the new Builder type easily in your code   Grow int  - gt  bytes Buffer Grow Len   int - gt  bytes Buffer Len Reset   - gt  bytes Buffer Reset String   string - gt  fmt Stringer Write   byte   int  error  - gt  io Writer WriteByte byte  error - gt  io ByteWriter WriteRune rune   int  error  - gt  bufio Writer WriteRune - bytes Buffer WriteRune WriteString string   int  error  - gt  io stringWriter   Differences from bytes Buffer  It can only grow or reset   It has a copyCheck mechanism built-in that prevents accidentially copying it  func  b  Builder  copyCheck            In bytes Buffer  one can access the underlying bytes like this    Buffer  Bytes     strings Builder prevents this problem  Sometimes  this is not a problem though and desired instead  For example  For the peeking behavior when the bytes are passed to an io Reader etc    bytes Buffer Reset   rewinds and reuses the underlying buffer whereas the strings Builder Reset   does not  it detaches the buffer     Note  Do not copy a StringBuilder value as it caches the underlying data  If you want to share a StringBuilder value  use a pointer to it    Check out its source code for more details  here

User · Answer

My original suggestion was  s12    fmt Sprint s1 s2    But above answer using bytes Buffer - WriteString   is the most efficient way   My initial suggestion uses reflection and a type switch  See  p  pp  doPrint and  p  pp  printArg There is no universal Stringer   interface for basic types  as I had naively thought   At least though  Sprint   internally uses a bytes Buffer  Thus   s12    fmt Sprint s1 s2 s3 s4     s1000     is acceptable in terms of memory allocations      Sprint   concatenation can be used for quick debug output     Otherwise use bytes Buffer     WriteString

User · Answer

s    fmt Sprintf   s s     byte s1     byte s2

User · Answer

strings Join   from the  strings  package  If you have a type mismatch like if you are trying to join an int and a string   you do RANDOMTYPE  thing you want to change   EX   package main  import        fmt       strings     var intEX   0 var stringEX    hello all you   var stringEX2    people in here    func main         s      string stringEX  stringEX2      fmt Println strings Join s           Output    hello all you people in here

User · Answer

This is actual version of benchmark provided by  cd1  Go 1 8  linux x86 64  with the fixes of bugs mentioned by  icza and  PickBoy   Bytes Buffer is only 7 times faster than direct string concatenation via   operator   package performance test  import        bytes       fmt       testing     const       concatSteps   100    func BenchmarkConcat b  testing B        for n    0  n  lt  b N  n             var str string         for i    0  i  lt  concatSteps  i                 str     x                     func BenchmarkBuffer b  testing B        for n    0  n  lt  b N  n             var buffer bytes Buffer         for i    0  i  lt  concatSteps  i                 buffer WriteString  x                       Timings   BenchmarkConcat-4                             300000          6869 ns op BenchmarkBuffer-4                            1000000          1186 ns op

User · Answer

Note added in 2018  From Go 1 10 there is a strings Builder type  please take a look at this answer for more detail   Pre-201x answer  The benchmark code of  cd1 and other answers are wrong  b N is not supposed to be set in benchmark function  It s set by the go test tool dynamically to determine if the execution time of the test is stable   A benchmark function should run the same test b N times and the test inside the loop should be the same for each iteration  So I fix it by adding an inner loop  I also add benchmarks for some other solutions   package main  import        bytes       strings       testing     const       sss    xfoasneobfasieongasbg      cnt   10000    var       bbb          byte sss      expected   strings Repeat sss  cnt     func BenchmarkCopyPreAllocate b  testing B        var result string     for n    0  n  lt  b N  n             bs    make   byte  cnt len sss           bl    0         for i    0  i  lt  cnt  i                 bl    copy bs bl    sss                    result   string bs            b StopTimer       if result    expected           b Errorf  unexpected result  got  s  want  s   string result   expected           func BenchmarkAppendPreAllocate b  testing B        var result string     for n    0  n  lt  b N  n             data    make   byte  0  cnt len sss           for i    0  i  lt  cnt  i                 data   append data  sss                       result   string data            b StopTimer       if result    expected           b Errorf  unexpected result  got  s  want  s   string result   expected           func BenchmarkBufferPreAllocate b  testing B        var result string     for n    0  n  lt  b N  n             buf    bytes NewBuffer make   byte  0  cnt len sss            for i    0  i  lt  cnt  i                 buf WriteString sss                    result   buf String             b StopTimer       if result    expected           b Errorf  unexpected result  got  s  want  s   string result   expected           func BenchmarkCopy b  testing B        var result string     for n    0  n  lt  b N  n             data    make   byte  0  64     same size as bootstrap array of bytes Buffer         for i    0  i  lt  cnt  i                 off    len data              if off len sss   gt  cap data                    temp    make   byte  2 cap data  len sss                   copy temp  data                  data   temp                           data   data 0   off len sss               copy data off    sss                    result   string data            b StopTimer       if result    expected           b Errorf  unexpected result  got  s  want  s   string result   expected           func BenchmarkAppend b  testing B        var result string     for n    0  n  lt  b N  n             data    make   byte  0  64          for i    0  i  lt  cnt  i                 data   append data  sss                       result   string data            b StopTimer       if result    expected           b Errorf  unexpected result  got  s  want  s   string result   expected           func BenchmarkBufferWrite b  testing B        var result string     for n    0  n  lt  b N  n             var buf bytes Buffer         for i    0  i  lt  cnt  i                 buf Write bbb                    result   buf String             b StopTimer       if result    expected           b Errorf  unexpected result  got  s  want  s   string result   expected           func BenchmarkBufferWriteString b  testing B        var result string     for n    0  n  lt  b N  n             var buf bytes Buffer         for i    0  i  lt  cnt  i                 buf WriteString sss                    result   buf String             b StopTimer       if result    expected           b Errorf  unexpected result  got  s  want  s   string result   expected           func BenchmarkConcat b  testing B        var result string     for n    0  n  lt  b N  n             var str string         for i    0  i  lt  cnt  i                 str    sss                   result   str           b StopTimer       if result    expected           b Errorf  unexpected result  got  s  want  s   string result   expected            Environment is OS X 10 11 6  2 2 GHz Intel Core i7  Test results   BenchmarkCopyPreAllocate-8         20000             84208 ns op          425984 B op          2 allocs op BenchmarkAppendPreAllocate-8       10000            102859 ns op          425984 B op          2 allocs op BenchmarkBufferPreAllocate-8       10000            166407 ns op          426096 B op          3 allocs op BenchmarkCopy-8                    10000            160923 ns op          933152 B op         13 allocs op BenchmarkAppend-8                  10000            175508 ns op         1332096 B op         24 allocs op BenchmarkBufferWrite-8             10000            239886 ns op          933266 B op         14 allocs op BenchmarkBufferWriteString-8       10000            236432 ns op          933266 B op         14 allocs op BenchmarkConcat-8                     10         105603419 ns op        1086685168 B op    10000 allocs op   Conclusion    CopyPreAllocate is the fastest way  AppendPreAllocate is pretty close to No 1  but it s easier to write the code  Concat has really bad performance both for speed and memory usage  Don t use it  Buffer Write and Buffer WriteString are basically the same in speed  contrary to what  Dani-Br said in the comment  Considering string is indeed   byte in Go  it makes sense  bytes Buffer basically use the same solution as Copy with extra book keeping and other stuff  Copy and Append use a bootstrap size of 64  the same as bytes Buffer Append use more memory and allocs  I think it s related to the grow algorithm it use  It s not growing memory as fast as bytes Buffer   Suggestion    For simple task such as what OP wants  I would use Append or AppendPreAllocate  It s fast enough and easy to use  If need to read and write the buffer at the same time  use bytes Buffer of course  That s what it s designed for

[string] How to efficiently concatenate strings in go

Examples related to string

Examples related to go

Examples related to string-concatenation