Most efficient way to append arrays in C

Question

I am pulling data out of an old-school ActiveX in the form of arrays of doubles  I don t initially know the final number of samples I will actually retrieve   What is the most efficient way to concatenate these arrays together in C  as I pull them  out of the system

User · Answer

Concatenating arrays is simple using linq extensions which come standard with .Net 4

Biggest thing to remember is that linq works with IEnumerable<T> objects, so in order to get an array back as your result then you must use the .ToArray() method at the end

Example of concatenating two byte arrays:

byte[] firstArray = {2,45,79,33};
byte[] secondArray = {55,4,7,81};
byte[] result = firstArray.Concat(secondArray).ToArray();

User · Answer

You might not need to concatenate end result into contiguous array  Instead  keep appending to the list as suggested by Jon  In the end you ll have a jagged array  well  almost rectangular in fact   When you need to access an element by index  use following indexing scheme   double x   list i   sampleSize  i   sampleSize     Iteration over jagged array is also straightforward   for  int iRow   0  iRow  lt  list Length    iRow      double   row   list iRow     for  int iCol   0  iCol  lt  row Length    iCol        double x   row iCol           This saves you memory allocation and copying at expense of slightly slower element access  Whether this will be a net performance gain depends on size of your data  data access patterns and memory constraints

User · Answer

If you can make an approximation of the number of items that will be there at the end  use the overload of the List constuctor that takes count as a parameter  You will save some expensive List duplications  Otherwise you have to pay for it

User · Answer

Olmo s suggestion is very good  but I d add this  If you re not sure about the size  it s better to make it a little bigger than a little smaller  When a list is full  keep in mind it will double its size to add more elements    For example  suppose you will need about 50 elements  If you use a 50 elements size and the final number of elements is 51  you ll end with a 100 sized list with 49 wasted positions

User · Answer

using this we can add two array with out any loop    I believe if you have 2 arrays of the same type that you want to combine into one of array  there s a very simple way to do that   Here s the code   String   TextFils   Directory GetFiles basePath     txt    String   ExcelFils   Directory GetFiles basePath     xls    String   finalArray   TextFils Concat ExcelFils  ToArray      or  String   Fils   Directory GetFiles basePath     txt    String   ExcelFils   Directory GetFiles basePath     xls    Fils   Fils Concat ExcelFils  ToArray

User · Answer

The solution looks like great fun  but it is possible to concatenate arrays in just two statements  When you re handling large byte arrays  I suppose it is inefficient to use a Linked List to contain each byte   Here is a code sample for reading bytes from a stream and extending a byte array on the fly        byte   buf   new byte 8192       byte   result   new byte 0       int count   0      do               count   resStream Read buf  0  buf Length           if  count    0                        Array Resize ref result  result Length   count               Array Copy buf  0  result  result Length - count  count                       while  count   0      any more data to read      resStream Close

User · Answer

Here is a usable class based on what Constantin said   class Program       static void Main string   args                FastConcat lt int gt  i   new FastConcat lt int gt             i Add new int     0  1  2  3  4             Console WriteLine i 0            i Add new int     5  6  7  8  9             Console WriteLine i 4             Console WriteLine  Enumerator             foreach  int val in i              Console WriteLine val            Console ReadLine             class FastConcat lt T gt    IEnumerable lt T gt        LinkedList lt T   gt   items   new LinkedList lt T   gt         int  count       public int Count               get                       return  count                       public void Add T   items                if  items    null              return          if  items Length    0              return            items AddLast items            count    items Length             private T   GetItemIndex int realIndex  out int offset                offset   0     Offset that needs to be applied to realIndex          int currentStart   0     Current index start           foreach  T   items in  items                        currentStart    items Length              if  currentStart  gt  realIndex                  return items              offset   currentStart                    return null             public T this int index                get                       int offset              T   i   GetItemIndex index  out offset               return i index - offset                     set                       int offset              T   i   GetItemIndex index  out offset               i index - offset    value                        region IEnumerable lt T gt  Members      public IEnumerator lt T gt  GetEnumerator                 foreach  T   items in  items              foreach  T item in items                  yield return item              endregion       region IEnumerable Members      System Collections IEnumerator System Collections IEnumerable GetEnumerator                 return GetEnumerator                endregion

User · Answer

using this we can add two array with out any loop    I believe if you have 2 arrays of the same type that you want to combine into one of array  there s a very simple way to do that   Here s the code   String   TextFils   Directory GetFiles basePath     txt    String   ExcelFils   Directory GetFiles basePath     xls    String   finalArray   TextFils Concat ExcelFils  ToArray      or  String   Fils   Directory GetFiles basePath     txt    String   ExcelFils   Directory GetFiles basePath     xls    Fils   Fils Concat ExcelFils  ToArray

User · Answer

Here is a usable class based on what Constantin said   class Program       static void Main string   args                FastConcat lt int gt  i   new FastConcat lt int gt             i Add new int     0  1  2  3  4             Console WriteLine i 0            i Add new int     5  6  7  8  9             Console WriteLine i 4             Console WriteLine  Enumerator             foreach  int val in i              Console WriteLine val            Console ReadLine             class FastConcat lt T gt    IEnumerable lt T gt        LinkedList lt T   gt   items   new LinkedList lt T   gt         int  count       public int Count               get                       return  count                       public void Add T   items                if  items    null              return          if  items Length    0              return            items AddLast items            count    items Length             private T   GetItemIndex int realIndex  out int offset                offset   0     Offset that needs to be applied to realIndex          int currentStart   0     Current index start           foreach  T   items in  items                        currentStart    items Length              if  currentStart  gt  realIndex                  return items              offset   currentStart                    return null             public T this int index                get                       int offset              T   i   GetItemIndex index  out offset               return i index - offset                     set                       int offset              T   i   GetItemIndex index  out offset               i index - offset    value                        region IEnumerable lt T gt  Members      public IEnumerator lt T gt  GetEnumerator                 foreach  T   items in  items              foreach  T item in items                  yield return item              endregion       region IEnumerable Members      System Collections IEnumerator System Collections IEnumerable GetEnumerator                 return GetEnumerator                endregion

User · Answer

You might not need to concatenate end result into contiguous array  Instead  keep appending to the list as suggested by Jon  In the end you ll have a jagged array  well  almost rectangular in fact   When you need to access an element by index  use following indexing scheme   double x   list i   sampleSize  i   sampleSize     Iteration over jagged array is also straightforward   for  int iRow   0  iRow  lt  list Length    iRow      double   row   list iRow     for  int iCol   0  iCol  lt  row Length    iCol        double x   row iCol           This saves you memory allocation and copying at expense of slightly slower element access  Whether this will be a net performance gain depends on size of your data  data access patterns and memory constraints

User · Answer

You might not need to concatenate end result into contiguous array  Instead  keep appending to the list as suggested by Jon  In the end you ll have a jagged array  well  almost rectangular in fact   When you need to access an element by index  use following indexing scheme   double x   list i   sampleSize  i   sampleSize     Iteration over jagged array is also straightforward   for  int iRow   0  iRow  lt  list Length    iRow      double   row   list iRow     for  int iCol   0  iCol  lt  row Length    iCol        double x   row iCol           This saves you memory allocation and copying at expense of slightly slower element access  Whether this will be a net performance gain depends on size of your data  data access patterns and memory constraints

User · Answer

Olmo s suggestion is very good  but I d add this  If you re not sure about the size  it s better to make it a little bigger than a little smaller  When a list is full  keep in mind it will double its size to add more elements    For example  suppose you will need about 50 elements  If you use a 50 elements size and the final number of elements is 51  you ll end with a 100 sized list with 49 wasted positions

User · Answer

The solution looks like great fun  but it is possible to concatenate arrays in just two statements  When you re handling large byte arrays  I suppose it is inefficient to use a Linked List to contain each byte   Here is a code sample for reading bytes from a stream and extending a byte array on the fly        byte   buf   new byte 8192       byte   result   new byte 0       int count   0      do               count   resStream Read buf  0  buf Length           if  count    0                        Array Resize ref result  result Length   count               Array Copy buf  0  result  result Length - count  count                       while  count   0      any more data to read      resStream Close

User · Answer

If you can make an approximation of the number of items that will be there at the end  use the overload of the List constuctor that takes count as a parameter  You will save some expensive List duplications  Otherwise you have to pay for it

User · Answer

You can t append to an actual array - the size of an array is fixed at creation time  Instead  use a List lt T gt  which can grow as it needs to  Alternatively  keep a list of arrays  and concatenate them all only when you ve grabbed everything  See Eric Lippert s blog post on arrays for more detail and insight than I could realistically provide

User · Answer

I recommend the answer found here  How do I concatenate two arrays in C    e g   var z   new int x Length   y Length   x CopyTo z  0   y CopyTo z  x Length

User · Answer

If you can make an approximation of the number of items that will be there at the end  use the overload of the List constuctor that takes count as a parameter  You will save some expensive List duplications  Otherwise you have to pay for it

User · Answer

You can t append to an actual array - the size of an array is fixed at creation time  Instead  use a List lt T gt  which can grow as it needs to  Alternatively  keep a list of arrays  and concatenate them all only when you ve grabbed everything  See Eric Lippert s blog post on arrays for more detail and insight than I could realistically provide

User · Answer

Olmo s suggestion is very good  but I d add this  If you re not sure about the size  it s better to make it a little bigger than a little smaller  When a list is full  keep in mind it will double its size to add more elements    For example  suppose you will need about 50 elements  If you use a 50 elements size and the final number of elements is 51  you ll end with a 100 sized list with 49 wasted positions

User · Answer

You can t append to an actual array - the size of an array is fixed at creation time  Instead  use a List lt T gt  which can grow as it needs to  Alternatively  keep a list of arrays  and concatenate them all only when you ve grabbed everything  See Eric Lippert s blog post on arrays for more detail and insight than I could realistically provide

User · Answer

I recommend the answer found here  How do I concatenate two arrays in C    e g   var z   new int x Length   y Length   x CopyTo z  0   y CopyTo z  x Length

User · Answer

You can t append to an actual array - the size of an array is fixed at creation time  Instead  use a List lt T gt  which can grow as it needs to  Alternatively  keep a list of arrays  and concatenate them all only when you ve grabbed everything  See Eric Lippert s blog post on arrays for more detail and insight than I could realistically provide

User · Answer

You might not need to concatenate end result into contiguous array  Instead  keep appending to the list as suggested by Jon  In the end you ll have a jagged array  well  almost rectangular in fact   When you need to access an element by index  use following indexing scheme   double x   list i   sampleSize  i   sampleSize     Iteration over jagged array is also straightforward   for  int iRow   0  iRow  lt  list Length    iRow      double   row   list iRow     for  int iCol   0  iCol  lt  row Length    iCol        double x   row iCol           This saves you memory allocation and copying at expense of slightly slower element access  Whether this will be a net performance gain depends on size of your data  data access patterns and memory constraints

User · Answer

Olmo s suggestion is very good  but I d add this  If you re not sure about the size  it s better to make it a little bigger than a little smaller  When a list is full  keep in mind it will double its size to add more elements    For example  suppose you will need about 50 elements  If you use a 50 elements size and the final number of elements is 51  you ll end with a 100 sized list with 49 wasted positions

User · Answer

Concatenating arrays is simple using linq extensions which come standard with .Net 4

Biggest thing to remember is that linq works with IEnumerable<T> objects, so in order to get an array back as your result then you must use the .ToArray() method at the end

Example of concatenating two byte arrays:

byte[] firstArray = {2,45,79,33};
byte[] secondArray = {55,4,7,81};
byte[] result = firstArray.Concat(secondArray).ToArray();

User · Answer

I believe if you have 2 arrays of the same type that you want to combine into a third array  there s a very simple way to do that    here s the code   String   theHTMLFiles   Directory GetFiles basePath     html    String   thexmlFiles   Directory GetFiles basePath     xml    List lt String gt  finalList   new List lt String gt  theHTMLFiles Concat lt string gt  thexmlFiles    String   finalArray   finalList ToArray

User · Answer

Here is a usable class based on what Constantin said   class Program       static void Main string   args                FastConcat lt int gt  i   new FastConcat lt int gt             i Add new int     0  1  2  3  4             Console WriteLine i 0            i Add new int     5  6  7  8  9             Console WriteLine i 4             Console WriteLine  Enumerator             foreach  int val in i              Console WriteLine val            Console ReadLine             class FastConcat lt T gt    IEnumerable lt T gt        LinkedList lt T   gt   items   new LinkedList lt T   gt         int  count       public int Count               get                       return  count                       public void Add T   items                if  items    null              return          if  items Length    0              return            items AddLast items            count    items Length             private T   GetItemIndex int realIndex  out int offset                offset   0     Offset that needs to be applied to realIndex          int currentStart   0     Current index start           foreach  T   items in  items                        currentStart    items Length              if  currentStart  gt  realIndex                  return items              offset   currentStart                    return null             public T this int index                get                       int offset              T   i   GetItemIndex index  out offset               return i index - offset                     set                       int offset              T   i   GetItemIndex index  out offset               i index - offset    value                        region IEnumerable lt T gt  Members      public IEnumerator lt T gt  GetEnumerator                 foreach  T   items in  items              foreach  T item in items                  yield return item              endregion       region IEnumerable Members      System Collections IEnumerator System Collections IEnumerable GetEnumerator                 return GetEnumerator                endregion

User · Answer

I believe if you have 2 arrays of the same type that you want to combine into a third array  there s a very simple way to do that    here s the code   String   theHTMLFiles   Directory GetFiles basePath     html    String   thexmlFiles   Directory GetFiles basePath     xml    List lt String gt  finalList   new List lt String gt  theHTMLFiles Concat lt string gt  thexmlFiles    String   finalArray   finalList ToArray

User · Answer

Here is a usable class based on what Constantin said   class Program       static void Main string   args                FastConcat lt int gt  i   new FastConcat lt int gt             i Add new int     0  1  2  3  4             Console WriteLine i 0            i Add new int     5  6  7  8  9             Console WriteLine i 4             Console WriteLine  Enumerator             foreach  int val in i              Console WriteLine val            Console ReadLine             class FastConcat lt T gt    IEnumerable lt T gt        LinkedList lt T   gt   items   new LinkedList lt T   gt         int  count       public int Count               get                       return  count                       public void Add T   items                if  items    null              return          if  items Length    0              return            items AddLast items            count    items Length             private T   GetItemIndex int realIndex  out int offset                offset   0     Offset that needs to be applied to realIndex          int currentStart   0     Current index start           foreach  T   items in  items                        currentStart    items Length              if  currentStart  gt  realIndex                  return items              offset   currentStart                    return null             public T this int index                get                       int offset              T   i   GetItemIndex index  out offset               return i index - offset                     set                       int offset              T   i   GetItemIndex index  out offset               i index - offset    value                        region IEnumerable lt T gt  Members      public IEnumerator lt T gt  GetEnumerator                 foreach  T   items in  items              foreach  T item in items                  yield return item              endregion       region IEnumerable Members      System Collections IEnumerator System Collections IEnumerable GetEnumerator                 return GetEnumerator                endregion

User · Answer

If you can make an approximation of the number of items that will be there at the end  use the overload of the List constuctor that takes count as a parameter  You will save some expensive List duplications  Otherwise you have to pay for it

[c#] Most efficient way to append arrays in C#?

Examples related to c#

Examples related to arrays

Examples related to memory-management