I am pulling data out of an old-school ActiveX in the form of arrays of doubles. I don't initially know the final number of samples I will actually retrieve.
What is the most efficient way to concatenate these arrays together in C# as I pull them out of the system?
This question is related to
c#
arrays
memory-management
You can't append to an actual array - the size of an array is fixed at creation time. Instead, use a List<T> which can grow as it needs to.
Alternatively, keep a list of arrays, and concatenate them all only when you've grabbed everything.
See Eric Lippert's blog post on arrays for more detail and insight than I could realistically provide :)
using this we can add two array with out any loop.
I believe if you have 2 arrays of the same type that you want to combine into one of array, there's a very simple way to do that.
Here's the code:
String[] TextFils = Directory.GetFiles(basePath, "*.txt");
String[] ExcelFils = Directory.GetFiles(basePath, "*.xls");
String[] finalArray = TextFils.Concat(ExcelFils).ToArray();
or
String[] Fils = Directory.GetFiles(basePath, "*.txt");
String[] ExcelFils = Directory.GetFiles(basePath, "*.xls");
Fils = Fils.Concat(ExcelFils).ToArray();
Olmo's suggestion is very good, but I'd add this: If you're not sure about the size, it's better to make it a little bigger than a little smaller. When a list is full, keep in mind it will double its size to add more elements.
For example: suppose you will need about 50 elements. If you use a 50 elements size and the final number of elements is 51, you'll end with a 100 sized list with 49 wasted positions.
Concatenating arrays is simple using linq extensions which come standard with .Net 4
Biggest thing to remember is that linq works with IEnumerable<T> objects, so in order to get an array back as your result then you must use the .ToArray() method at the end
Example of concatenating two byte arrays:
byte[] firstArray = {2,45,79,33};
byte[] secondArray = {55,4,7,81};
byte[] result = firstArray.Concat(secondArray).ToArray();
You might not need to concatenate end result into contiguous array. Instead, keep appending to the list as suggested by Jon. In the end you'll have a jagged array (well, almost rectangular in fact). When you need to access an element by index, use following indexing scheme:
double x = list[i / sampleSize][i % sampleSize];
Iteration over jagged array is also straightforward:
for (int iRow = 0; iRow < list.Length; ++iRow) {
double[] row = list[iRow];
for (int iCol = 0; iCol < row.Length; ++iCol) {
double x = row[iCol];
}
}
This saves you memory allocation and copying at expense of slightly slower element access. Whether this will be a net performance gain depends on size of your data, data access patterns and memory constraints.
Here is a usable class based on what Constantin said:
class Program
{
static void Main(string[] args)
{
FastConcat<int> i = new FastConcat<int>();
i.Add(new int[] { 0, 1, 2, 3, 4 });
Console.WriteLine(i[0]);
i.Add(new int[] { 5, 6, 7, 8, 9 });
Console.WriteLine(i[4]);
Console.WriteLine("Enumerator:");
foreach (int val in i)
Console.WriteLine(val);
Console.ReadLine();
}
}
class FastConcat<T> : IEnumerable<T>
{
LinkedList<T[]> _items = new LinkedList<T[]>();
int _count;
public int Count
{
get
{
return _count;
}
}
public void Add(T[] items)
{
if (items == null)
return;
if (items.Length == 0)
return;
_items.AddLast(items);
_count += items.Length;
}
private T[] GetItemIndex(int realIndex, out int offset)
{
offset = 0; // Offset that needs to be applied to realIndex.
int currentStart = 0; // Current index start.
foreach (T[] items in _items)
{
currentStart += items.Length;
if (currentStart > realIndex)
return items;
offset = currentStart;
}
return null;
}
public T this[int index]
{
get
{
int offset;
T[] i = GetItemIndex(index, out offset);
return i[index - offset];
}
set
{
int offset;
T[] i = GetItemIndex(index, out offset);
i[index - offset] = value;
}
}
#region IEnumerable<T> Members
public IEnumerator<T> GetEnumerator()
{
foreach (T[] items in _items)
foreach (T item in items)
yield return item;
}
#endregion
#region IEnumerable Members
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
#endregion
}
Olmo's suggestion is very good, but I'd add this: If you're not sure about the size, it's better to make it a little bigger than a little smaller. When a list is full, keep in mind it will double its size to add more elements.
For example: suppose you will need about 50 elements. If you use a 50 elements size and the final number of elements is 51, you'll end with a 100 sized list with 49 wasted positions.
If you can make an approximation of the number of items that will be there at the end, use the overload of the List constuctor that takes count as a parameter. You will save some expensive List duplications. Otherwise you have to pay for it.
Here is a usable class based on what Constantin said:
class Program
{
static void Main(string[] args)
{
FastConcat<int> i = new FastConcat<int>();
i.Add(new int[] { 0, 1, 2, 3, 4 });
Console.WriteLine(i[0]);
i.Add(new int[] { 5, 6, 7, 8, 9 });
Console.WriteLine(i[4]);
Console.WriteLine("Enumerator:");
foreach (int val in i)
Console.WriteLine(val);
Console.ReadLine();
}
}
class FastConcat<T> : IEnumerable<T>
{
LinkedList<T[]> _items = new LinkedList<T[]>();
int _count;
public int Count
{
get
{
return _count;
}
}
public void Add(T[] items)
{
if (items == null)
return;
if (items.Length == 0)
return;
_items.AddLast(items);
_count += items.Length;
}
private T[] GetItemIndex(int realIndex, out int offset)
{
offset = 0; // Offset that needs to be applied to realIndex.
int currentStart = 0; // Current index start.
foreach (T[] items in _items)
{
currentStart += items.Length;
if (currentStart > realIndex)
return items;
offset = currentStart;
}
return null;
}
public T this[int index]
{
get
{
int offset;
T[] i = GetItemIndex(index, out offset);
return i[index - offset];
}
set
{
int offset;
T[] i = GetItemIndex(index, out offset);
i[index - offset] = value;
}
}
#region IEnumerable<T> Members
public IEnumerator<T> GetEnumerator()
{
foreach (T[] items in _items)
foreach (T item in items)
yield return item;
}
#endregion
#region IEnumerable Members
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
#endregion
}
Concatenating arrays is simple using linq extensions which come standard with .Net 4
Biggest thing to remember is that linq works with IEnumerable<T> objects, so in order to get an array back as your result then you must use the .ToArray() method at the end
Example of concatenating two byte arrays:
byte[] firstArray = {2,45,79,33};
byte[] secondArray = {55,4,7,81};
byte[] result = firstArray.Concat(secondArray).ToArray();
You might not need to concatenate end result into contiguous array. Instead, keep appending to the list as suggested by Jon. In the end you'll have a jagged array (well, almost rectangular in fact). When you need to access an element by index, use following indexing scheme:
double x = list[i / sampleSize][i % sampleSize];
Iteration over jagged array is also straightforward:
for (int iRow = 0; iRow < list.Length; ++iRow) {
double[] row = list[iRow];
for (int iCol = 0; iCol < row.Length; ++iCol) {
double x = row[iCol];
}
}
This saves you memory allocation and copying at expense of slightly slower element access. Whether this will be a net performance gain depends on size of your data, data access patterns and memory constraints.
using this we can add two array with out any loop.
I believe if you have 2 arrays of the same type that you want to combine into one of array, there's a very simple way to do that.
Here's the code:
String[] TextFils = Directory.GetFiles(basePath, "*.txt");
String[] ExcelFils = Directory.GetFiles(basePath, "*.xls");
String[] finalArray = TextFils.Concat(ExcelFils).ToArray();
or
String[] Fils = Directory.GetFiles(basePath, "*.txt");
String[] ExcelFils = Directory.GetFiles(basePath, "*.xls");
Fils = Fils.Concat(ExcelFils).ToArray();
If you can make an approximation of the number of items that will be there at the end, use the overload of the List constuctor that takes count as a parameter. You will save some expensive List duplications. Otherwise you have to pay for it.
Olmo's suggestion is very good, but I'd add this: If you're not sure about the size, it's better to make it a little bigger than a little smaller. When a list is full, keep in mind it will double its size to add more elements.
For example: suppose you will need about 50 elements. If you use a 50 elements size and the final number of elements is 51, you'll end with a 100 sized list with 49 wasted positions.
I recommend the answer found here: How do I concatenate two arrays in C#?
e.g.
var z = new int[x.Length + y.Length];
x.CopyTo(z, 0);
y.CopyTo(z, x.Length);
The solution looks like great fun, but it is possible to concatenate arrays in just two statements. When you're handling large byte arrays, I suppose it is inefficient to use a Linked List to contain each byte.
Here is a code sample for reading bytes from a stream and extending a byte array on the fly:
byte[] buf = new byte[8192];
byte[] result = new byte[0];
int count = 0;
do
{
count = resStream.Read(buf, 0, buf.Length);
if (count != 0)
{
Array.Resize(ref result, result.Length + count);
Array.Copy(buf, 0, result, result.Length - count, count);
}
}
while (count > 0); // any more data to read?
resStream.Close();
I believe if you have 2 arrays of the same type that you want to combine into a third array, there's a very simple way to do that.
here's the code:
String[] theHTMLFiles = Directory.GetFiles(basePath, "*.html");
String[] thexmlFiles = Directory.GetFiles(basePath, "*.xml");
List<String> finalList = new List<String>(theHTMLFiles.Concat<string>(thexmlFiles));
String[] finalArray = finalList.ToArray();
I recommend the answer found here: How do I concatenate two arrays in C#?
e.g.
var z = new int[x.Length + y.Length];
x.CopyTo(z, 0);
y.CopyTo(z, x.Length);
Here is a usable class based on what Constantin said:
class Program
{
static void Main(string[] args)
{
FastConcat<int> i = new FastConcat<int>();
i.Add(new int[] { 0, 1, 2, 3, 4 });
Console.WriteLine(i[0]);
i.Add(new int[] { 5, 6, 7, 8, 9 });
Console.WriteLine(i[4]);
Console.WriteLine("Enumerator:");
foreach (int val in i)
Console.WriteLine(val);
Console.ReadLine();
}
}
class FastConcat<T> : IEnumerable<T>
{
LinkedList<T[]> _items = new LinkedList<T[]>();
int _count;
public int Count
{
get
{
return _count;
}
}
public void Add(T[] items)
{
if (items == null)
return;
if (items.Length == 0)
return;
_items.AddLast(items);
_count += items.Length;
}
private T[] GetItemIndex(int realIndex, out int offset)
{
offset = 0; // Offset that needs to be applied to realIndex.
int currentStart = 0; // Current index start.
foreach (T[] items in _items)
{
currentStart += items.Length;
if (currentStart > realIndex)
return items;
offset = currentStart;
}
return null;
}
public T this[int index]
{
get
{
int offset;
T[] i = GetItemIndex(index, out offset);
return i[index - offset];
}
set
{
int offset;
T[] i = GetItemIndex(index, out offset);
i[index - offset] = value;
}
}
#region IEnumerable<T> Members
public IEnumerator<T> GetEnumerator()
{
foreach (T[] items in _items)
foreach (T item in items)
yield return item;
}
#endregion
#region IEnumerable Members
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
#endregion
}
Olmo's suggestion is very good, but I'd add this: If you're not sure about the size, it's better to make it a little bigger than a little smaller. When a list is full, keep in mind it will double its size to add more elements.
For example: suppose you will need about 50 elements. If you use a 50 elements size and the final number of elements is 51, you'll end with a 100 sized list with 49 wasted positions.
You might not need to concatenate end result into contiguous array. Instead, keep appending to the list as suggested by Jon. In the end you'll have a jagged array (well, almost rectangular in fact). When you need to access an element by index, use following indexing scheme:
double x = list[i / sampleSize][i % sampleSize];
Iteration over jagged array is also straightforward:
for (int iRow = 0; iRow < list.Length; ++iRow) {
double[] row = list[iRow];
for (int iCol = 0; iCol < row.Length; ++iCol) {
double x = row[iCol];
}
}
This saves you memory allocation and copying at expense of slightly slower element access. Whether this will be a net performance gain depends on size of your data, data access patterns and memory constraints.
The solution looks like great fun, but it is possible to concatenate arrays in just two statements. When you're handling large byte arrays, I suppose it is inefficient to use a Linked List to contain each byte.
Here is a code sample for reading bytes from a stream and extending a byte array on the fly:
byte[] buf = new byte[8192];
byte[] result = new byte[0];
int count = 0;
do
{
count = resStream.Read(buf, 0, buf.Length);
if (count != 0)
{
Array.Resize(ref result, result.Length + count);
Array.Copy(buf, 0, result, result.Length - count, count);
}
}
while (count > 0); // any more data to read?
resStream.Close();
Source: Stackoverflow.com