[c#] Insert entire DataTable into database at once instead of row by row?

I have a DataTable and need the entire thing pushed to a Database table.

I can get it all in there with a foreach and inserting each row at a time. This goes very slow though since there are a few thousand rows.

Is there any way to do the entire datatable at once that might be faster?

The DataTable has less columns than the SQL table. the rest of them should be left NULL.

This question is related to c# sql sql-server

The answer is


Since you have a DataTable already, and since I am assuming you are using SQL Server 2008 or better, this is probably the most straightforward way. First, in your database, create the following two objects:

CREATE TYPE dbo.MyDataTable -- you can be more speciifc here
AS TABLE
(
  col1 INT,
  col2 DATETIME
  -- etc etc. The columns you have in your data table.
);
GO

CREATE PROCEDURE dbo.InsertMyDataTable
  @dt AS dbo.MyDataTable READONLY
AS
BEGIN
  SET NOCOUNT ON;

  INSERT dbo.RealTable(column list) SELECT column list FROM @dt;
END
GO

Now in your C# code:

DataTable tvp = new DataTable();
// define / populate DataTable

using (connectionObject)
{
    SqlCommand cmd = new SqlCommand("dbo.InsertMyDataTable", connectionObject);
    cmd.CommandType = CommandType.StoredProcedure;
    SqlParameter tvparam = cmd.Parameters.AddWithValue("@dt", tvp);
    tvparam.SqlDbType = SqlDbType.Structured;
    cmd.ExecuteNonQuery();
}

If you had given more specific details in your question, I would have given a more specific answer.


Consider this approach, you don't need a for loop:

using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
    bulkCopy.DestinationTableName = 
        "dbo.BulkCopyDemoMatchingColumns";

    try
    {
        // Write from the source to the destination.
        bulkCopy.WriteToServer(ExitingSqlTableName);
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
    }
}

I would prefer user defined data type : it is super fast.

Step 1 : Create User Defined Table in Sql Server DB

CREATE TYPE [dbo].[udtProduct] AS TABLE(
  [ProductID] [int] NULL,
  [ProductName] [varchar](50) NULL,
  [ProductCode] [varchar](10) NULL
)
GO

Step 2 : Create Stored Procedure with User Defined Type

CREATE PROCEDURE ProductBulkInsertion 
@product udtProduct readonly
AS
BEGIN
    INSERT INTO Product
    (ProductID,ProductName,ProductCode)
    SELECT ProductID,ProductName,ProductCode
    FROM @product
END

Step 3 : Execute Stored Procedure from c#

SqlCommand sqlcmd = new SqlCommand("ProductBulkInsertion", sqlcon);
sqlcmd.CommandType = CommandType.StoredProcedure;
sqlcmd.Parameters.AddWithValue("@product", productTable);
sqlcmd.ExecuteNonQuery();

Possible Issue : Alter User Defined Table

Actually there is no sql server command to alter user defined type But in management studio you can achieve this from following steps

1.generate script for the type.(in new query window or as a file) 2.delete user defied table. 3.modify the create script and then execute.


You can do this with a table value parameters.

Have a look at the following article:

http://www.codeproject.com/Articles/39161/C-and-Table-Value-Parameters


If can deviate a little from the straight path of DataTable -> SQL table, it can also be done via a list of objects:

1) DataTable -> Generic list of objects

public static DataTable ConvertTo<T>(IList<T> list)
{
    DataTable table = CreateTable<T>();
    Type entityType = typeof(T);
    PropertyDescriptorCollection properties = TypeDescriptor.GetProperties(entityType);

    foreach (T item in list)
    {
        DataRow row = table.NewRow();

        foreach (PropertyDescriptor prop in properties)
        {
            row[prop.Name] = prop.GetValue(item);
        }

        table.Rows.Add(row);
    }

    return table;
}

Source and more details can be found here. Missing properties will remain to their default values (0 for ints, null for reference types etc.)

2) Push the objects into the database

One way is to use EntityFramework.BulkInsert extension. An EF datacontext is required, though.

It generates the BULK INSERT command required for fast insert (user defined table type solution is much slower than this).

Although not the straight method, it helps constructing a base of working with list of objects instead of DataTables which seems to be much more memory efficient.


Examples related to c#

How can I convert this one line of ActionScript to C#? Microsoft Advertising SDK doesn't deliverer ads How to use a global array in C#? How to correctly write async method? C# - insert values from file into two arrays Uploading into folder in FTP? Are these methods thread safe? dotnet ef not found in .NET Core 3 HTTP Error 500.30 - ANCM In-Process Start Failure Best way to "push" into C# array

Examples related to sql

Passing multiple values for same variable in stored procedure SQL permissions for roles Generic XSLT Search and Replace template Access And/Or exclusions Pyspark: Filter dataframe based on multiple conditions Subtracting 1 day from a timestamp date PYODBC--Data source name not found and no default driver specified select rows in sql with latest date for each ID repeated multiple times ALTER TABLE DROP COLUMN failed because one or more objects access this column Create Local SQL Server database

Examples related to sql-server

Passing multiple values for same variable in stored procedure SQL permissions for roles Count the Number of Tables in a SQL Server Database Visual Studio 2017 does not have Business Intelligence Integration Services/Projects ALTER TABLE DROP COLUMN failed because one or more objects access this column Create Local SQL Server database How to create temp table using Create statement in SQL Server? SQL Query Where Date = Today Minus 7 Days How do I pass a list as a parameter in a stored procedure? SQL Server date format yyyymmdd