Few weeks ago I was working with a small internal project that involves importing CSV file to Sql
Server database and thought I'd share the simple implementation that I did on the project.
In this post I will demonstrate how to upload and import CSV file to SQL
Server database. As some may have already know, importing CSV file to SQL
Server is easy and simple but difficulties arise when the CSV file contains, many columns with different data types. Basically, the provider cannot differentiate data types between the columns or the rows, blindly it will consider them as a data type based on first few rows and leave all the data which does not match the data type. To overcome this problem, I used schema.ini file to define the data type of the CSV file and allow the provider to read that and recognize the exact data types of each column.
Now what is schema.ini?
Taken from the documentation: The Schema.ini is a information file, used to define the data structure and format of each column that contains data in the CSV file. If schema.ini file exists in the directory, Microsoft.Jet.OLEDB provider automatically reads it and recognizes the data type information of each column in the CSV file. Thus, the provider intelligently avoids the misinterpretation of data types before inserting the data into the database. For more information see: http://msdn.microsoft.com/en-us/library/ms709353%28VS.85%29.aspx
Points to remember before creating schema.ini:
1. The schema information file, must always named as 'schema.ini'.
2. The schema.ini file must be kept in the same directory where the CSV file exists.
3. The schema.ini file must be created before reading the CSV file.
4. The first line of the schema.ini, must the name of the CSV file, followed by the properties of the CSV file, and then the properties of the each column in the CSV file.
Here's an example of how the schema looked like:
[Employee.csv]
ColNameHeader=False
Format=CSVDelimited
DateTimeFormat=dd-MMM-yyyy
Col1=EmployeeID Long
Col2=EmployeeFirstName Text Width 100
Col3=EmployeeLastName Text Width 50
Col4=EmployeeEmailAddress Text Width 50
To get started lets's go a head and create a simple blank database. Just for the purpose of this demo I created a database called TestDB.
After creating the database then lets go a head and fire up Visual Studio and then create a new WebApplication project.
Under the root application create a folder called UploadedCSVFiles and then place the schema.ini on that folder. The uploaded CSV files will be stored in this folder after the user imports the file.
Now add a WebForm in the project and set up the HTML mark up and add one (1) FileUpload control one(1)Button and three (3) Label controls.
After that we can now proceed with the codes for uploading and importing the CSV file to SQL
Server database. Here are the full code blocks below:
1: using System;
2: using System.Data;
3: using System.Data.SqlClient;
4: using System.Data.OleDb;
5: using System.IO;
6: using System.Text;
7:
8: namespace WebApplication1
9: {
10: public partial class CSVToSQLImporting : System.Web.UI.Page
11: {
12: private string GetConnectionString()
13: {
14: return System.Configuration.ConfigurationManager.ConnectionStrings["DBConnectionString"].ConnectionString;
15: }
16: private void CreateDatabaseTable(DataTable dt, string tableName)
17: {
18:
19: string sqlQuery = string.Empty;
20: string sqlDBType = string.Empty;
21: string dataType = string.Empty;
22: int maxLength = 0;
23: StringBuilder sb = new StringBuilder();
24:
25: sb.AppendFormat(string.Format("CREATE TABLE {0} (", tableName));
26:
27: for (int i = 0; i < dt.Columns.Count; i++)
28: {
29: dataType = dt.Columns[i].DataType.ToString();
30: if (dataType == "System.Int32")
31: {
32: sqlDBType = "INT";
33: }
34: else if (dataType == "System.String")
35: {
36: sqlDBType = "NVARCHAR";
37: maxLength = dt.Columns[i].MaxLength;
38: }
39:
40: if (maxLength > 0)
41: {
42: sb.AppendFormat(string.Format(" {0} {1} ({2}), ", dt.Columns[i].ColumnName, sqlDBType, maxLength));
43: }
44: else
45: {
46: sb.AppendFormat(string.Format(" {0} {1}, ", dt.Columns[i].ColumnName, sqlDBType));
47: }
48: }
49:
50: sqlQuery = sb.ToString();
51: sqlQuery = sqlQuery.Trim().TrimEnd(',');
52: sqlQuery = sqlQuery + " )";
53:
54: using (SqlConnection sqlConn = new SqlConnection(GetConnectionString()))
55: {
56: sqlConn.Open();
57: SqlCommand sqlCmd = new SqlCommand(sqlQuery, sqlConn);
58: sqlCmd.ExecuteNonQuery();
59: sqlConn.Close();
60: }
61:
62: }
63: private void LoadDataToDatabase(string tableName, string fileFullPath, string delimeter)
64: {
65: string sqlQuery = string.Empty;
66: StringBuilder sb = new StringBuilder();
67:
68: sb.AppendFormat(string.Format("BULK INSERT {0} ", tableName));
69: sb.AppendFormat(string.Format(" FROM '{0}'", fileFullPath));
70: sb.AppendFormat(string.Format(" WITH ( FIELDTERMINATOR = '{0}' , ROWTERMINATOR = '\n' )", delimeter));
71:
72: sqlQuery = sb.ToString();
73:
74: using (SqlConnection sqlConn = new SqlConnection(GetConnectionString()))
75: {
76: sqlConn.Open();
77: SqlCommand sqlCmd = new SqlCommand(sqlQuery, sqlConn);
78: sqlCmd.ExecuteNonQuery();
79: sqlConn.Close();
80: }
81: }
82: protected void Page_Load(object sender, EventArgs e)
83: {
84:
85: }
86: protected void BTNImport_Click(object sender, EventArgs e)
87: {
88: if (FileUpload1.HasFile)
89: {
90: FileInfo fileInfo = new FileInfo(FileUpload1.PostedFile.FileName);
91: if (fileInfo.Name.Contains(".csv"))
92: {
93:
94: string fileName = fileInfo.Name.Replace(".csv", "").ToString();
95: string csvFilePath =
Server.MapPath("UploadedCSVFiles") + "\\" + fileInfo.Name;
96:
97: //Save the CSV file in the
Server inside 'MyCSVFolder'
98: FileUpload1.SaveAs(csvFilePath);
99:
100: //Fetch the location of CSV file
101: string filePath =
Server.MapPath("UploadedCSVFiles") + "\\";
102: string strSql = "SELECT * FROM [" + fileInfo.Name + "]";
103: string strCSVConnString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + filePath + ";" + "Extended Properties='text;HDR=YES;'";
104:
105: // load the data from CSV to DataTable
106:
107: OleDbDataAdapter adapter = new OleDbDataAdapter(strSql, strCSVConnString);
108: DataTable dtCSV = new DataTable();
109: DataTable dtSchema = new DataTable();
110:
111: adapter.FillSchema(dtCSV, SchemaType.Mapped);
112: adapter.Fill(dtCSV);
113:
114: if (dtCSV.Rows.Count > 0)
115: {
116: CreateDatabaseTable(dtCSV, fileName);
117: Label2.Text = string.Format("The table ({0}) has been successfully created to the database.", fileName);
118:
119: string fileFullPath = filePath + fileInfo.Name;
120: LoadDataToDatabase(fileName, fileFullPath, ",");
121:
122: Label1.Text = string.Format("({0}) records has been loaded to the table {1}.", dtCSV.Rows.Count, fileName);
123: }
124: else
125: {
126: LBLError.Text = "File is empty.";
127: }
128: }
129: else
130: {
131: LBLError.Text = "Unable to recognize file.";
132: }
133:
134: }
135: }
136: }
137: }
The code above consists of three (3) private methods which are the GetConnectionString(), CreateDatabaseTable() and LoadDataToDatabase(). The GetConnectionString() is a method that returns a string. This method basically gets the connection string that is configured in the web.config file. The CreateDatabaseTable() is method that accepts two (2) parameters which are the DataTable and the filename. As the method name already suggested, this method automatically create a Table to the database based on the source DataTable and the filename of the CSV file. The LoadDataToDatabase() is a method that accepts three (3) parameters which are the tableName, fileFullPath and delimeter value. This method is where the actual saving or importing of data from CSV to SQL
server happend.
The codes at BTNImport_Click event handles the uploading of CSV file to the specified location and at the same time this is where the CreateDatabaseTable() and LoadDataToDatabase() are being called. If you notice I also added some basic trappings and validations within that event.
Now to test the importing utility then let's create a simple data in a CSV format. Just for the simplicity of this demo let's create a CSV file and name it as "Employee" and add some data on it. Here's an example below:
1,VMS,Durano,
[email protected]
2,Jennifer,Cortes,
[email protected]
3,Xhaiden,Durano,
[email protected]
4,Angel,Santos,
[email protected]
5,Kier,Binks,
[email protected]
6,Erika,Bird,
[email protected]
7,Vianne,Durano,
[email protected]
8,Lilibeth,Tree,
[email protected]
9,Bon,Bolger,
[email protected]
10,Brian,Jones,
[email protected]
Now save the newly created CSV file in some location in your hard drive.
Okay let's run the application and browse the CSV file that we have just created. Take a look at the sample screen shots below:
After browsing the CSV file.
After clicking the Import Button
Now if we look at the database that we have created earlier you'll notice that the Employee table is created with the imported data on it. See below screen shot.
That's it! I hope someone find this post useful!
Technorati Tags: ASP.NET,CSV,SQL,C#,ADO.NET
