Improving the performance of a db import process
- by mmr
I have a program in Microsoft Access that processes text and also inserts data in MySQL database. This operation takes 30 mins or less to finished.
I translated it into VB.NET and it takes 2 hours to finish.
The program goes like this:
A text file contains individual swipe from a corresponding person,
it contains their id, time and date of swipe in the machine, and an
indicator if it is a time-in or a time-out.
I process this text, segregate the information and insert the time-in and time-out per row.
I also check if there are double occurrences in the database.
After checking, I simply merge the time-in and time-out of the corresponding person into one row only.
This process takes 2 hours to finished in VB.NET considering I have a table to compare which contains 600,000+ rows.
Now, I read in the internet that python is best in text processing, i already have a test but i doubt in database operation.
What do you think is the best programming language for this kind of problem?
How can I speed up the process? My first idea was using python instead of VB.NET, but since people here telling me here on SO that this most probably won't help I am searching for different solutions.
