How to handle editing a large file for a non-technical user
- by Luke
I have a client who is given a tab delimited .txt file containing hundreds of thousands of rows.
I have a user story as follows:
As a user I want to take the text file and add a new value at the end of each line which contains the concatenated value of two of the columns.
for example if the file read
text_one text_two
I need to output the following (preferably to a .txt file)
text_one text_two text_onetext_two
My first approach was to ask the vendor supplying the file to do the concatenation before providing the file, the easiest way to solve a problem is to eliminate it right? however they are very uncooperative and have point blank refused.
I've looked at building a simple javascript application that does this client side so a non-technical user could select the file using a file selector. This approach has a few problems
The file could be over a GB in size and so can't be loaded straight into memory, I've tried and the browser crashes
There is no means to write a file in javascript so I'd need to output the content to the screen and have the user save it (somehow)
I was thinking if I could get around the filesize limitations I could just output the edited content to the page and have the user save the page as a .txt file, however I think there is a better way than using javascript that will still accommodate the users lack of technical know-how.
Please consider this question to be stack agnostic, but bear in mind that a nice little shell script or python script would be deemed unsuitable for a non technical user unless there is a way of "packaging" it nicely for a non-technical user.
Updates
The file is too large to open in excel.
The process needs to be run weekly, but it doesn't require scheduling or automation...(yet)
