Deleting Duplicated Lines In TEXT File?

Posted by echolab on Super User See other posts from Super User or by echolab
Published on 2012-06-24T10:44:28Z Indexed on 2012/06/24 21:18 UTC
Read the original article Hit count: 349

Filed under:

microsoft-office

|

regex

|

text

|

editor

I am trying to cleanup a text and for some reason every line duplicated 3 times am i able to get ride of duplicates with regex or tricks or do you know a software which could do that , text file is like this

Party Started 10:17 (89/1/2)
Party Started 10:17 (89/1/2)
Party Started 10:17 (89/1/2)
Jessica At Dinner 17:54 (89/1/2)
Jessica At Dinner 17:54 (89/1/2)
Jessica At Dinner 17:54 (89/1/2)

How can i clean it up , and get ride of duplicated lines , it's about 69,587 lines

© Super User or respective owner

Related posts about microsoft-office

Direct Links To Download Microsoft Office 2007 Products From Microsoft Download Servers

as seen on Tech Dreams - Search for 'Tech Dreams'
Downloading installers of Microsoft Office 2007 products from Microsoft Office website is not an easy task. To download any of the Microsoft Office 2007 products – you need to sign-in using Windows Live/Passport/Hotmail account, fill a big profile form and accept their terms & conditions… >>> More
Microsoft.Office.Interop.Excel.ListObject vs Microsoft.Office.Tools.Excel.ListObject

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to access the Selected Event of all the listobjects in all the worksheets of my workbook but when I access worksheet.listobject, that object apparently belongs to Microsoft.Office.Interop.Excel.ListObject and so doesn't have any events where as the table list object belongs to Microsoft.Office… >>> More
ERROR: Attempted to read or write protected memory. This is often an indication that other memory is corrupt

as seen on Server Fault - Search for 'Server Fault'
I get this error after having edited a few pages in SharePoint 2010. I have to do an IISReset on both front ends to get this to resolve. I don't know how to fix it or even what else to supply here, but please let me know as the resets now happen several times per day. Log Name: Application Source:… >>> More
Errors Event 7888, 6102, 6641 MOSS

as seen on Server Fault - Search for 'Server Fault'
I'm getting this errors on my Server. Version MOSS 2007, on Windows Server Standar 2008 and this message appear every minute. Next, you can find its descriptions: Can any one help me with this? Thanks Event ID 7888 A runtime exception was detected. Details follow. Message: Attempted to perform… >>> More
Microsoft Office 2010 sortirait en Mai avec un mois d'avance, la version RTM annoncée pour Avril

as seen on Developper.com - Search for 'Developper.com'
Microsoft Office 2010 sortirait en Mai avec un mois d'avance, la version RTM annoncée pour Avril >>> More

Related posts about regex

Find multiple regex in each line and skip result if one of the regex doesn't match

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a list of variables: variables = ['VariableA', 'VariableB','VariableC'] which I'm going to search for, line by line ifile = open("temp.txt",'r') d = {} match = zeros(len(variables)) for line in ifile: emptyCells=0 for i in range(len(variables)): regex = r'('+variables[i]+r')[:|=|\(](-… >>> More
OWASP Regex Repository: Is this regex correct?

as seen on Stack Overflow - Search for 'Stack Overflow'
I was looking at the regular expression for validating various data types from the (OWASP Regex Repository). One of the regular expressions in there is called safetext and looks like: ^[a-zA-Z0-9\s.\-]+$ My first question is: Is this regular expression correct? complementary question If this… >>> More
Make a Perl-style regex interpreter behave like a basic or extended regex interpreter

as seen on Stack Overflow - Search for 'Stack Overflow'
I am writing a tool to help students learn regular expressions. I will probably be writing it in Java. The idea is this: the student types in a regular expression and the tool shows which parts of a text will get matched by the regex. Simple enough. But I want to support several different regex… >>> More
JS regex isn't matching, even thought it works with a regex tester

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm writing a piece of client-side javascript code that takes a function and finds the derivative of it, however, the regex that's supposed to match with the power rule fails to work in the context of the javascript program, even though it sucessfully matches when it's used with an independent regex… >>> More
c# RegEx with "|"

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to be able to check for a pattern with | in them. For example an expression like d*|*t should return true for a string like "dtest|test". I'm no regex hero so I just tried a couple of things, like: Regex Pattern = new Regex("s*\|*d"); //unable to build because of single backslash Regex Pattern… >>> More