optimize python code

Posted by user283405 on Stack Overflow See other posts from Stack Overflow or by user283405
Published on 2010-04-26T07:48:43Z Indexed on 2010/04/26 7:53 UTC
Read the original article Hit count: 243

Filed under:

python

|

optimization

|

beautifulsoup

i have code that uses beautifulsoup library for parsing. But it is very slow. The code is written in such a way that threads cannot be used. Can anyone help me about this?

optimize python code

Posted by user283405 on Stack Overflow See other posts from Stack Overflow or by user283405
Published on 2010-04-26T09:58:27Z Indexed on 2010/04/26 10:03 UTC
Read the original article Hit count: 243

Filed under:

python

|

optimization

|

beautifulsoup

i have code that uses BeautifulSoup library for parsing. But it is very slow. The code is written in such a way that threads cannot be used. Can anyone help me about this?

I am using beautifulsoup library for parsing and than save in DB. if i comment the save statement, than still it takes time so there is no problem with database.

def parse(self,text):                
                soup = BeautifulSoup(text)
                arr = soup.findAll('tbody')                




                for i in range(0,len(arr)-1):
                        data=Data()
                        soup2 = BeautifulSoup(str(arr[i]))
                        arr2 = soup2.findAll('td')

                        c=0
                        for j in arr2:                                       
                                if str(j).find("<a href=") > 0:
                                        data.sourceURL = self.getAttributeValue(str(j),'<a href="')
                                else:  
                                        if c == 2:
                                                data.Hits=j.renderContents()

                    #and few others...
                    #...        

                                c = c+1

            data.save()

Any suggestions?

Note: I already ask this question here but that was closed due to incomplete information.

Developer IT