Python: Is there a way to get HTML that was dynamically created by Javascript?
- by Joschua
As far as I can tell, this is the case for LyricWikia. The lyrics (example) can be accessed from the browser, but can't be found in the source code (can be opened with CTRL + U in most browsers) or reading the contents of the site with Python:
from urllib.request import urlopen
URL = 'http://lyrics.wikia.com/Billy_Joel:Piano_Man'
r = urlopen(URL).read().decode('utf-8')
And the test:
>>> 'Now John at the bar is a friend of mine' in r
False
>>> 'John' in r
False
But when you select and look at the source code of the box in which the lyrics are displayed, you can see that there is: <div class="lyricbox">[...]</div>
Is there a way to get the contents of that div-element with Python?
