How to scrape a table that is populated by javascript?

I am learning scraping. Scraping site.

I am able to select:

जिला:, अनुमंडल:, अंचल:

from dropdown using selenium.

I can select from मौजा का नाम चुने:.

enter image description here

Afterwards, I am able to click on the खाता खोजें button.

enter image description here

As a result, a table is populated at the bottom by javascript.

enter image description here

The button’s div code:

<input type="submit" name="ctl00$ContentPlaceHolder1$BtnSearch" value="खाता खोजें" onclick="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;ctl00$ContentPlaceHolder1$BtnSearch&quot;, &quot;&quot;, true, &quot;S&quot;, &quot;&quot;, false, false))" id="ctl00_ContentPlaceHolder1_BtnSearch" style="width:146px;">

Pagination is done by:

javascript:__doPostBack('ctl00$ContentPlaceHolder1$GridView1','Page$11')

I am not able to scrape this table.

What I have tried:

  1. phantomjs isn’t supported with selenium
  2. Table’s id, ctl00_ContentPlaceHolder1_GridView1, is not in HTML source code. Tried some approaches, no luck so far.
#p_element = driver.find_element_by_id(id_='ctl00_ContentPlaceHolder1_GridView1')

p_element = driver.find_element_by_xpath('//*[@id="aspnetForm"]/table/tbody/tr/td/table/tbody/tr/td[2]/table/tbody/tr[3]/td/table/tbody/tr/td/table/tbody/tr[4]')

print(p_element.text)


path_for_table='//*[@id="aspnetForm"]/table/tbody/tr/td/table/tbody/tr/td[2]/table/tbody/tr[3]/td/table/tbody/tr/td/table/tbody/tr[4]'

table_list = WebDriverWait(driver, 2).until(lambda driver: driver.find_element_by_xpath(path_for_table))

print(table_list)

Pages I have looked at:

14 thoughts on “How to scrape a table that is populated by javascript?”

Leave a Comment