May-29-2022, 10:32 PM
Hello. I need to scrape following page:
https://www.semanticscholar.org/paper/BE...1594e0e992
I don't know how to get data which is loaded with ajax when page numbers are clicked. I know how to retrieve data from single page.
https://www.semanticscholar.org/paper/BE...1594e0e992
I don't know how to get data which is loaded with ajax when page numbers are clicked. I know how to retrieve data from single page.
from bs4 import BeautifulSoup as BS
import requests
r = requests.get("https://www.semanticscholar.org/paper/BERT%3A-Pre-training-of-Deep-Bidirectional-for-Devlin-Chang/df2b0e26d0599ce3e70df8a9da02e51594e0e992")
html = BS(r.content, 'html.parser')
index = 0
for el in html.select(".citation-list__citations > .cl-paper-row"):
title = el.select('a')
index += 1
if index == 11:
break
print(title[0].text)Please help.
