Extract data with Selenium and BeautifulSoup

[nestor]

Hi, I'm trying to extract live betting data from this sports betting site, https://bwin.com this is the code I use

from selenium import webdriver
from bs4 import BeautifulSoup
import time
 

browser = webdriver.Chrome()
url = 'https://sports.bwin.com/sports/events/benfica-tondela-10089945'
browser.get(url)
time.sleep(3)
soup = BeautifulSoup(browser.page_source, 'html.parser')
events = []

for i in soup.select('div[class="name"]'):
    events.append(i.text)
print(events)

if I open the url of the event in this case Benfica-Tondela with this code I extract the data of that event but if I want to extract the data of all the events in live I have to open the events page and click with Selenium to open all event links or is there a faster method? thank you all for your help

[Larz60+]

If you have to traverse several pages that contain JavaScript, there are a couple of options.

1. You can continue to use beautiful soup to parse a page, but selenium when you have to follow a link.
   
2. Do it all with selenium
   

[nestor]

Thanks for the reply, so doing everything with Selenium or using Beautifulsoap is the same or one method is better than another ?

[Larz60+]

Neither fits all websites.
Beautifulsoup where no JavaScript expansion is necessary, Selenium otherwise.
It would be helpful for you if you went through Snippsat's web scraping tutorial,
which only takes a few minutes and will give you a better outlook on which package fits which situation.
web scraping part 1
web scraping part 2