Python Forum
Selenium chrome driver not returning solution
Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Selenium chrome driver not returning solution
#1
Hi i have created a web scraper using in python using selenium.
Website i am scraping
Please remove the #

(to access website please select promoter , in division select konkan and and in district select mumbai city, and select projects you will see list of projects)

this is for 1 district and 1 division.

what my scraper do it it loops through every district and every division and select project and get details of those projects.

But nowadays it scraps only 1 project name and stops it doesn't loop through list of projects and this is the error i receive:

Error:
Traceback (most recent call last):  File "C:\Users\prince.bhatia\Desktop\Version\Maha Rera.py", line 64, in <module>    while len(selectProject.options) == 1:  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\support\select.py", line 47, in options    return self._el.find_elements(By.TAG_NAME, 'option')  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webelement.py", line 527, in find_elements    {"using": by, "value": value})['value']  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webelement.py", line 493, in _execute    return self._parent.execute(command, params)  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 254, in execute    response = self.command_executor.execute(driver_command, params)  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 464, in execute    return self._request(command_info[0], url, body=data)  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 487, in _request    self._conn.request(method, parsed_url.path, body, headers)  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1239, in request    self._send_request(method, url, body, headers, encode_chunked)  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1285, in _send_request    self.endheaders(body, encode_chunked=encode_chunked)  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1234, in endheaders    self._send_output(message_body, encode_chunked=encode_chunked)  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1026, in _send_output    self.send(msg)  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 964, in send    self.connect()  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 936, in connect    (self.host,self.port), self.timeout, self.source_address)  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\socket.py", line 722, in create_connection    raise err  File "C:\Users\prince.bhatia\AppData\Local\Programs\Python\Python36\lib\socket.py", line 713, in create_connection    sock.connect(sa) OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException


import os
import time
import csv

driver = webdriver.Chrome("./chromedriver")

driver.get('https://maharerait.mahaonline.gov.in/searchlist/searchlist')

# try:
#     element = WebDriverWait(driver, 100).until(
#         EC.presence_of_element_located((By.ID, "Promoter"))
#     )
# finally:
#     print('0000000000000000000000')
#     driver.quit()

time.sleep(1)

driver.find_element_by_id('Promoter').click()

divisionLength = len(Select(driver.find_element_by_id('Division')).options)
print('*********{}'.format(divisionLength))

firstRow = 0
titleRow = []
contentRows = []

for divisionElement in range(1,divisionLength):
    selectDivision = Select(driver.find_element_by_id('Division'))
    selectDivision.options
    selectDivision.select_by_index(divisionElement)
    time.sleep(1)
    districtLength = len(Select(driver.find_element_by_id('District')).options)
    while districtLength == 1:
        pass
    print(districtLength)
    for districtElement in range(1,districtLength):
        selectDistrict = Select(driver.find_element_by_id('District'))
        selectDistrict.options
        
        selectDistrict.select_by_index(districtElement)
      
        time.sleep(2)
        projectLength = len(Select(driver.find_element_by_id('Project')).options)
        
        print('/------------------------------/')
        print('/-----project number: {}-------/'.format(projectLength))
        print('/------------------------------/')
        if projectLength == 1:
            continue
        
        for projectElement in range(1,projectLength):
            selectProject = Select(driver.find_element_by_id('Project'))
            selectProject.options

            while len(selectProject.options) == 1:
                pass
            # c = len(select.options)
            # print('---------------{}'.format(c))

            # titleRow = []
            # contentRows = []
            # firstRow = 0

            # for i in range(1,c):
            #     select = Select(driver.find_element_by_id('Project'))
            #     while len(select.options) == 1:
            #         pass
            time.sleep(1)
            selectProject.select_by_index(projectElement)

            driver.find_element_by_id('btnSearch').click()
            tableRows = driver.find_element_by_class_name('table').find_elements_by_tag_name('tr')
            
            if firstRow == 0:
                headRow = tableRows[0].find_elements_by_tag_name('th')
                for headRowData in range(0,len(headRow)):
                    text = headRow[headRowData].find_element_by_tag_name('span').text
                    titleRow.append(text)
                firstRow = firstRow + 1

            for dataRowsNumbers in range(1,len(tableRows)):
                dataRow = tableRows[dataRowsNumbers].find_elements_by_tag_name('td')
                tempList = []
                for dataRowContents in range(0,len(dataRow)):
                    try:
                        a_link = dataRow[dataRowContents].find_element_by_tag_name('a').get_attribute('href')
                        tempList.append(str(a_link))
                    except NoSuchElementException:

                        tempList.append(str(dataRow[dataRowContents].text))
                    # if dataRow[dataRowContents].text == 'View':
                    #     a_link = dataRow[dataRowContents].find_element_by_tag_name('a').get_attribute('href')
                    #     tempList.append(str(a_link))
                    # else:
                    #     tempList.append(str(dataRow[dataRowContents].text))
                    print(dataRow[dataRowContents].text)
                print(tempList)
                contentRows.append(tempList)
# print('Automated check is over')
# print('Stored data in programs is as below:')
# print(contentRows)
with open("./data.csv",'w') as csvfile:
    csvfile = csv.writer(csvfile, delimiter=',')
    csvfile.writerow(titleRow)
    csvfile.writerow("")
    for i in range(0,len(contentRows)):
        csvfile.writerow(contentRows[i])

driver.close()
is someone can tell me what exact should i do to make it run or someone can make it run.
Python 3.6 i am using.
It should loop through every district and every division and scrape every project name with details.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Selenium returning web element instead of desired text newbie_programmer 2 9,541 Nov-12-2024, 07:27 PM
Last Post: brownbanned
  Selenium- Can't access "Chrome is being controlled by automated test software" Lahva 0 1,673 Sep-19-2024, 11:37 AM
Last Post: Lahva
  How do I iterate over an array and perform actions using selenium chrome webdriver? master 0 3,718 Sep-14-2020, 05:28 AM
Last Post: master
  Selenium/chrome console WiPi 4 8,765 Apr-12-2020, 11:45 AM
Last Post: WiPi
  Chrome instead of IE Friend 1 3,351 Feb-09-2020, 07:31 AM
Last Post: Larz60+
  Headless Chrome: How to login into a webpage? amandacstr 1 6,237 Feb-06-2020, 02:19 AM
Last Post: metulburr
  Interfacing Google Chrome & Python akornrumpf 5 6,529 Mar-16-2019, 10:20 PM
Last Post: Larz60+
  Selenium Web Driver Exe not running in other PC's Utkarsh29 2 6,701 Feb-28-2019, 05:48 PM
Last Post: snippsat
  Cannot open Chrome gahhon 3 5,791 Jan-26-2019, 04:56 AM
Last Post: snippsat
  selenium - chrome crashes when chrome is running. RvBVakama 4 19,915 Dec-16-2018, 06:32 PM
Last Post: metulburr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020