Selenium - "Element is not currently visible and may not be manipulated"

[tkj80]

Hi,

I have been working on the following code to scrape data from the website for my research. However, I got the following error message. "errorMessage":"Element is not currently visible and may not be manipulated"

When I try out each line of the code on the Python interpreter, they would work fine. However, when I save the code as scrapeEcoData_weekly.py and run the file, that's when I get the error message.

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.common.by import By
import time

driver = webdriver.PhantomJS(executable_path='C:/Users/AppData/Local/Programs/Python/Python35-32/myScript/Misc/PhantomJS')
driver.get("https://www.dailyfx.com/calendar")

weeklyView = driver.find_element_by_xpath('//*[@class="btn btn-default webinar-hover dfx-has-spinner hidden-xs"]')
weeklyView.click()
time.sleep(10)
pageSource = driver.page_source
bsObj = BeautifulSoup(pageSource, "html.parser")
tblTag = bsObj.findAll("table", {"class":"table dfx-calendar-table tab-pane fade in active "})[0]
tRows = tblTag.findAll("tr")
for index, item in enumerate(tRows):
print("index: ", index, " ", "item: ", item.get_text())

I tried to locate the "Weekly View" button element using the code instead. The codes would run fine. I took a screenshot of the website after weeklyView.click(), the weekly page didn't show up. It was still showing the daily page as if the button was never clicked on.

weeklyView = driver.find_element_by_xpath('//*[contains(., "Weekly View")]

However, if someone could help me with this.
Thank you!

[Larz60+]

first, I use python's builtin urllib.request to get a page:
Here's a module you can take code from, or save it as GetUrl.py
and just import into your program

# GetUrl - Fetch files or web pages from internet
#
# Author: Larry McCaig (Larz60+)
import urllib.request as ur
import os
from time import sleep
import sys


class GetUrl:
    def __init__(self, returndata=False):
        self.returndata = returndata

    def get_url(self, url, ret=False, tofile=None, bin=False):
        rdata = None
        head, tail = os.path.split(url)
        try:
            if tofile:
                if os.path.exists(tofile):
                    os.remove(tofile)
                if bin:
                    with open(tofile, 'wb') as f:
                        rdata = ur.urlopen(url).read()
                        f.write(rdata)
                else:
                    with open(tofile, 'w') as f:
                        rdata = ur.urlopen(url).read().decode('utf8')
                        f.write(rdata)
                sleep(.5)
            if ret:
                return rdata
        except:
            print("Unexpected error:", sys.exc_info()[0])

I don't know which browser you use, mine is firefox, so I'll give instructions on how I'd go about debugging the error

In firefox, select tools->Web Developer->Page Source
The page html will come up
save it locally to a file (file->save as->filename.html

In your code, instead of fetching the page, open it and read into 'pageSource'

    with open(filename.html, 'r') as f:
        pagesource = f.read()

Now you can play around with it as much as you like without reloading each time.
You can also load the html file into an editor so you can manually search for variables.

Not an immediate fix for your problem, but perhaps will help with debugging

[snippsat]

Now you can play around with it as much as you like without reloading each time.
You can also load the html file into an editor so you can manually search for variables.

Not an immediate fix for your problem, but perhaps will help with debugging

Little on wrong track here Lars60+,
because this site use JavaScript heavily and he shall also push a button(which load ouical.js).
The problem is that urllib(or Requests as you should have used) can not get this info.

 It was still showing the daily page as if the button was never clicked on.

 FireFox web driver has a click() method,but when you load page trough PhantomJS(has no click() method) has to use other tool.
jQuery selector can be used and execute bye PhantomJS.

You can try this.

#weeklyView = driver.find_element_by_xpath('//*[@class="btn btn-default webinar-hover dfx-has-spinner hidden-xs"]')
#weeklyView.click()
driver.execute_script("$('#calendarbody > div:nth-child(12) > div.col-sm-2.calendar-home-weekly > button').click()")

Your indention is wrong in last two lines but this i guess is a copy fault.
The default path that 3.5 use is horrible,i would choose C:\Python35 so same as i had for 10-year ago C:\Python25