Headless browser

[Truman]

Trying to run this simple code ( which should be a part of longer script ):

from selenium import webdriver
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

opts = Options()
opts.set_headless()
assert opts.headless # Operating in headless mode
browser = webdriver.Firefox(options=opts)
browser.get('https://duckduckgo.com')

...but from some reason it won't open duckduckgo web site.

[metulburr]

try adding headless=True in the argument of set_headless as show in this tutorial
https://python-forum.io/Thread-Web-scraping-part-2

[Larz60+]

Do you get any errors?
I tried it with chromedriver and it worked fine (although code a bit different (see below)):

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import os


class TryDucky:
    def __init__(self):
        os.chdir(os.path.dirname(__file__))
        self.duck()

    def duck(self):
        chrome_options = Options()
        chrome_options.add_argument("--headless")
        chrome_options.add_argument('--disable-gpu')
        chrome_options.add_argument('--log-level=3')
        browser = webdriver.Chrome(chrome_options=chrome_options)
        browser.get('https://duckduckgo.com')

        # Try to find something (otherwise wont see anything since headless)
        search_comment = browser.find_element_by_xpath('/html/body/div/div[2]/div/div/div[3]/div/div')
        print(f'\n{search_comment.text}')

if __name__ == '__main__':
    TryDucky()

Don't forget if headless, you won't see anything until you actually print it.

result:

Output:
The search engine that doesn't track you. Help Spread DuckDuckGo!

options part of code courtesy snippsat

[Truman]

try adding headless=True in the argument of set_headless as show in this tutorial
https://python-forum.io/Thread-Web-scraping-part-2

added it and it's struggling to execute the code. After a while I receive this:

Error:
C:\Python36\kodovi>sel6.py
Traceback (most recent call last):
  File "C:\Python36\kodovi\sel6.py", line 9, in <module>
    browser = webdriver.Firefox(options=opts)
  File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\webdriver.py",
line 174, in __init__
    keep_alive=True)
  File "C:\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "C:\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "C:\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 321, in execute
    self.error_handler.check_response(response)
  File "C:\Python36\lib\site-packages\selenium\webdriver\remote\errorhandler.py"
, line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused

larz60+, I'm looking at your code and it looks a bit above my league.

[metulburr]

Check your geckodriver.log file (should be in the same directory as python file)

If it says Error: GDK_BACKEND does not match available displays then install pyvirtualdisplay:

pip install pyvirtualdisplay selenium

You might need xvfb too:

[Truman]

metulburr, I installed pyvirtualdisplay but still get the same error.
Not sure what did you mean by xvfb, didn't add anything...

[snippsat]

You may need to update FireFox,Selenium and geckodriver.
In first post you use older code.
opts.set_headless() is Deprecated,but it still work.

If you don't get error with first code,can try to get something back.
Remember headless mean that you don't see anything as browser dos not start.
set_headless(headless=True) to see browser set_headless(headless=False)
And comment out #assert opts.headless.

from selenium import webdriver
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

opts = Options()
opts.set_headless()
assert opts.headless # Operating in headless mode
browser = webdriver.Firefox(options=opts)
browser.get('https://duckduckgo.com')
logo = browser.find_elements_by_css_selector('#logo_homepage_link')
print(logo[0].text)

Output:
About DuckDuckGo

FireFox has messed a lot with setting for there driver,so updated browser may need this.
Both this method testet here work fine for me.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
import time

#--| Setup
options = Options()
options.add_argument("--headless")
caps = webdriver.DesiredCapabilities().FIREFOX
caps["marionette"] = True
browser = webdriver.Firefox(firefox_options=options, capabilities=caps, executable_path=r"geckodriver.exe")
#--| Parse
browser.get('https://duckduckgo.com')
logo = browser.find_elements_by_css_selector('#logo_homepage_link')
print(logo[0].text)

Output:
About DuckDuckGo

[Truman]

snippsat, your last solution works on my end. Thank you, still don't understand it. I guess that this #--| Setup part can be used in other scripts too. Will save it.

---

by the way, how to stop music
I tried with

browser.close()
quit()

but it won't work.

also, is there any point to add

Output:
time.sleep()

if I want to listen to music for some time? Would try it myself but first have to figure out how to stop music.

[snippsat]

I guess that this #--| Setup part can be used in other scripts too. Will save it.

Yes can use it for all when work parse or automation ,just comment out --headless then it work without.
Example search and show images of cars.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.keys import Keys
import time

#--| Setup
options = Options()
#options.add_argument("--headless")
caps = webdriver.DesiredCapabilities().FIREFOX
caps["marionette"] = True
browser = webdriver.Firefox(firefox_options=options, capabilities=caps, executable_path=r"geckodriver.exe")
#--| Parse or automation
browser.get('https://duckduckgo.com')
input_field = browser.find_elements_by_css_selector('#search_form_input_homepage')
input_field[0].send_keys('car' + Keys.RETURN)
time.sleep(3)
images_link = browser.find_elements_by_link_text('Images')
images_link[0].click()
time.sleep(5)
browser.quit()

if I want to listen to music for some time? Would try it myself but first have to figure out how to stop music.

No time.sleep() is not for that,you have to push start and stop button on web-site that has music.
Also you find button eg bye CSS selector,then call click() on eg stop button.

[Truman]

Thank you. The next step for this car script is to download these images. Will get back to that later when I finish the script I started. Remember that there is a chapter in Automate the boring stuff with Python about that stuff.

---

regarding push and stop button I tried this but it won't stop the music.

browser.find_element_by_css_selector('#playbutton playing')[0].click()

'playbutton playing' is what I see when the music starts.