Project: “I’m Feeling Lucky” Google Search

[Truman]

This is web scraping project from Automate the Boring Stuff with Python. From some reason solution doesn't work and I have some doubts myself. The goal is to type a search term on the command line and have my computer automatically open a browser with all the top search results in new tabs.

#! python3
# lucky.py - Opens several Google search results.

import webbrowser, bs4, requests, sys

print('Googling...') # display text while downloading the Google page
res = requests.get('http://google.com/search?source=' + ' '.join(sys.argv[1:]))
print(res.raise_for_status())

# Retreive top search result links.
soup = bs4.BeautifulSoup(res.text, "html.parser")

# Open a browser tab for each result
linkElems = soup.select('.r a')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
	webbrowser.open('http://google.com' + linkElems[i].get('href'))

I'm also not sure how to start this program. Should I type a search term in command prompt where I usually start my python files?

[Larz60+]

I'm also not sure how to start this program

you can create a function, for example named test_it() which executes a test case, then
start that with (untested):

#! python3
# lucky.py - Opens several Google search results.
 
import webbrowser
import bs4
import requests
import sys

def my_func(search_str) :
    print('Googling...') # display text while downloading the Google page
    res = requests.get('http://google.com/search?source=' + ' '.join(search_str))
    print(res.raise_for_status())
 
    # Retreive top search result links.
    soup = bs4.BeautifulSoup(res.text, "html.parser")
 
    # Open a browser tab for each result
    linkElems = soup.select('.r a')
    numOpen = min(5, len(linkElems))
    for i in range(numOpen):
        webbrowser.open('http://google.com' + linkElems[i].get('href'))

def test_it():
    my_func(sys.argv[1:])

if __name__ == '__main__':
    test_it()

this code will only run if the program is started from within an IDE, or from the command line, but will not run if your main code is called from another program.

[Truman]

Error:
Traceback (most recent call last):
  File "C:\Python36\kodovi\lucky.py", line 4, in <module>
    import webbrowser, bs4, requests, sys
ModuleNotFoundError: No module named 'bs4'
>>> %Run lucky1.py
Traceback (most recent call last):
  File "C:\Python36\kodovi\lucky1.py", line 5, in <module>
    import bs4
ModuleNotFoundError: No module named 'bs4'

Now I just feel completely lost. lucky is my code filename and I named lucky1 what you Larz wrote. I used Thonny IDE.
I installed bs4 of course.

and when I run code from Notepad++ with command prompt I get this:

Output:
C:\Python36\kodovi>lucky novak djokovic
Googling...
None

my search term in command line is "novak djokovic" and it's not giving any result obviously.
I'll try to find solution on stackoverflow or smth like that.

[Larz60+]

whenever you see: ModuleNotFoundError: No module named '...', you need to install a package.

In this case, you need to install BeautifulSoup4:

first test to see that pip version is correct one:

pip -V

You want to see something like:

Output:
pip 10.0.1 from ... (python 3.6)

if you don't see proper python version, substitute pip3 for pip below

pip install BeautifulSoup4

[Truman]

Output:
C:\Users\user>pip -V
pip 10.0.1 from c:\python36\lib\site-packages\pip (python 3.6)

C:\Users\user>pip3 install BeautifulSoup4
Requirement already satisfied: BeautifulSoup4 in c:\python36\lib\site-packages (
4.6.0)

this is what I get...From some reason I still receive the same Traceback error with Thonny and the same output with command prompt.

[Larz60+]

that's good, that means you can use:

pip install BeautifulSoup4

I see it's already installed.
Then make the following changes on your code (by line number)
In reverse order (so line numbers remain constant until done)
line 15

# from:
    soup = bs4.BeautifulSoup(res.text, "html.parser")
# to:
    soup = BeautifulSoup(res.text, "html.parser")

line 5

# From:
import bs4
# to:
from bs4 import BeautifulSoup

[Truman]

I edited my last post in the meantime. Now the question is how to start this program.

[Larz60+]

run from command line

python myprogramname.py

[Truman]

Output:
C:\Python36\kodovi>python lucky1.py
Googling...
None

The point of this program, if I understand it correctly, is to add search word in command line and immediately get 5 open tasks with first 5 search results. I tried with adding search words too after filename but the outcome is the same.

[Larz60+]

It appears to have run
you are getting none from:

print(res.raise_for_status())

after line 18, add:

for link in linkElems:
    print(f'link: {link}')

so you can see if the soup.select (which is css select by the way) is finding anything

also, you need to supply a value to the command line e.g.

python -m myprog.py searchvalue