Login and access website

[mariolopes]

Hi
I want to login on this website

https://www.cpcdi.pt/Account/Login
I have the credentials and I try the following code:

import requests
import sys
import urllib.request
import re
URL = 'https://www.cpcdi.pt/Account/Login'
def main():
    # Start a session so we can have persistant cookies
    #session = requests.session(config={'verbose': sys.stderr})
    # This is the form data that the page sends when logging in
    login_data = {
        'CodCliente': 'mycode',
        'UserName': 'myuser',
        'Password': 'mypass',
        'submit':'submit',
    }

    # Authenticate
    r = session.post(URL, data=login_data)

    # Try accessing a page that requires you to be logged in
    URL='https://www.cpcdi.pt/Produtos/Referencia?referencia=8745B006AA'
    r = session.get(URL)

I got no error on this code but I'm not sure that it's work.
I need to download pictures from the address like
URL='https://www.cpcdi.pt/Produtos/Referencia?referencia=8745B006AA'

for that I try the code:

req = urllib.request.Request(URL, headers={'User-Agent': 'Mozilla/5.0'})        
htmltext = urllib.request.urlopen(req).read()
if htmltext is None:
    print("nada")
else:
    regex='<img src="/(.+?)"'
    pattern=re.compile(regex)
    imagem=pattern.findall(str(htmltext))
    print(imagem[0])
#descarrega imagem
import urllib.request
urllib.request.urlretrieve(URL+imagem[0], "local-filename.jpg")

But no luck. I got

Error:
Traceback (most recent call last):
  File "C:\python-ficheiros\abrir.py", line 33, in <module>
    print(imagem[0])
IndexError: list index out of range

Any help on this matter?
Thank you

[metulburr]

If you get an indexerror on the first element, then the list is empty. which means imagem = [] and your pattern is not matching. Dont use regex to parse html, use beautifulsoup as that is what it was made for.

https://python-forum.io/Thread-Web-Scraping-part-1
https://python-forum.io/Thread-Web-scraping-part-2

you can download the image via requests

import shutil
import requests

url = 'http://example.com/img.png'
response = requests.get(url, stream=True)
with open('img.png', 'wb') as out_file:
    shutil.copyfileobj(response.raw, out_file)

[mariolopes]

Thank you for your help, but I think the problem is that I can't login with Python
In my code:

import requests
import sys
import urllib.request
import re
import shutil
URL = 'https://www.cpcdi.pt/Account/Login'
def main():
    # Start a session so we can have persistant cookies
    #session = requests.session(config={'verbose': sys.stderr})
    # This is the form data that the page sends when logging in
    login_data = {
        'CodCliente': 'xxx',
        'UserName': 'xx',
        'Password': 'xxxx',
        'submit':'submit',
    }
    # Authenticate
    r = session.post(URL, data=login_data)
    # Try accessing a page that requires you to be logged in
    URL='https://www.cpcdi.pt/Produtos/Referencia?referencia=8745B006AA'
    r = session.get(URL)
    print(r.text)

The last instrution returns nothing therefore I can't pass the login. I have the credentials and I can access with the browser but not with python. I think the submit button has no name and I don't know how to handle it.
Is that the best option to login with python?
regards

[metulburr]

Looking at the code, the blue "Entrar" button executes javascript. Which means you are going to need Selenium to click the button. The tutorial links I gave above for scraping websites has selenium examples in them too.

[snippsat]

Here a setup you can look.
If new to this always start with webdriver that you see what's going on like Chrome.
So can go headless later.
I use CSS selector like #UserName that find the user name field.
It fill out all fields and push log in button,so if all where correct i'll be logged in.

from selenium import webdriver
from bs4 import BeautifulSoup
import time

# Activate Phantom(headless) and deactivate Chrome to not load browser
#browser = webdriver.PhantomJS()
browser = webdriver.Chrome()
url = 'https://www.cpcdi.pt/Account/Login'
browser.get(url)
user_name = browser.find_element_by_css_selector('#CodCliente')
user_name.send_keys("Foo")
password = browser.find_element_by_css_selector('#UserName')
password.send_keys("Bar")
password = browser.find_element_by_css_selector('#Password')
password.send_keys("xxxxxxxxx")
time.sleep(5)
submit = browser.find_elements_by_css_selector('button.btn')
submit[0].click()
time.sleep(5)

# Give source code to BeautifulSoup
soup = BeautifulSoup(browser.page_source, 'lxml')
title = soup.select('head > title')
print(title[0].text)

[mariolopes]

Great help
Many thanks for that. But there is some strange behavoir with my code. Please look at it

from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen
# Activate Phantom(headless) and deactivate Chrome to not load browser
#browser = webdriver.PhantomJS()
browser = webdriver.Firefox()
url = 'https://www.cpcdi.pt/Account/Login'
browser.get(url)
user_name = browser.find_element_by_css_selector('#CodCliente')
user_name.send_keys("111")
password = browser.find_element_by_css_selector('#UserName')
password.send_keys("1222")
password = browser.find_element_by_css_selector('#Password')
password.send_keys("11222")
time.sleep(5)
submit = browser.find_elements_by_css_selector('button.btn')
submit[0].click()
time.sleep(5)
 
# Give source code to BeautifulSoup
goUrl="https://www.cpcdi.pt/Produtos/Referencia?referencia=8745B006AA"
browser.get(goUrl)
soup = BeautifulSoup(urlopen(goUrl), 'lxml')
for a in soup.find_all('a', href=True):
    print ("Found the URL:", a['href'])

works fine the browser goes to the link but I get the links from the first page, not from the page where the browser is. What I'm doing wrong?

[metulburr]

you dont need urlopen() if your using requests or selenium. Especially if you are using selenium to bypass javascript that urllib cannot bypass. You are basically opening a new page from python separate from selenium to pass to bs4 instead of the source from selenium

soup = BeautifulSoup(urlopen(goUrl), 'lxml')

do this

soup = BeautifulSoup(browser.page_source, 'lxml')

PS you might need some form of delay after browser.get and page source but unsure til you check it out

[mariolopes]

Simply perfect
Thank all.