Web-scraping, multiple webpages

[Pabloty92]

Hello there,

I'm a newbie and still learning python and I have some problem with my scraping code. Namely, I would like to scrap data from 273 pages of the same website . My code works when I want to scrap one page, but when I add more pages to code it scrapes only from the last one. There is code below:

import requests
import bs4
from bs4 import BeautifulSoup
headers = {
    'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'
}
page = 1
while page !=274:
    url=f"https://www.kmotorshop.com/en/article-list/list/0/0/tree-shop%7C258?page={page}&itemsPerPage=50&style=1"
    page = page + 1
r = requests.get(url,{'headers':headers})
soup = bs4.BeautifulSoup(r.text,'html.parser')
i = 0
while i != 50:
    print (soup.findAll('div',{'class': 'product-line__heading'})[i].find('a').text)
    i+=1

Can someone help me to connect all pages and scrape them all together?

Yoriz write Dec-28-2022, 02:04 PM:
Please post all code, output and errors (in their entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.

[Yoriz]

In the while loop, the variable url is overwritten each time it loops leaving it with only the last value,
When using variables in loops you could either interact with the current item ieurl inside of the loop or use something that can contain multiple items like a list