Jan-23-2020, 02:19 PM
Dear Members,
I want to pass both name and link_only from parse to parse_country. I know we use the meta function to transfer. Could you please tell me how can I change the meta function below to transfer both items? With the following code, I get blank cells in the CSV file for the link_only column.
In my actual project, I have 7/8 items to transfer from one parse to another. Please suggest me how to make it
I want to pass both name and link_only from parse to parse_country. I know we use the meta function to transfer. Could you please tell me how can I change the meta function below to transfer both items? With the following code, I get blank cells in the CSV file for the link_only column.
In my actual project, I have 7/8 items to transfer from one parse to another. Please suggest me how to make it
import scrapy
import logging
class CountriesSpider(scrapy.Spider):
name = 'countries'
allowed_domains = ['www.worldometers.info']
start_urls = ['https://www.worldometers.info/world-population/population-by-country/']
def parse(self, response):
countries=response.xpath("//td/a")
for country in countries:
name=country.xpath(".//text()").get()
link_only=country.xpath(".//@href/text()").get()
link=country.xpath(".//@href").get()
yield response.follow(url=link, callback=self.parse_country, meta={'country_name': name, 'link_only':link_only})
def parse_country(self, response):
name=response.request.meta['country_name']
link_only=response.request.meta['link_only']
rows = response.xpath("(//table[@class='table table-striped table-bordered table-hover table-condensed table-list'])[1]/tbody/tr")
for row in rows:
year=row.xpath(".//td[1]/text()").get()
population=row.xpath(".//td[2]/strong/text()").get()
yield{
'country_name': name,
'link_only': link_only,
'year': year,
'population': population
}
