Following Selenium, I've uploaded a scraping tutorial using the information-gathering tool BeautifulSoup. This example outputs a product list from the Amazon site to a CSV file. By comparing it with the Selenium edition, I believe you'll gain a deeper understanding of the actual scraping process.
#SideHustle
#BeautifulSoup
#Python
#Programming
0:00:00 Intro
0:00:10 Representative of Scraping
0:02:00 ➀ Avoiding Breakdowns
0:07:36 ② Clean Code with BeautifulSoup
0:13:24 ③ Displaying Images in CSV
0:16:25 Ending
◆ Program Introduced in the Video
* The development environment is PyCharm. It was functioning as of March 2022. Please use the code at your own risk.
【selenium_amazon.py】
import csv
import datetime
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
# URL for scraping (Amazon site search results for Rolex watches)
hdr = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Accept-Encoding': 'none',
'Accept-Language': 'ja-JP,en-US;q=0.7,en-GB;q=0.3',
'Connection': 'keep-alive'}
req = Request(url, headers=hdr)
page = urlopen(req)
soup = BeautifulSoup(page, "lxml")
# Retrieve tags for product clusters (all div elements with class sg-col-inner)
goods_list = soup.find_all('div', {'class': 'sg-col-inner'})
maker_list = soup.findAll('span', {'class': 'a-size-base-plus a-color-base'})
price_list = soup.findAll('span', {'class': 'a-price-whole'})
image_list = soup.findAll('img', {'class': 's-image'})
description_list = soup.findAll('span', {'class': 'a-size-base-plus a-color-base a-text-normal'})
# Uncomment the following comments to see the retrieved elements in the console.
# for maker in maker_list:
# print(maker.string.strip())
# for price in price_list:
# print(price.string.strip())
# for image in image_list:
# print(image.get('src'))
# for description in description_list:
# print(description.string.strip())
# print(len(goods_list))
# print(len(maker_list))
# print(len(price_list))
# print(len(image_list))
# print(len(description_list))
# Output to CSV format
today = datetime.datetime.now().strftime('%Y.%m.%d')
with open('Amazon' + str(today) + '.csv', 'w', encoding='CP932', errors='replace') as f:
writer = csv.writer(f, lineterminator="\n")
writer.writerow(['No', 'Product Name', 'Price', 'Image URL', 'Image', 'Product Description'])
no = 1
previousMaker, previousPrice, previousImage, previousDescription = '', '', '', ''
for goods in goods_list:
try:
maker = goods.find('span', {'class': 'a-size-base-plus a-color-base'}).string.strip()
except:
maker = '[No Maker]'
try:
price = goods.find('span', {'class': 'a-price-whole'}).string.strip()
except:
price = '[No Price]'
try:
image = goods.find('img', {'class': 's-image'}).get('src')
except:
image = '[No Image]'
try:
description = goods.find('span', {'class': 'a-size-base-plus a-color-base a-text-normal'}).string.strip()
except:
description = '[No Description]'
if not ((maker == '[No Maker]' or maker == '')
and price == '[No Price]' and image == '[No Image]'
and description == '[No Description]') and \
not (maker == previousMaker and price == previousPrice
and image == previousImage and description == previousDescription):
writer.writerow([no, maker, price, image, '=IMAGE(D' + str(no + 1) + ')', description])
no += 1
previousMaker = maker
previousPrice = price
previousImage = image
previousDescription = description
This program explains scraping with Python and BeautifulSoup using an example from the Amazon site developed in PyCharm.