Scraping or crawling a large site can generate a large amount of data to store. To avoid duplicating data it can be better to use multiple tables in a database.
Here I show a very simple way to use conditional logic with Python code in a Scrapy spider to send data to different tables based upon the values being scraped. This is intended as an overview for anyone who has already looked at web scraping to a sqlite3 database or similar.
🟢 The perils of using CSV (eg Excel) are also discussed!
The aim of this video is to give you the inspiration to explore the possibilities of using a database, multiple tables, and web scrape in a more structured way, and consider what will be the eventual purpose of your data, and how it might be used/interrogated.
If this has been of interest, the next step will be to learn about SQL, and creating tables.
Thanks for watching.
Dr Pi.
'Illegitimi non carborundum'
def parse_item(self, response):
wsearch = ['train','murder']
lstitle = []
x = response.url
booktitle = str(x)
booktitle = booktitle.split("/")[4]
booktitle = booktitle.split("_")[0]
[lstitle.append(i) for i in booktitle.split("-")]
check = any(item in wsearch for item in lstitle)
if check is True:
myquery = """INSERT INTO links (url,book)
values(%s,%s)
"""
val=(x,booktitle)
cursor.execute(myquery,val)
mydb.commit()
else:
myquery = """INSERT INTO links2 (url,book)
values(%s,%s)
"""
val=(x,booktitle)
cursor.execute(myquery,val)
mydb.commit()
#webscraping #sqlite #database