Create an account to use AI

Create a free account to unlock all features:

Ask questions about videos with AI
Use smart search (AI)
Save your favorite videos
Bookmark your favorite categories
Organize your favorite subcategories
Access your personalized favorites page
Access your conversation history
Personalize your homepage with your favorites

Create an account

Already have an account? Log in

Scrapy selectors : XPATH 'contains' and 'starts-with' | Web Scraping Tips

jobstr

Web scraping using Scrapy and Python - Some tips you may find useful.

Scrapy lets you use CSS or XPATH for the selectors, and here we look at how powerful XPATH can be when used with "contains" and "starts-with" inside the predicate to pick out parts of the page that we need to scrape.

Another benefit is that if the CSS changes on the web page our code may still pick out the parts we need.

See my code at : https://github.com/RGGH/Scrapy5

eg.

response.xpath("//a[starts-with(@href, 'https://www.yelp')]")

response.xpath('//*[contains(@href,"biz")]/@href')

Credit : Thanks to Aaron S for the suggestion/tip.

Dr P.

Scrapy selectors : XPATH 'contains' and 'starts-with' | Web Scraping Tips | Knoovi