This is Carolyn Holzman and Welcome to Report 64 of the Crawl or No Crawl Report
April 19 2022 Semrush Volatility Sensor is reading 8.0 - very. High
All of the data in this project is being logged and made public. Today is day 236 of the indexation research project.
There’s a link to the History log
Or you can search for google index detector
So Let’s play Crawl or No Crawl.
Today is a test day-
The test page published at 7:31 AM - requested indexing at 7:31
Google bot arrived at 7:31 and pulled the robots.txt file
THEN it came back at 7:31 and 40 seconds to pull the html
We now have three pages that have been published subsequent to the search console coverage update on April 15th very early in the morning.
We now have a very high volatility reading on one the seo weather tools used.
Based on previous data this typically ends up with a late night/early morning search console coverage update.
I think at that time we’ll see all the pages that were published after the last update on the 15th and prior to next coverage date will show up in the index.
I suspect the coverage update will occur between today and tomorrow very early in the morning.
I think at that time the coverage will show an update from the 15th to the 19th and any pages published between the 15th and the 19th will show as indexed by their keywords - which means two of the three published test pages that are still not indexed yet. So today’s published page may not be in that group but if they go from the 15th to the 20th - then it will.
This has my mind working on a theory
I will definitely want to pull this data apart and take a look. I will be doing a webinar next week - the week of the 25th - few more things to work out and here is going to be the first place I announce it -
As I mentioned yesterday Here’s what I’m pondering now -
I have a strong sense that search console provides more of a peek into Google’s supplemental index than we really appreciate. I’m not completely sold that this is an update yet - something was reported in SE Roundtable on the 18th
Will the test page published on the 15th show in the index before Google updates the search console again? So far no.
OK now to the bots
Only one of the desktop bots came today
10 Desktop bot 1 - crawls the directives
0 of Desktop bot 2 - refresh render crawls
Google mobile bots - 15 - 1 render crawls -
With the weather tool showing high volatility - I checked if there were any time frames between yesterday and today - where there was a long absence of bots -
Found it between 2am and 7:32 am this morning.
16 Feed fetcher bots
Bing -
There were 60 records - all desktop
Like previous days 3 different Bing IPS working so far today
And I did see signs of a strong division of labor
One crawled directive files - sitemaps and robots text
One performed rending crawls
One did both
1 bing preview
No DUCKDUCKGO favicon bots
That’s it for today.
Please subscribe.
This week’s Confessions of an SEO - Expectations Managed - there is no accounting for the amount or nature of unspoken expectations when it comes to SEO clients and I suspect there is on major contributing factor - find out on this week’s episode Friday at 8 am CST -
Check out Friday’s Confessions of an SEO - it’s like Confessions with a Buddy - Ted Kubaitis and the Disappearing URL Parameter Tool that is being shutdown on April 26 - T-minus 6 days
For those of us in Forensic SEO - this could be very enlightening
Thanks for adding this to your day and I’ll see you tomorrow
Https://www.facebook.com/AmericanWayMedia