The ‘GA Not Matched’ report has been replaced with the new ‘GA & GSC Not Matched Report’ which now provides consolidated information on URLs discovered via the Google Search Analytics API, as well as the Google Analytics API, but were not found in the crawl. Therefore there is an argument that they can act as a bit of a dead end, so I’d recommend reviewing just how many are being disallowed, how well linked they are, and their depth for example. Google just can’t crawl the content of the page itself, or see the outlinks of the URL to pass the PageRank onwards. URLs which are linked to internally (or externally), but are blocked by robots.txt can obviously accrue PageRank, be indexed and appear under search. If you’d prefer to not see URLs blocked by robots.txt in the crawl, then simply untick the relevant boxes. We have therefore introduced it as an optional configuration, for both internal and external URLs in a crawl. However, there are plenty of scenarios where using robots.txt to control crawling and understanding quickly what URLs are blocked by robots.txt is valuable, and it’s something that has been requested by users over the years. I always felt that it wasn’t required as users should know already what URLs are being blocked, and whether robots.txt should be ignored in the configuration. Historically the SEO Spider hasn’t shown URLs that are disallowed by robots.txt in the interface (they were only available via the logs). This should make auditing robots.txt files simple! The ‘Blocked by Robots.txt’ filter also displays a ‘Matched Robots.txt Line’ column, which provides the line number and disallow path of the robots.txt entry that’s excluding each URL. You can now view URLs disallowed by the robots.txt protocol during a crawl.ĭisallowed URLs will appear with a ‘status’ as ‘Blocked by Robots.txt’ and there’s a new ‘Blocked by Robots.txt’ filter under the ‘Response Codes’ tab, where these can be viewed efficiently. 2) View & Audit URLs Blocked By Robots.txt We plan to extend our integration further as well, but at the moment the Search Console API is fairly limited. The API is currently limited to 5k rows of data, which we hope Google will increase over time. Remember, you might see URLs appear here which are ‘noindex’ or ‘canonicalised’, unless you have ‘ respect noindex‘ and ‘ respect canonicals‘ ticked in the advanced configuration tab. In the example above, we can see the URLs appearing under the ‘No GSC Data’ filter are all author pages, which are actually ‘noindex’, so this is as expected. There’s a couple of filters currently for ‘Clicks Above 0’ when a URL has at least a single click, and ‘No GSC Data’, when the Google Search Analytics API did not return any data for the URL. When you hit ‘Start’ and the API progress bar has reached 100%, data will appear in real time during the crawl under the ‘Search Console’ tab, and dynamically within columns at the far right in the ‘Internal’ tab if you’d like to export all data together. Similar again to our GA integration, we have some common URL matching scenarios covered, such as matching trailing and non trailing slash URLs and case sensitivity.
You can then select the relevant site profile, date range, device results (desktop, tablet or mobile) and country filter. The Search Analytics API doesn’t provide us with the account name in the same way as the Analytics integration, so once connected it will appear as ‘New Account’, which you can rename manually for now. You just need to give permission to our app to access data under ‘Configuration > API Access > Google Search Console’ – We were part of the Search Analytics beta, so have had this for some time internally, but delayed the release a little, while we finished off a couple of other new features detailed below, for a larger release.įor those already familiar with our Google Analytics integration, the set-up is virtually the same. Alongside Google Analytics integration, this should be valuable for Panda and content audits respectively.
You can now connect to the Google Search Analytics API and pull in impression, click, CTR and average position data from your Search Console profile. Let’s get straight to it, version 5.0 includes the following new features – 1) Google Search Analytics Integration In July we released version 4.0 (and 4.1) of the Screaming Frog SEO Spider, and I am pleased to announce the release of version 5.0, codenamed internally as ‘toothache’.
0 kommentar(er)
