Bing webmaster indicating crawl errors on http to https 301 redirect

by XIMRX   Last Updated December 16, 2016 08:01 AM - source

I moved my site form http to https, now sitemap contains https urls, but Bing webmaster indicates increased "crawl errors" and my indexed pages have decreased to 0.

The list of error says http pages are redirecting, my question is why is it looking at http pages when I have https pages in sitemap and why it is not indexing only https pages form sitemap.



Answers 2


It sounds like all you did was submit a new site map and not actually implement a valid 301 redirect.

If you are on an Apache server, add this to your htaccess file in the root folder of your site.

A simple approach would be;

#Force www:

RewriteEngine on
RewriteCond %{HTTPS_HOST} ^example.com [NC]
RewriteRule ^(.*)$ https://www.example.com/$1 [L,R=301,NC]

#Force non-www:

RewriteEngine on
RewriteCond %{HTTPS_HOST} ^www\.example\.com [NC]
RewriteRule ^(.*)$ https://example.com/$1 [L,R=301]

I would pick the one that you are currently indexed under the most in searches.

my question is why is it looking at http pages when I have https pages in sitemap

Because you did not specify which to use in their webmaster tools and/or no redirects to tell it otherwise.

Most likely, you have at least 4 versions out there either partially of fully indexed.

http://example.com
https://example.com
http://www.example.com
https://www.example.com

with a possibility of

http://example.com/
https://example.com/
http://www.example.com/
https://www.example.com/
norcal johnny
norcal johnny
December 15, 2016 22:36 PM

...my question is why is it looking at http pages when I have https pages in sitemap...

This is the key.

The answer is simple. Sitemaps are not the authority by which search engines submits pages to the fetch queue from. Their index is! Your pages are indexed as HTTP and therefore that is exactly what will be submitted to the fetch queue. Until each page is requested, redirected, and the URL updated in the index, then, and only then, will the search engine not request HTTP URLs. The exception is, of course, following any existing links to your pages made with HTTP. The search engine will always attempt to fetch pages based upon link URLs even if the same page exists within it's index as HTTPS and not HTTP. That would be the right and responsible thing to do.

The sitemap has nothing to do with this process. Not speaking for Bing, I suspect they are exactly like Google in this respect, Google will only use the sitemap to audit that they can properly crawl your site. Nothing more. Generally, sitemaps direct entries into the fetch queue only when the site is so huge that links to all pages are not possible or when pages exist behind a login or paywall. And then only for those pages. That is it. As far as Google is concerned, sitemaps are not used to feed the fetch queue when a site can properly be crawled.

closetnoc
closetnoc
December 16, 2016 02:17 AM

Related Questions



Bing indexed pages decreasing rapidly.

Updated January 27, 2017 14:01 PM

Bing removed website from indexing

Updated April 23, 2018 18:04 PM


Bing Showing A Wrong Description For My Website Tag

Updated March 15, 2019 17:04 PM