Severe crawling problems

by Nenad   Last Updated April 16, 2018 08:04 AM

I have issue with my website that I need to address ASAP or I'm done :/

The site is now showing increasing numbers of 404s for the url that are there - both in the url and can be fetche and rendered.

The problems started about two weeks ago with the the Divi theme - it wasn't working right. long story short - after my wordpress guy tried to fixed issue I started seeing this 404s. I went to the hosting company and asked them to terurn the site to the state it was in before the 404 problems started.

I submitted a sitemap yesterday and now it seems to be even worse:

  • it shows images like separate 404 urls
  • as i said, it shows existing urls as 404s but renders them

I have no idea what to do, if anybody can help it's gonna be major karma points - i'm desperate at this point.

Thanks in advance

Tags : 404


Answers 1


There could be a plethora of reasons that this is occurring.

The most likely reason is probably that your webhosting provider isn't giving you 100% reliable service. This can happen if you're on shared servers. Other users on your network could be overloading your servers which is crashing yours. It can also happen if you've received a spike in traffic or bots that is overloading the network.

The other issue could be that your htaccess files or database has somehow been corrupted. Though if your pages and images are loading some of the time but not others, this is more likely to be a server and hosting related issue.

To correct the problem, you may have to make the difficult switch to a secondary hosting service and see if that fixes the issue. You can also try to backup your site through cloudflare so that your website loads properly through them. The other option is to reinstall Wordpress while keeping all of your files if you feel confident that important things won't be deleted in the process.

If you are unsure of being able to successfully move Wordpress to another server or being able to reinstall it, you may want to choose to upgrade your servers with your webhost. This is especially the case if you're on a shared server. Getting your own IP address and server computing power could fix your issue.

And before doing any of that, you may want to check your log files first. It is possible that you are undergoing a DDoS attack from bots. If bots start hitting your site many times per second, this could easily crash your system and cause your pages to display a 404 some of the time. The log files on your server will show you IP addresses that are connecting to your site. Try to find IP addresses that belong to unknown bots that are hitting you too much. Ban the bad bots in your htaccess. MAKE ABSOLUTE CERTAIN that these IP addresses do not belong to good bots such as Google and Bing, as they might often be crawling you frequently, and you do not want to ban them..

Michael d
Michael d
April 16, 2018 07:47 AM

Related Questions