Google crawls the net constantly and it takes between 4 days and 4 weeks on average for them to index, or reindex each page. If you've just launched a new website, or even a new blog post, waiting to be indexed can be frustratingly slow.
How can you speed up this process?
There are 4 ways to force Google to quickly index your site for free. The first and second of these will require you to create a Google account (if you've already got a gmail account, that'll do fine).
- Submit URL to Google
- Submit a sitemap to Google
- Build quality links to your page
- Change code that prevents indexing
You may also wish to:
- Submit your website to Bing
- Make other changes to speed up automatic indexing
- Take steps to avoid being deindexed in the future
If you are indexed by any version of Google (UK, USA, Canada, etc.) then you will be indexed by all versions, therefore the following guide applies regardless of which country you're targeting.
Option 1 - Submit URL to Google (One page at a time)
Google used to let you submit any URL (web address) one page at a time, regardless of whether you owned the website or not. This option no longer exists and to submit a URL you must now first either prove that you own the website, or use an indirect method (scroll down to Option 4 if you want to index a page on a website you don't own). We also recommend the following option instead of 'Fetch as Google', as this option has also been removed in Webmaster Tools.
To use this method, you must now follow the steps below:
- Visit https://search.google.com/search-console and login.
- Follow the instructions to add a new website. Make sure to put https:// at the start if you have a secure website and use www if you want the www version of the site to appear in the search results.
If you already have a site registered and want to add a second, click the drop-down arrow, then select "Add Property".
- At the top of the screen there's an option to "Inspect any URL in ..."
- Copy and paste your URL here, press Enter and wait a moment.
- You will receive one of 2 messages:
- "URL is not on Google".
- "URL is on Google" (e.g. where you have changed a page, but want to reindex it).
Either way, after "Page changed?" press "REQUEST INDEXING"
Normally Google will index new pages within one or a few hours.
This method is relatively simple, however requires you to submit one URL at a time, which unless you have a very small site could take a while.
Option 2 - Submit a sitemap to Google
A site map is a file containing a list of all the pages on your website. The main systems, like WordPress, Drupal and Joomla are CMS, which have tools that allow you to create a site map quite easily. If you use one of these CMS, we recommend the following modules:
|CMS||Module / Addon|
|Drupal 8||Simple XML sitemap module|
|Drupal 7||XML sitemap module|
|WordPress||Google XML Sitemaps addon|
In order to create a sitemap, first determine which pages of your site you would like Google to index, as well as the canonical version for each page. You will then need to create the sitemap itself either manually or by using a specific tool. If you're the only one who logs in to your site, you may wish to remove the login page from the sitemap.
At this point, you can test your sitemap to see if the pages are indexed using the tool provided by Google. You can then submit your sitemap to Google via the Search Console. All of Google's relevant tools are accessible with a simple Google Account, which you will already have if you have a Gmail account, Google analytics, etc.
Note that sitemaps cannot be more than 50 Mb per site and not have more than 50,000 links per site. If you ever had a larger file or more URLs, then you will have to divide it into several files that you must submit to Google multiple times, or create a single index file, which in turn redirects to the different components of the sitemap. For more information, follow the instructions here - Submit a sitemap to Google.
Option 3 - Build quality links to your page
If you don't own a website, then it's no longer possible to submit a page to Google. If it's a high traffic website with regular new content, Google is likely to start showing a new page in its search results within a few hours or days anyway, however if the waiting option doesn't appeal to you, one other thing you can do to speed up the process is to build links to the new page. A good starting point is to Tweet about it, or share it on your LinkedIn or other social media profile.
Presumably you're not just looking for a page to appear in the search results, but also to appear near the top of them. If you have more high quality links going to your site, particularly from other relevant blogs, news sites, etc. then your domain's authority will increase and this means that the Google bot will crawl your site on a more regular basis, so they will notice changes sooner and index new pages more quickly.
Options 1 to 3 are all about indexing a web page (getting it included in Google's search results). Building quality links also helps to rank well, as does improving the on-page SEO and fixing any code issues on your site (keep reading for more information on these)
Option 4 - Change code that prevents indexing
If your website, or a URL is still not appearing in Google's index, then there may be an issue with your site itself.
Check the following:
- robots.txt - The robots.txt file determines what URLs the Googlebot can crawl, and which URLs are blocked and this is the most common cause of Google not being able to index a particular URL.
- nofollow tag - This tag literally tells search engines not to show a particular URL in their search results. It appears in the <head> section of a page and will either look like <meta name=”robots” content=”noindex, follow”>, or could be an X-Robots-Tag in the HTTP header: X-Robots-Tag: noindex.
- Content you need to login to see - If you have to login to see content, then Google will not be able to see it. If you're not sure whether you're logged in or not, then try viewing the same web address from a different browser.
- Hashtag in URL - The hashtag # is used to navigate to a different part of the same page, or sometimes to track where visitors have come from. Google therefore also can't index different content that can only be accessed by changing the web address after the # in a URL, so /page1#uk and /page1#canada would both be treated the same by Google.
- Content in a frame - If you use an <iframe> then it picks up content from somewhere else and shows that, and the content within the <iframe> will not be indexed.
How to get indexed by Bing?
Like Google, Bing no longer lets you add your site without an account, however they also crawl the net on a regular basis looking for new content, so will find your site after a few days, particularly if it has some links going to it from elsewhere. The process to follow with Bing is:
- Visit https://www.bing.com/toolbox/webmaster/
- Create your account (you can confirm your identity via a Google account, Facebook, or email address)
- Under "Add a site" enter your site's home page address and click "Add"
- Enter more information about yourself and your site (many of the fields are optional)
- Prove you own the site by choosing one of the verification methods shown and following the instructions there.
Is there anything else I can do to speed up the process?
Yes, Google uses an artificial intelligence that mimics a human being when deciding which websites to index and how high to rank them. There are several things you can do that will make Google consider your website more important and therefore index pages more quickly by default:
- Quality - Google looks for content that's in-depth and valuable to the reader.
- User-experience - Design a website that has a clear user-interface and a mobile-friendly design.
Google's artificial intelligence algorithm is becoming more and more sophisticated and if you neglect to take user-experience and true quality content into account, then you are doomed to fail. It's important to spend as much time focusing on developing your writing skills, as it is developing your SEO skills.
- Navigation - Include a main navigation menu that links to all your most important pages and links in the footer or sidebar to your secondary content.
- Metatags - Google likes every page on a website to have its own unique Meta Title and Meta Description tag. Including a keyword in these, the URL and h1 title will also help you appear for these keywords, and if you write enticing metatags, more people will click them, which also helps you rise in the rankings.
- HTML sitemap - As well as creating a sitemap as detailed above, you can also add a html sitemap that lists all the pages (or for very large sites, all the categories) and add a link to this from the footer.
- New content - If you frequently update your website, then Google will visit it more often by default.
See more tips on how to do SEO with Google RankBrain in mind or contact us using the details below for a free quote to help you improve your position in Google and drive more traffic to your site.
How to avoid being deindexed
John Mueller recently said that Google can deindex a website after only a couple of days, so once you're indexed, it's important to take steps to ensure that you're not deindexed again!
Some simple tips to avoid Google dropping your URL are:
- Ensure you use a reliable host - check out their reviews on a range of websites, as a near 100% up-time is essential.
- Be extra careful when editing the robots.txt file - it's surprisingly easy to edit this file and inadvertently remove your whole site, or a section of it from Google's index.
- Logout as an administrator when doing testing - when you're logged in to WordPress, Drupal or whatever CMS you use, you will have different permissions and most likely be able to see everything, including unpublished pages or a site that's actually offline. Whenever you make major website edits, logout before you start testing, so you're seeing the site as a real end-user will.
- Do regular security updates - Google can deindex a site, or flag a warning message in the search results if you get a virus and regular security updates are the best way to prevent this.
- Avoid spam marketing tactics - If Google perceive that you are doing bulk link building from low quality sites, use automated translations, or have low quality on-page content, these can get your site, or sections of it penalised or even deindexed. Make sure you use a professional SEO strategy focusing on quality content, improved user-experience and building quality backlinks.