{"id":4045,"date":"2021-09-21T11:00:30","date_gmt":"2021-09-21T11:00:30","guid":{"rendered":"https:\/\/www.adspyglass.com\/blog\/glossary-crawl-budget\/"},"modified":"2021-09-28T15:18:56","modified_gmt":"2021-09-28T15:18:56","slug":"crawl-budget","status":"publish","type":"post","link":"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/","title":{"rendered":"Crawl Budget"},"content":{"rendered":"<p><span>Crawl budget is the number of pages that search engines can crawl and index within a certain timeframe. If you have exceeded the crawl budget and some pages were not indexed, they will not appear on the search engine results pages.&nbsp;<\/span><\/p>\n<p><span>It is rare that websites exceed the budget though. It may become a problem only if:<\/span><\/p>\n<ul>\n<li><span>you own a really big website<\/span><\/li>\n<li><span>you have just added too many pages<\/span><\/li>\n<li><span>you have too many redirects<\/span><\/li>\n<\/ul>\n<h2>Tips for Not Exceeding the Crawl Budget<\/h2>\n<p><span>If you think that you may have to worry about exceeding the crawl budget, you can use the following tactics:<\/span><\/p>\n<h3>Reduce the Page Loading Time<\/h3>\n<p><span>Let&rsquo;s imagine that search engine crawlers gave your website 15 minutes. If your page loads in under a second, there are around 900 pages that a search engine can go through. If it takes more than 3 seconds to load, it is 3 times fewer pages to be indexed.&nbsp;<\/span><\/p>\n<p><span>You can reduce loading time by reducing the size of images, optimizing the cache, applying AJAX, using light social buttons, etc.&nbsp;<\/span><\/p>\n<h3>Use Internal Linking<\/h3>\n<p><span>Make sure that crawlers can discover all of your pages by using external backlinks and when the page did not receive any, internal ones. Avoid dead-end pages.&nbsp;<\/span><\/p>\n<h3>Avoid Duplicate Content<\/h3>\n<p><span>Search engines crawlers will not waste their time crawling the page with the same content. It means that some of your older pages may receive more attention from crawlers than fresh updated ones. Make sure to delete old pages that are not up-to-date.&nbsp;<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Crawl budget is the number of pages that search engines can crawl and index within a certain timeframe. If you have exceeded the crawl budget and some pages were not indexed, they will not appear on the search engine results pages.&nbsp; It is rare that websites exceed the budget though. It may become a problem [&hellip;]<\/p>\n","protected":false},"author":13,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[93,96],"tags":[123,380,379,381],"class_list":["post-4045","post","type-post","status-publish","format-standard","hentry","category-glossary","category-c","tag-adspyglass-glossary","tag-crawl-budget-definition","tag-crawl-budget-term","tag-what-is-crawl-budget"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Crawl Budget<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Crawl Budget\" \/>\n<meta property=\"og:description\" content=\"Crawl budget is the number of pages that search engines can crawl and index within a certain timeframe. If you have exceeded the crawl budget and some pages were not indexed, they will not appear on the search engine results pages.&nbsp; It is rare that websites exceed the budget though. It may become a problem [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/\" \/>\n<meta property=\"og:site_name\" content=\"AdSpyglass\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/adspyglass\/\" \/>\n<meta property=\"article:published_time\" content=\"2021-09-21T11:00:30+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-09-28T15:18:56+00:00\" \/>\n<meta name=\"author\" content=\"zerg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AdSpyglass\" \/>\n<meta name=\"twitter:site\" content=\"@AdSpyglass\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"zerg\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawl-budget\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawl-budget\\\/\"},\"author\":{\"name\":\"zerg\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#\\\/schema\\\/person\\\/381353662ed5ec215106b146822869fb\"},\"headline\":\"Crawl Budget\",\"datePublished\":\"2021-09-21T11:00:30+00:00\",\"dateModified\":\"2021-09-28T15:18:56+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawl-budget\\\/\"},\"wordCount\":258,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#organization\"},\"keywords\":[\"adspyglass glossary\",\"crawl budget definition\",\"crawl budget term\",\"what is crawl budget\"],\"articleSection\":[\"AdSpyglass Webmaster's Glossary\",\"C\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawl-budget\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawl-budget\\\/\",\"url\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawl-budget\\\/\",\"name\":\"Crawl Budget\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#website\"},\"datePublished\":\"2021-09-21T11:00:30+00:00\",\"dateModified\":\"2021-09-28T15:18:56+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawl-budget\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawl-budget\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawl-budget\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog\",\"item\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"AdSpyglass Webmaster's Glossary\",\"item\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/glossary\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"C\",\"item\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/glossary\\\/c\\\/\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Crawl Budget\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/\",\"name\":\"AdSpyglass\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#organization\",\"name\":\"Adspyglass\",\"url\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/10\\\/logo-asg-1.svg\",\"contentUrl\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/10\\\/logo-asg-1.svg\",\"width\":\"1024\",\"height\":\"1024\",\"caption\":\"Adspyglass\"},\"image\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/adspyglass\\\/\",\"https:\\\/\\\/x.com\\\/AdSpyglass\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCjwWrtCc1UwDil7ZRwiGZDg\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#\\\/schema\\\/person\\\/381353662ed5ec215106b146822869fb\",\"name\":\"zerg\",\"url\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/author\\\/zerg\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Crawl Budget","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/","og_locale":"en_US","og_type":"article","og_title":"Crawl Budget","og_description":"Crawl budget is the number of pages that search engines can crawl and index within a certain timeframe. If you have exceeded the crawl budget and some pages were not indexed, they will not appear on the search engine results pages.&nbsp; It is rare that websites exceed the budget though. It may become a problem [&hellip;]","og_url":"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/","og_site_name":"AdSpyglass","article_publisher":"https:\/\/www.facebook.com\/adspyglass\/","article_published_time":"2021-09-21T11:00:30+00:00","article_modified_time":"2021-09-28T15:18:56+00:00","author":"zerg","twitter_card":"summary_large_image","twitter_creator":"@AdSpyglass","twitter_site":"@AdSpyglass","twitter_misc":{"Written by":"zerg","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/#article","isPartOf":{"@id":"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/"},"author":{"name":"zerg","@id":"https:\/\/www.adspyglass.com\/blog\/#\/schema\/person\/381353662ed5ec215106b146822869fb"},"headline":"Crawl Budget","datePublished":"2021-09-21T11:00:30+00:00","dateModified":"2021-09-28T15:18:56+00:00","mainEntityOfPage":{"@id":"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/"},"wordCount":258,"commentCount":0,"publisher":{"@id":"https:\/\/www.adspyglass.com\/blog\/#organization"},"keywords":["adspyglass glossary","crawl budget definition","crawl budget term","what is crawl budget"],"articleSection":["AdSpyglass Webmaster's Glossary","C"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.adspyglass.com\/blog\/crawl-budget\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/","url":"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/","name":"Crawl Budget","isPartOf":{"@id":"https:\/\/www.adspyglass.com\/blog\/#website"},"datePublished":"2021-09-21T11:00:30+00:00","dateModified":"2021-09-28T15:18:56+00:00","breadcrumb":{"@id":"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.adspyglass.com\/blog\/crawl-budget\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.adspyglass.com\/blog\/crawl-budget\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog","item":"https:\/\/www.adspyglass.com\/blog\/"},{"@type":"ListItem","position":2,"name":"AdSpyglass Webmaster's Glossary","item":"https:\/\/www.adspyglass.com\/blog\/glossary\/"},{"@type":"ListItem","position":3,"name":"C","item":"https:\/\/www.adspyglass.com\/blog\/glossary\/c\/"},{"@type":"ListItem","position":4,"name":"Crawl Budget"}]},{"@type":"WebSite","@id":"https:\/\/www.adspyglass.com\/blog\/#website","url":"https:\/\/www.adspyglass.com\/blog\/","name":"AdSpyglass","description":"","publisher":{"@id":"https:\/\/www.adspyglass.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.adspyglass.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.adspyglass.com\/blog\/#organization","name":"Adspyglass","url":"https:\/\/www.adspyglass.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.adspyglass.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.adspyglass.com\/blog\/wp-content\/uploads\/2021\/10\/logo-asg-1.svg","contentUrl":"https:\/\/www.adspyglass.com\/blog\/wp-content\/uploads\/2021\/10\/logo-asg-1.svg","width":"1024","height":"1024","caption":"Adspyglass"},"image":{"@id":"https:\/\/www.adspyglass.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/adspyglass\/","https:\/\/x.com\/AdSpyglass","https:\/\/www.youtube.com\/channel\/UCjwWrtCc1UwDil7ZRwiGZDg"]},{"@type":"Person","@id":"https:\/\/www.adspyglass.com\/blog\/#\/schema\/person\/381353662ed5ec215106b146822869fb","name":"zerg","url":"https:\/\/www.adspyglass.com\/blog\/author\/zerg\/"}]}},"views":27,"_links":{"self":[{"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/posts\/4045","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/comments?post=4045"}],"version-history":[{"count":1,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/posts\/4045\/revisions"}],"predecessor-version":[{"id":4967,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/posts\/4045\/revisions\/4967"}],"wp:attachment":[{"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/media?parent=4045"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/categories?post=4045"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/tags?post=4045"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}