{"id":4047,"date":"2021-09-21T11:00:30","date_gmt":"2021-09-21T11:00:30","guid":{"rendered":"https:\/\/www.adspyglass.com\/blog\/glossary-crawling\/"},"modified":"2021-09-28T15:18:58","modified_gmt":"2021-09-28T15:18:58","slug":"crawling","status":"publish","type":"post","link":"https:\/\/www.adspyglass.com\/blog\/crawling\/","title":{"rendered":"Crawling"},"content":{"rendered":"<p><span>&nbsp;Crawling refers to the process in which a program or bot (such as search engine bots) browse through the web, starting from known websites in search of new or updated content. The content can vary from web pages to videos, images, etc., but regardless, they&rsquo;re discovered by the bots through links.<\/span><\/p>\n<p><span>Google bots, for instance, start the process with several known web pages and find new URLs by following the links on these pages. As they follow these links, they find new content, which they add to their index to be retrieved later when a user seeks information related to the content. Google has a seed list &ndash; a list of trusted sites with links to many other websites &ndash; where the bot starts the crawling process.<\/span><\/p>\n<p><span>Internet crawling is an unending process for search engines. They have to find newly published pages and old pages that are updated. Yet, they don&rsquo;t want to spend their resources and time on web pages that aren&rsquo;t good enough for search results. So, there&rsquo;s a priority for crawling pages.<\/span><\/p>\n<p><span>Google prioritizes this process based on:<\/span><\/p>\n<ul>\n<li><span>Page popularity (how often it is linked)<\/span><\/li>\n<li><span>High-quality content<\/span><\/li>\n<li><span>Frequent updates<\/span><\/li>\n<\/ul>\n<p><span>Newly published websites with high-quality content are given more priority.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp;Crawling refers to the process in which a program or bot (such as search engine bots) browse through the web, starting from known websites in search of new or updated content. The content can vary from web pages to videos, images, etc., but regardless, they&rsquo;re discovered by the bots through links. Google bots, for instance, [&hellip;]<\/p>\n","protected":false},"author":13,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[93,96],"tags":[123,386,385,387],"class_list":["post-4047","post","type-post","status-publish","format-standard","hentry","category-glossary","category-c","tag-adspyglass-glossary","tag-crawling-definition","tag-crawling-term","tag-what-is-crawling"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Crawling<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.adspyglass.com\/blog\/crawling\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Crawling\" \/>\n<meta property=\"og:description\" content=\"&nbsp;Crawling refers to the process in which a program or bot (such as search engine bots) browse through the web, starting from known websites in search of new or updated content. The content can vary from web pages to videos, images, etc., but regardless, they&rsquo;re discovered by the bots through links. Google bots, for instance, [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.adspyglass.com\/blog\/crawling\/\" \/>\n<meta property=\"og:site_name\" content=\"AdSpyglass\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/adspyglass\/\" \/>\n<meta property=\"article:published_time\" content=\"2021-09-21T11:00:30+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-09-28T15:18:58+00:00\" \/>\n<meta name=\"author\" content=\"zerg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AdSpyglass\" \/>\n<meta name=\"twitter:site\" content=\"@AdSpyglass\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"zerg\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawling\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawling\\\/\"},\"author\":{\"name\":\"zerg\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#\\\/schema\\\/person\\\/381353662ed5ec215106b146822869fb\"},\"headline\":\"Crawling\",\"datePublished\":\"2021-09-21T11:00:30+00:00\",\"dateModified\":\"2021-09-28T15:18:58+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawling\\\/\"},\"wordCount\":213,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#organization\"},\"keywords\":[\"adspyglass glossary\",\"crawling definition\",\"crawling term\",\"what is crawling\"],\"articleSection\":[\"AdSpyglass Webmaster's Glossary\",\"C\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawling\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawling\\\/\",\"url\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawling\\\/\",\"name\":\"Crawling\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#website\"},\"datePublished\":\"2021-09-21T11:00:30+00:00\",\"dateModified\":\"2021-09-28T15:18:58+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawling\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawling\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/crawling\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog\",\"item\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"AdSpyglass Webmaster's Glossary\",\"item\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/glossary\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"C\",\"item\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/glossary\\\/c\\\/\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Crawling\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/\",\"name\":\"AdSpyglass\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#organization\",\"name\":\"Adspyglass\",\"url\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/10\\\/logo-asg-1.svg\",\"contentUrl\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/10\\\/logo-asg-1.svg\",\"width\":\"1024\",\"height\":\"1024\",\"caption\":\"Adspyglass\"},\"image\":{\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/adspyglass\\\/\",\"https:\\\/\\\/x.com\\\/AdSpyglass\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCjwWrtCc1UwDil7ZRwiGZDg\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/#\\\/schema\\\/person\\\/381353662ed5ec215106b146822869fb\",\"name\":\"zerg\",\"url\":\"https:\\\/\\\/www.adspyglass.com\\\/blog\\\/author\\\/zerg\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Crawling","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.adspyglass.com\/blog\/crawling\/","og_locale":"en_US","og_type":"article","og_title":"Crawling","og_description":"&nbsp;Crawling refers to the process in which a program or bot (such as search engine bots) browse through the web, starting from known websites in search of new or updated content. The content can vary from web pages to videos, images, etc., but regardless, they&rsquo;re discovered by the bots through links. Google bots, for instance, [&hellip;]","og_url":"https:\/\/www.adspyglass.com\/blog\/crawling\/","og_site_name":"AdSpyglass","article_publisher":"https:\/\/www.facebook.com\/adspyglass\/","article_published_time":"2021-09-21T11:00:30+00:00","article_modified_time":"2021-09-28T15:18:58+00:00","author":"zerg","twitter_card":"summary_large_image","twitter_creator":"@AdSpyglass","twitter_site":"@AdSpyglass","twitter_misc":{"Written by":"zerg","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.adspyglass.com\/blog\/crawling\/#article","isPartOf":{"@id":"https:\/\/www.adspyglass.com\/blog\/crawling\/"},"author":{"name":"zerg","@id":"https:\/\/www.adspyglass.com\/blog\/#\/schema\/person\/381353662ed5ec215106b146822869fb"},"headline":"Crawling","datePublished":"2021-09-21T11:00:30+00:00","dateModified":"2021-09-28T15:18:58+00:00","mainEntityOfPage":{"@id":"https:\/\/www.adspyglass.com\/blog\/crawling\/"},"wordCount":213,"commentCount":0,"publisher":{"@id":"https:\/\/www.adspyglass.com\/blog\/#organization"},"keywords":["adspyglass glossary","crawling definition","crawling term","what is crawling"],"articleSection":["AdSpyglass Webmaster's Glossary","C"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.adspyglass.com\/blog\/crawling\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.adspyglass.com\/blog\/crawling\/","url":"https:\/\/www.adspyglass.com\/blog\/crawling\/","name":"Crawling","isPartOf":{"@id":"https:\/\/www.adspyglass.com\/blog\/#website"},"datePublished":"2021-09-21T11:00:30+00:00","dateModified":"2021-09-28T15:18:58+00:00","breadcrumb":{"@id":"https:\/\/www.adspyglass.com\/blog\/crawling\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.adspyglass.com\/blog\/crawling\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.adspyglass.com\/blog\/crawling\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog","item":"https:\/\/www.adspyglass.com\/blog\/"},{"@type":"ListItem","position":2,"name":"AdSpyglass Webmaster's Glossary","item":"https:\/\/www.adspyglass.com\/blog\/glossary\/"},{"@type":"ListItem","position":3,"name":"C","item":"https:\/\/www.adspyglass.com\/blog\/glossary\/c\/"},{"@type":"ListItem","position":4,"name":"Crawling"}]},{"@type":"WebSite","@id":"https:\/\/www.adspyglass.com\/blog\/#website","url":"https:\/\/www.adspyglass.com\/blog\/","name":"AdSpyglass","description":"","publisher":{"@id":"https:\/\/www.adspyglass.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.adspyglass.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.adspyglass.com\/blog\/#organization","name":"Adspyglass","url":"https:\/\/www.adspyglass.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.adspyglass.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.adspyglass.com\/blog\/wp-content\/uploads\/2021\/10\/logo-asg-1.svg","contentUrl":"https:\/\/www.adspyglass.com\/blog\/wp-content\/uploads\/2021\/10\/logo-asg-1.svg","width":"1024","height":"1024","caption":"Adspyglass"},"image":{"@id":"https:\/\/www.adspyglass.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/adspyglass\/","https:\/\/x.com\/AdSpyglass","https:\/\/www.youtube.com\/channel\/UCjwWrtCc1UwDil7ZRwiGZDg"]},{"@type":"Person","@id":"https:\/\/www.adspyglass.com\/blog\/#\/schema\/person\/381353662ed5ec215106b146822869fb","name":"zerg","url":"https:\/\/www.adspyglass.com\/blog\/author\/zerg\/"}]}},"views":27,"_links":{"self":[{"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/posts\/4047","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/comments?post=4047"}],"version-history":[{"count":1,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/posts\/4047\/revisions"}],"predecessor-version":[{"id":4969,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/posts\/4047\/revisions\/4969"}],"wp:attachment":[{"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/media?parent=4047"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/categories?post=4047"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.adspyglass.com\/blog\/wp-json\/wp\/v2\/tags?post=4047"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}