Which robit txt corerct - User-agent: * Disallow or User-agent: * Disallow: /wp-admin/…
- User
Which robit txt corerct - User-agent: * Disallow or User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/themes/ Disallow: /readme.html Disallow: /refer/ Disallow: /cgi-bin/
Sitemap: https://seowriter.in/sitemap.xmlFor better search ranking
- Assistant
The second
robots.txtconfiguration is more comprehensive and is likely to contribute better towards managing how web crawlers index your site and hence potentially improve search rankings. Here's a closer look at the two options:-
Simple Disallow All:
User-agent: * DisallowThis configuration essentially tells all web crawlers that they are not restricted from accessing any part of the site. This may be suitable in cases where you want all parts of your website indexed, but it doesn’t provide the fine-grained controls necessary for optimizing search engine ranking.
-
Comprehensive Configuration with Rules:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/themes/ Disallow: /readme.html Disallow: /refer/ Disallow: /cgi-bin/ Sitemap: https://seowriter.in/sitemap.xmlThis configuration provides a detailed set of rules for web crawlers:
User-agent: *: Applies these rules to all web crawlers.Disallow: /wp-admin/: Prevents crawlers from indexing the WordPress admin area, except for:Allow: /wp-admin/admin-ajax.php: Allows access toadmin-ajax.phpwhich might be used for AJAX calls.Disallow: /wp-includes/,/wp-content/plugins/,/wp-content/themes/,/readme.html,/refer/,/cgi-bin/: Prevents crawlers from accessing these directories, which are not typically relevant for indexing content.Sitemap: https://seowriter.in/sitemap.xml: Provides the URL of the sitemap to help crawlers understand the structure of your site.
To summarize: The second
robots.txtconfiguration is better for optimizing how search engines index your site. It restricts access to non-essential areas and directs crawlers to the sitemap, likely resulting in more efficient and effective indexing. -