Top-notch Web Design

See Our Most Recent Web Design Work

Lease A Website

Save With Innovative Website Lease Deals!

Contact Us

581 Boylston St. Suite 604
Boston, MA 02116
Tel: (617) 535-7653

How to write correct robots.txt

Robots.txt is a text file containing instructions for search engines with the aim of site indexing. It allows you to disable the indexing of pages or entire sections of the site, to indicate the correct mirror of the domain, setting the way to the sitemap. This is far not the whole list of options of robots.txt file, but only those that are most often used.

Before indexing the site, search engines examine robots.txt. Therefore the inclusion of this file is highly important, let alone the importance of its content. Please note that all the commands contained in robots.txt file are not required for operation.

Robots.txt is a plain text file which is created using any text editor and is placed in the root directory of the site. It is important that the file name is exactly in the lower register, that is to say, the name Robots.txt or ROBOTS.TXT is incorrect.

While creating a robots.txt file take the syntax of its writing into a thorough consideration. The standard for writing guidelines to follow the robot was developed early in 1994 and since then it has not changed. Therefore, most bots contain commands which are out of standard.

Basic commands for robots.txt:

1. User-Agent specifies the name of the robot, to which the set of commands refers to. If the instruction set is designed for all bots, then mention asterisk instead of the name (User-agent: *).

2. Disallow and Allow denies or allows access to certain pages on the site. Note that Allow option is out of standard, and despite this it is supported by Google.

3. Host directive is supported only by SE showing which mirror of the site should be prioritized. Under the term mirror site domains with the prefix www and without it is meant. The domain should be set without the abbreviation of protocol http:// excluding the closing slash. This directive must be specified after all the Disallow commands.

4. Sitemap reports on the availability of the sitemap and shows the way to it. This directive is cross-sectional.

Rules for writing instructions for the file robots.txt:

ai??? Each line with Disallow instructions must include only one file or directory

ai??? The file name must be in the register at the bottom

ai??? User-agent line cannot be left empty. To access all the robots, use an asterisk (*)

ai??? No substitution symbols, like Disallow: file *. html., should be used in Disallow directive

ai??? Comments on the instructions should be written in a single line

ai??? Disallow instruction is considered to be mandatory. So, if there is nothing to be disallowed, just leave this field empty

ai??? While banning the indexing of directories be sure to use slashes (Disallow: / folder /)

ai??? Blank lines are used only to separate sections

To avoid content duplication on WordPress blogs, the following lines may be closed for indexing

ai??? Tag pages: Disallow: / tag /

ai??? Archive pages: Disallow: / archives /

ai??? Category pages: Disallow: / category /

It is up to you to close these pages or not. After all, each of these pages may be operated based on a key request. And if you sell links on stock exchanges, such pages can generate a good income.


No comments yet.

Add a comment

* mandatory fields


! Без рубрики 1
500 cbd 1
Academic Essay Writers 1
Academic Writing 2018 1
article 3
Best Essay Writer of 2018 for students 5
Best Essay Writers 1
Best Essays 1
Best place for meeting 1
blog 46
bride blog 1
bride masters 1
Business 1
c 1
Clinics 1
College News 1
crypto news 1
Cryptocurrencies 1
Custom Essay 1
Custom Paper Writers 1
Custom Writers Reviews 1
Custom Writings 1
dissertation paper for sale online 1
essay 8
Essay Editor Help 1
essay papers writing service 2
Essay Samples 1
Essay Service 2018 for Students 15
Essay Solutions 1
Essay Writers Ranking 1
Essays to purchase 1
Featured Website 73
Finding The Best Dating Site For You 1
Free Essay Samples 1
Greetings 1
Hello world 2
hello world 2018 1
Homework Assignment 1
Homework Tips 1
ico 1
Local Marketing 3
Man by Finding Yourself First 1
MMJ Card 1
mob top review sites 1
new 1
News 2018 1
Online dating 1
Online Essay Editing 1
Online Homework Services 1
Order Essay Online 1
Paper Writing Service 1
Plagiarism checker for students 1
Qualified Essay Writers 1
redy_articles 1
Resume Writing 1
Retail prices 1
Ripple 1
SEO 63
Social Media 25
soderjanki online 1
Sports Activity 1
sugar daddy russia 1
Term Paper Writers 1
top sugar daddy content 1
true hookup tips 1
Uncategorised 4
Uncategorized 1918
Video 5
Virtual data storage 1
Web Design 55
Web Development 69
writers for hire 1
writing 12
Writing Services 1
Writing Tips For Students 1
Writiting Sites Rating 1
youtube 2
Высшее образование в Австралии 1
Высшее образование в Германии 1