A standard which is used to define data associated with a Web site. Text using this standard describes the access privileges of robots visiting a site; for example, the text could forbid all access to the Web site by robots or restrict it to a small number of pages.