title

RFC 9309 - Robots Exclusion Protocol

charset

utf-8

X-UA-Compatible

IE=edge

viewport

width=device-width, initial-scale=1

description

Robots Exclusion Protocol (RFC 9309, )

og:title

RFC 9309: Robots Exclusion Protocol

og:url

https://datatracker.ietf.org/doc/html/rfc9309

og:site_name

IETF Datatracker

og:description

This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.

og:type

article

twitter:card

summary_large_image

alternate

/feed/document-changes/rfc9309/

apple-touch-icon

https://static.ietf.org/dt/12.44.0/ietf/images/ietf-logo-nor-180.png

canonical

https://datatracker.ietf.org/doc/html/rfc9309

icon

https://static.ietf.org/dt/12.44.0/ietf/images/ietf-logo-nor-32.png

icon

https://static.ietf.org/dt/12.44.0/ietf/images/ietf-logo-nor-16.png