User-agent: * Disallow: X-Robots-Tag: noai, notdm HTML meta tag (global header) HTTP header (site configuration) X-Robots-Tag: noai, notdm