If you are seeing this page, you are probably following up on the OCW indexing crawler. A web crawler, also known as a bot or robot, retrieves web pages, and in the case of OCW Search, the pages it retrieves are indexed to power the search engine.
The indexing crawler is rate limited to limit any stress it may cause on your servers. Usually, the time between requests is 5-20 seconds.
If you would like to remove your website from OCW Search, please contact us.