web crawler

Example Sentences

Examples are provided to illustrate real-world usage of words in context. Any opinions expressed do not reflect the views of Dictionary.com.

Publishers could also try to protect their content from Google by forbidding its web crawler from sharing any content snippets from their sites.

From New York Times • Jun. 1, 2024

The company originally operated a web crawler that searched for GIFs on the internet.

From Slate • Nov. 30, 2021

He rewrote the code for a slow web crawler that Mr. Page had created to understand the relationship between links on different websites.

From New York Times • Aug. 20, 2021

Bumgarner said the domain on Sony Corporation of America's network where the application was located was carefully hidden from view, so a web crawler or casual surfer would not have located it.

From Reuters • May 13, 2011