Channel
Listed newest first. Channel on YouTube
Teaching or conference talk
Web crawlers are responsible for a large share of internet traffic, according to this lecture, which explains how they work, why crawling is hard and gives a short look at Nutch.
Older: check current guidanceTeaching or conference talk
Why is web crawling hard? This brief talk goes through being polite to servers, keeping data fresh, covering enough pages, handling scale and dodging traps.
Older: check current guidanceTeaching or conference talk
PageRank and HITS are two classic ways to score how important a web page is, and this lecture works through both, including the random surfer idea and Markov chains.
Older: check current guidance
