World wide web is just like a network of computers connected via a huge wide area network called internet, this internet allows you to share the information between the information saved on connected computers, In the same way search engine bots travels the web, Web is just like a network of websites with links to other sites in this manner when search engine robot visit one website it also visit the other site that the crawled website is linking in this way when search bots crawl a directory like Dmoz or Yahoo it crawls the websites in massive manner and in this manner sites got indexed
Content by - RRWDG search engine optimization