Men and women use DA to determine which Web-sites are reliable inside their area and to determine the place to receive great inbound links. txt file is then parsed and may instruct the robotic as to which web pages usually are not to generally be crawled. To be a search https://www.youtube.com/watch?v=7cBHAtTOxoI