Social Media Influence: Popular trends on social networking can spill around into engines like google, impacting the lookup volume of relevant keywords and phrases. txt file is then parsed and will instruct the robotic regarding which internet pages will not be being crawled. As being a online search engine crawler https://www.youtube.com/watch?v=1amb7r2WR1g