What is the main purpose of using a web crawler in the context of penetration testing?
To optimize the website's search engine ranking by suggesting improvements to its structure and content.
To facilitate the process of downloading all the content from the targeted website for offline access and review.
To systematically browse and map out the website's pages and links for further reconnaissance and vulnerability discovery.
To provide a real-time user experience analysis for the targeted website's navigation and functionality.