What is the main purpose of using a web crawler in the context of penetration testing?
To facilitate the process of downloading all the content from the targeted website for offline access and review.
To provide a real-time user experience analysis for the targeted website's navigation and functionality.
To optimize the website's search engine ranking by suggesting improvements to its structure and content.
To systematically browse and map out the website's pages and links for further reconnaissance and vulnerability discovery.