Urlgrab – A Golang Utility To Spider Through A Website Searching For Additional Links

A golang utility to spider through a website searching for additional links with support for JavaScript rendering. Install go get -u github.com/iamstoxe/urlgrab Features Customizable Parallelism Ability to Render JavaScript (including Single Page Applications such as Angular and React) Usage Usage of urlgrab: -cache-dir string Specify a directory to utilize caching. Works between sessions as well.Read More

Gospider – Fast Web Spider Written In Go

GoSpider – Fast web spider written in Go Installation go get -u github.com/jaeles-project/gospider Features Fast web crawling Brute force and parse sitemap.xml Parse robots.txt Generate and verify link from JavaScript files Link Finder Find AWS-S3 from response source Find subdomains from response source Get URLs from Wayback Machine, Common Crawl, Virus Total, Alien Vault FormatRead More