Add --allow option to allow more domains to crawl
Currently all crawling is restricted to the initial domain and path. While that is generally good, we should add options to allow links to other domains/paths.
Currently all crawling is restricted to the initial domain and path. While that is generally good, we should add options to allow links to other domains/paths.