检查robots.txt
如https://www.baidu.com/robots.txt

pip install builtwith
pip install python-whois
pip install urllib2
#python3找不到urllib2
pip install requests
pip install selenium
pip install aiohttp