一、Python爬虫基础知识介绍
1.1 Python相关库
1、requests、re
2、BeautifulSoup
3、hackhttp
1.2 Python BeautifulSoup
Python BeautifulSoup模块的使用介绍∶
1、解析内容
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc)
2、浏览数据
soup.title
soup.title.string
3、BeautifulSoup正则使用
soup.find_all(name= ‘x’,attrs={‘xx’:re.compile(‘xxx’)})
二、Python爬虫简单实现
示例一：
import requests
from bs4 import BeautifulSoup
url = 'https://www.ichunqiu.com/competition/all'
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0',
}
r = requests.get(url,headers=headers)
soup = BeautifulSoup(r.content,'lxml')#解析html
print(soup.title)#输出title标签
print(soup.title.string)#输出title标签的值
示例二：
import requests
from bs4 import BeautifulSoup
url = 'https://www.ichunqiu.com/competition/all'
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0',
}
r = requests.get(url,headers=headers)
soup = BeautifulSoup(r.content,'lxml')#解析html
#print(soup.title)#输出title标签
#print(soup.title.string)#输出title标签的值
com_nes = soup.find_all(name='a')#name=标签值，attrs=标签内的内容(可以不),示例：com_nes = soup.find_all(name='a',attrs={'class':'ui_colorG'}),
for coms in com_nes:
if coms.string != None:
print(coms.string)
示例三：
import requests
from bs4 import BeautifulSoup
import re
url = 'https://www.ichunqiu.com/competition/all'
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0',
}
r = requests.get(url,headers=headers)
soup = BeautifulSoup(r.content,'lx服务器托管网ml')#解析html
#print(soup.title)#输出title标签
#print(soup.title.string)#输出title标签的值
com_nes = soup.find_all(name='a',attrs={'href':re.compile('ichunqiu')})#获取标签a中超链接含有ichunqiu的所有链接
for coms in com_nes:
print(coms)万金油的使用方法BeautifulSoup
三、Py服务器托管网thon爬虫基础知识介绍
3.1 Python hackhttp
Python hackhttp模块的使用介绍∶
安装： pip install hackhttp
import hackhttp
hh = hackhttp.hackhttp()
url = “http://www.baidu.com”
code, head, html, redirect_url, log = hh.http(url)
发起get、post请求，发起http原始数据包
hackhttp介绍补充链接：
http://www.voidcc.com/project/hack-requests
示例一：
import HackRequests
from bs4 import BeautifulSoup
import re
url = 'https://www.cnvd.org.cn/'
hack = HackRequests.hackRequests()
url = "http://www.hacking8.com"
hh = hack.http(url)
print(hh.status_code)#
print(hh.content())
服务器托管，北京服务器托管，服务器租用 http://www.fwqtg.net
机房租用，北京机房租用，IDC机房托管， http://www.fwqtg.net
相关推荐: 「GPT虚拟直播」实战篇二｜无人直播间如何接入虚拟人
摘要 虚拟人和数字人是人工智能技术在现实生活中的具体应用，它们可以为人们的生活和工作带来便利和创新。在直播间场景里，虚拟人和数字人可用于直播主播、智能客服、营销推广等。接入GPT的虚拟人像是加了超强buff，具备更强大的自然语言处理能力和智能对话能力，可以实现…
