简介 / x) V. n! J3 R* a* K" M! k k AutoScraper 是使用 Python 实现的 Web 爬虫，兼容 Python 3，能快速且智能获取指定网站上的数据，这些数据可以是网页文本、URL 地址或者是其它 HTML 元素。另外，它还可以学习抓取规则并返回类似的元素。% T2 Q- S. M0 Q6 F( r+ ~- q 下载安装 # F& \" r7 N; g. W5 Z 项目的源码地址是：https://github.com/alirezamika/autoscraper 2 O7 [- K% O9 {0 \0 n$ E$ v . n: u$ G. {! {9 j* ]' y （1）从git获取安装 $ pip install git+https://github.com/alirezamika/autoscraper.git , p/ T8 k4 x; b ) n0 U) }+ k2 t7 Y3 b0 ?2 u5 ] . p" X4 |, w1 t$ R0 _# |2 d 简单使用 4 f9 I; b0 Y' Z* `% h* T9 |% c 9 l' ]; y! `; \/ f* x: I) ` from autoscraper import AutoScraper
url = 'https://stackoverflow.com/questions/2081586/web-scraping-with-python'
wanted_list = ["How to call an external command?"]
scraper = AutoScraper()
result = scraper.build(url, wanted_list)
print(result)3 \8 O4 k' q: ^/ }0 Y3 s- \ [
'How do I merge two dictionaries in a single expression in Python (taking union of dictionaries)?',
'How to call an external command?',
'What are metaclasses in Python?',
'Does Python have a ternary conditional operator?',
'How do you remove duplicates from a list whilst preserving order?',
'Convert bytes to a string',
'How to get line count of a large file cheaply in Python?',
"Does Python have a string 'contains' substring method?",
'Why is “1000000000000000 in range(1000000000000001)” so fast in Python 3?'
]抓取相似结果 0 d: T& h; [. G% x! ~) E ) }8 K7 b9 R1 Q+ | scraper.get_result_similar('https://stackoverflow.com/questions/606191/convert-bytes-to-a-string')4 S0 D3 ?) g4 d9 T" C8 g. i
; N7 v7 ^, W9 T' ?7 ~& U8 i 抓取确切结果 * E0 ]' m4 G; i( h8 D% A 1 ?- P& X: i$ f: T$ j scraper.get_result_exact('https://stackoverflow.com/questions/606191/convert-bytes-to-a-string')# ]1 [ z! s8 e# D) a, }" ~
' x8 b( Y1 w9 C, l" R$ i! Y9 b 自定义请求模块参数 , [. {6 P% t# N9 b* P8 V 你还可以传递任何自定义请求模块参数。例如，你可能想使用代理或自定义头：, \4 g6 i" }1 M ?# r, z proxies = {
"http": 'http://127.0.0.1:8001',
"https": 'https://127.0.0.1:8001',
}
result = scraper.build(url, wanted_list, request_args=dict(proxies=proxies))抓取多项信息 , ?2 c5 T; R) W% A% S Z6 ?4 t 4 R/ s D' t6 A& _9 C2 q from autoscraper import AutoScraper
url = 'https://github.com/alirezamika/autoscraper'
wanted_list = ['A Smart, Automatic, Fast and Lightweight Web Scraper for Python', '2.5k', 'https://github.com/alirezamika/autoscraper/issues']
scraper = AutoScraper()
scraper.build(url, wanted_list)( y$ ^' k& }( S" z
, ~! [$ M- X3 P4 Y) R 保存模型 $ g4 X+ R. a9 e- Q6 b) D$ [) \9 b 4 v0 u. A% O0 _" @! i # 指定保存的文件路径
scraper.save('stackoverflow')
# 调用方法：
scraper.load('stackoverflow')4 k' ~1 W' C& W. G; r) d # E3 y8 i2 X% `5 |1 E* } 
|手机版|小黑屋|paopaomj.COM
( 
渝ICP备18007172号|
渝公网安备50010502503914号 )
发表于 2021-1-11 13:34:43
