Python实现网络爬虫蜘蛛

《Python实现网络爬虫蜘蛛》由会员分享，可在线阅读，更多相关《Python实现网络爬虫蜘蛛（99页珍藏版）》请在装配图网上搜索。

1、.python 中如何提取网页正文啊 谢谢import urllib.request url= response=urllib.request.urlopen page=response.read python提取网页中的文本1. import os,sys,datetime 2. import httplib,urllib, re 3. from sgmllib import SGMLParser 4. import types 5. class Html2txt: 6. def reset: 7. self.text = 8. self.inbody = True 9. SGMLParse

2、r.reset 10. def handle_data: 11. if self.inbody: 12. self.text += text 13. def start_head: 14. self.inbody = False 15. def end_head: 16. self.inbody = True 17. if _name_ = _main_: 18. parser = Html2txt 19. parser.feedurllib.urlopen.read 20. parser.close 21. print parser.text.strip python 下载网页import

3、httplibconn=httplib.HTTPConnectionconn.requestr1=conn.getresponseprint r1.status,r1.reasondata=r1.readprint dataconn.close用python下载网页,超级简单！from urllib import urlopenwebdata = urlopen.readprint webdata深入python里面有python下载网页内容,用python的pycurl模块实现1. 用python 下载网页内容还是很不错的,之前是使用urllib模块实验的,但听说有pycurl这个模块,而且

4、比urllib好,所以尝试下,废话不说,以下是代码2. #!/usr/bin/env python3. # -*- coding: utf-8 -*-4. import StringIO5. import pycurl6. def writefile: f=open f.write f.close1. html = StringIO.StringIO2. c = pycurl.Curl3. myurl=4. c.setopt5. #写的回调6. c.setopt7. c.setopt8. #最大重定向次数,可以预防重定向陷阱9. c.setopt10. #连接超时设置11. c.setopt1

5、2. c.setopt13. #模拟浏览器14. c.setoptpycurl.USERAGENT, Mozilla/4.0 15. #访问,阻塞到访问结束16. c.perform17. #打印出 20018. print c.getinfo19. #输出网页的内容20. print html.getvalue21. #保存成down.txt文件22. writefilehtml.getvalue,down.txtpython的pycurl模块的安装可以到不同系统使用不同版本,自己看看总结下,Python 下载网页的几种方法1 fd = urllib2.urlopen data = fd.r

6、ead 这是最简洁的一种,当然也是Get的方法2通过GET的方法def GetHtmlSource: try: htmSource = req = urllib2.Request fd = urllib2.urlopen while 1: data = fd.read if not len: break htmSource += data fd.close del fd del req htmSource = htmSource.decode htmSource = formatStr return htmSource except socket.error, err: str_err = %

7、s % err return 3 通过GET的方法def GetHtmlSource_Get: htmSource = try: urlx = httplib.urlsplit conn = httplib.HTTPConnection conn.connect conn.putrequest conn.putheader conn.putheader conn.endheaders res = conn.getresponse htmSource = res.read except Exception, err: trackback.print_exec conn.close return

8、htmSource通过POST的方法def GetHtmlSource_Post: htmSource = try: url = httplib.urlsplit conn = httplib.HTTPConnection conn.connect conn.putrequest conn.putheaderContent-Length, len conn.putheader conn.putheader conn.endheaders conn.send f = conn.getresponse if not f: raise socket.error, timed out htmSourc

9、e = f.read f.close conn.close return htmSource except Exception, err: trackback.print_exec conn.close return htmSource本文来自CSDN博客,转载请标明出处：Django+python+BeautifulSoup组合的垂直搜索爬虫使用python+BeautifulSoup完成爬虫抓取特定数据的工作,并使用Django搭建一个管理平台,用来协调抓取工作。因为自己很喜欢Django admin后台,所以这次用这个后台对抓取到的链接进行管理,使我的爬虫可以应对各种后期的需求。比如分时

10、段抓取,定期的对已经抓取的地址重新抓取。数据库是用python自带的sqlite3,所以很方便。这几天正好在做一个电影推荐系统,需要些电影数据。本文的例子是对豆瓣电影抓取特定的数据。第一步：建立Django模型模仿nutch的爬虫思路,这里简化了。每次抓取任务开始先从数据库里找到未保存的的链接,放到抓取链表里。你也可以根据自己的需求去过滤链接。python代码：view plaincopy to clipboardprint?01.class Crawl_URL: 02. url = models.URLField 03. weight = models.SmallIntegerField#抓

11、取深度起始1 04. is_save = models.BooleanField# 05. date = models.DateTimeField 06. def _unicode_: 07. return self.url class Crawl_URL: url = models.URLField weight = models.SmallIntegerField#抓取深度起始1 is_save = models.BooleanField# date = models.DateTimeField def _unicode_: return self.url 然后生成相应的表。还需要一个ad

12、min管理后台view plaincopy to clipboardprint?01.class Crawl_URLAdmin: 02. list_display = 03. ordering = 04. list_filter = 05. fields = 06.admin.site.register class Crawl_URLAdmin: list_display = ordering = list_filter = fields = admin.site.register 第二步,编写爬虫代码爬虫是单线程,并且每次抓取后都有相应的暂定,豆瓣网会禁止一定强度抓取的爬虫爬虫根据深度来控制

13、,每次都是先生成链接,然后抓取,并解析出更多的链接,最后将抓取过的链接is_save=true,并把新链接存入数据库中。每次一个深度抓取完后都需要花比较长的时候把链接导入数据库。因为需要判断链接是否已存入数据库。这个只对满足正则表达式 的地址进行数据解析。并且直接忽略掉不是电影模块的链接。第一次抓取需要在后台加个链接,比如python代码：#这段代码不能格式化发# coding=UTF-8import urllib2from BeautifulSoup import *from urlparse import urljoinfrom pysqlite2 import dbapi2 as sql

14、itefrom movie.models import *from django.contrib.auth.models import Userfrom time import sleepimage_path = C:/Users/soul/djcodetest/picture/user = User.objects.getdef crawl: for i in range: print 开始抓取 for %d.%i pages = Crawl_URL.objects.filter newurls= for crawl_page in pages: page = crawl_page.url

15、try: c=urllib2.urlopen except: continue try: #解析元数据和url soup=BeautifulSoupc.read #解析电影页面 if re.searchr read_html #解析出有效的链接,放入newurls links=soup for link in links: if href in dict: url=urljoin if url.find!=-1: continue if len 60: continue url=url.split0 # removie location portion if re.search: newurl

16、surl= crawl_page.weight + 1 #连接有效。存入字典中 try: print add url : except: pass except Exception.args: try: print Could not parse : %s % args except: pass #newurls存入数据库 is_save=False weight=i crawl_page.is_save = True crawl_page.save #休眠2.5秒 sleep save_url #保存url,放到数据库里def save_url: for in newurls.items:

17、url = Crawl_URL try: url.save except: try: print url重复: except: pass return True第三步,用BeautifulSoup解析页面抽取出电影标题,图片,剧情介绍,主演,标签,地区。关于BeautifulSoup的使用可以看这里BeautifulSoup技术文档view plaincopy to clipboardprint?01.#抓取数据 02.def read_html: 03. #解析出标题 04. html_title = soup.html.head.title.string 05. title = html_

18、title:len-5 06. #解析出电影介绍 07. try: 08. intro = soup.find.text 09. except: 10. try: 11. node = soup.find.previousSibling 12. intro = node.contents0+node.contents2 13. except: 14. try: 15. contents = soup.find.previousSibling.previousSibling.text 16. intro = contents:len-22 17. except: 18. intro = u暂无

19、19. 20. #取得图片 21. html_image = soupa,href=pile 22. data = urllib2.urlopen.read 23. image = 201003/+html_imagehtml_image.rfind+1: 24. f = file 25. f.write 26. f.close 27. 28. 29. #解析出地区 30. try: 31. soup_obmo = soup.find.findAll 32. html_area = soup_obmo0.nextSibling.split 33. area = html_area0.lstri

20、p 34. except: 35. area = 36. 37. #time = soup_obmo1.nextSibling.split1 38. #time = time.strptime 39. 40. #生成电影对象 41. new_movie = Movie 42. new_movie.save 43. try: 71. #try: 72. 73. #except Exception.args: 74. # print Could not download : %s % args 75. print rdownload success 76. #抓取数据def read_html:

21、#解析出标题 title = html_title:len-5 #解析出电影介绍 try: intro = soup.find.text except: try: node = soup.find.previousSibling intro = node.contents0+node.contents2 except: try: intro = contents:len-22 except: intro = u暂无 #取得图片 html_image = soupa,href=pile data = urllib2.urlopen.read image = 201003/+html_imageh

22、tml_image.rfind+1: f = file f.write f.close #解析出地区 try: soup_obmo = soup.find.findAll html_area = soup_obmo0.nextSibling.split area = html_area0.lstrip except: area = #time = soup_obmo1.nextSibling.split1 #time = time.strptime #生成电影对象 new_movie = Movie new_movie.save try: actors = soup.find.findAll5

23、.nextSibling.nextSibling.string.split0 actors_list = Actor.objects.filter if len = 1: actor = actors_list0 new_movie.actors.add else: actor = Actor actor.save new_movie.actors.add except: pass #tag tags = soup.find.findAll for tag_html in tags: tag_str = tag_html.string if len 4: continue tag_list =

24、 Tag.objects.filter if len = 1: tag = tag_list0 new_movie.tags.add else: tag = Tag tag.save new_movie.tags.add #try: #except Exception.args: # print Could not download : %s % args print rdownload success豆瓣的电影页面并不是很对称,所以有时候抓取的结果可能会有点出入本文来自CSDN博客,转载请标明出处：爬Google的查询页最近没有没有Google API key了,因此只能自己将查询对应的UR

25、L准备好,然后通过脚本将该链接对应的网页爬下来。我们假定,我们要爬这样一个页面：我们可以直接在浏览器输入上面的URL,可以看到,是Google对应IBM这个查询的返回页。我们现在的目的是通过python程序把这个返回页download下来,存在本地,为后面的工作准备数据。一下就是我的代码：import urllib2 , urlparse , gzip from StringIO import StringIOUSER_AGENT = Mozilla/4.0 class SmartRedirectHandler: def http_error_301: result = urllib2.HTT

26、PRedirectHandler.http_error_301 result.status = code return result def http_error_302: result = urllib2.HTTPRedirectHandler.http_error_302 result.status = code return result class DefaultErrorHandler: def http_error_default: result = urllib2.HTTPErrorreq.get_full_url , code , msg , headers , fp resu

27、lt.status = code return result def openAnything: if hasattr: return source if source = - : return sys.stdin if urlparse.urlparse0 = http : request = urllib2.Request request.add_header request.add_header if etag : request.add_header if lastmodified : request.add_header # request.add_header # request.

28、add_header opener = urllib2.build_openerSmartRedirectHandler , DefaultErrorHandler return opener.open try : return open except : pass return StringIOstrdef fetch: result = f = openAnything resultdata = f.read if hasattr: resultetag = f.headers.get resultlastmodified = f.headers.get if f.headers.get=

29、gzip : resultdate=gzip.GzipFilefileobj = StringIO .read if hasattr : resulturl = f.url resultstatus = 200 if hasattr : resultstatus = f.status f.close return result if _name_=_main_ : result = fetch print resultdata print resulturl print resultstatus要注意一下几点：1、要加User-agent,否则GOOGLE会返回403的forbidden信息,

30、因为Python默认的User-agent是python.2、如果不想被重定向到Google 中国,哦,NO,是Google 香港,那么要记得在自己的cookie中找到对应的cookie加上去3、这种东西要多抓包,一个字节一个字节的比对,要相信,浏览器也是程序写的,浏览器能做到的程序也能做到本文来自CSDN博客,转载请标明出处：下载糗事百科的内容_python版1. #coding:utf-82.3.import urllib.request5.import sqlite36.import threading7.import time8.9.class logger:10. def log:1

31、1. for i in msg:12. print13.14.Log = logger15.Log.log16.17.class downloader:18. 19. def _init_:20. self.url = url21. 22. def download:23. Log.log24. try:25. content = urllib.request.urlopen.read26. #req = urllib.request.Request27. #response = urllib.request.urlopen28. #content = response.read29. Log

32、.log30. return 31. except:32. Log.log33. return34. 35. 36.class parser:37. 38. def _init_:39. #获得根节点40. self.html = xml.dom.minidom.parseString41. 42. def parse:43. Log.log44. contents = content:,url:Name47. #获得content节点48. for div in divs:49. if div.hasAttribute and 50. div.getAttribute = content:5

33、1. #获得糗事百科的内容52. textNode = div.childNodes053. qContent = textNode.data54. #数据填充55. contentscontent = qContent56. 57. #获得上一糗事、下一糗事节点58. spans = self.html.getElementsByTagName59. for span in spans:60. pspan = span.parentNode61. if pspan.tagName = a:62. #pspan为对应的链接,此时需要将对应的地址加入数据库63. url = pspan.getA

34、ttribute64. qid = url10:-465. #数据填充66. contentsurl.append67. Log.log68. return69.70.def downloadPage:71. url = content = downloader.download73. if content:74. contents = parser.parse75. if contentscontent:76. db.updateContent77. for i in contentsurl:78. db.addQID79. if len = 2:80. db.updateStatus81.

35、82.#下载池,表示同时允许下载的链接个数83.class downloaderPool:84. def _init_:85. self.downloaders = None*maxLength86. self.downloadList = 87. self.db = None88. 89. def setDownloadList:90. self.downloadList = listset91. 92. def setdb:93. self.db = db94. 95. def daemon:96. #每隔一秒查询线程的状态,为非活动线程则设置为None97. Log.log98. for

36、 index,downloader in enumerate:99. if downloader:100. if not downloader.isAlive:101. Log.log102. self.downloadersindex = None103. 104. #检查线程池状态105. for index,downloader in enumerate:106. if not downloader:107. qid = self.getQID108. if qid:109. #创建线程110. t = threading.Threadtarget=downloadPage,args= 111. self.downloadersindex = t112. t.start113. t.join114. Log.log115. #间隔一秒执行一次116. time.sleep 117. 118. def getQID:119. try:120. tmp = self.downloadList0