使用python采集脚本之家电子书资源并自动下载到本地的实例脚本-创新互联-成都创新互联网站建设

关于创新互联

多方位宣传企业产品与服务 突出企业形象

公司简介 公司的服务 荣誉资质 新闻动态 联系我们

使用python采集脚本之家电子书资源并自动下载到本地的实例脚本-创新互联

jb51上面的资源还比较全,就准备用python来实现自动采集信息,与下载啦。

专注于为中小企业提供网站建设、网站制作服务,电脑端+手机端+微信端的三站合一,更高效的管理,为中小企业千阳免费做网站提供优质的服务。我们立足成都,凝聚了一批互联网行业人才,有力地推动了千余家企业的稳健成长,帮助中小企业通过网站建设实现规模扩充和转变。

Python具有丰富和强大的库,使用urllib,re等就可以轻松开发出一个网络信息采集器!

下面,是我写的一个实例脚本,用来采集某技术网站的特定栏目的所有电子书资源,并下载到本地保存!

软件运行截图如下:

使用python采集脚本之家电子书资源并自动下载到本地的实例脚本

在脚本运行时期,不但会打印出信息到shell窗口,还会保存日志到txt文件,记录采集到的页面地址,书籍的名称,大小,服务器本地下载地址以及百度网盘的下载地址!

实例采集并下载创新互联成都网站设计公司的python栏目电子书资源:

# -*- coding:utf-8 -*-
import re
import urllib2
import urllib
import sys
import os
reload(sys)
sys.setdefaultencoding('utf-8')
def getHtml(url):
 request = urllib2.Request(url)
 page = urllib2.urlopen(request)
 htmlcontent = page.read()
 #解决中文乱码问题
 htmlcontent = htmlcontent.decode('gbk', 'ignore').encode("utf8",'ignore')
 return htmlcontent
def report(count, blockSize, totalSize):
 percent = int(count*blockSize*100/totalSize)
 sys.stdout.write("r%d%%" % percent + ' complete')
 sys.stdout.flush()
def getBookInfo(url):
 htmlcontent = getHtml(url);
 #print "htmlcontent=",htmlcontent; # you should see the ouput html
 #

crifan

regex_title = '(?P.+?)</h2>'; title = re.search(regex_title, htmlcontent); if(title): title = title.group("title"); print "书籍名字:",title; file_object.write('书籍名字:'+title+'r'); #<li>书籍大小:<span itemprop="fileSize">27.2MB</span></li> filesize = re.search('<spans+?itemprop="fileSize">(?P<filesize>.+?)</span>', htmlcontent); if(filesize): filesize = filesize.group("filesize"); print "文件大小:",filesize; file_object.write('文件大小:'+filesize+'r'); #<div class="picthumb"><a href="//img.jbzj.com/do/uploads/litimg/151210/1A9262GO2.jpg" target="_blank" bookimg = re.search('<divs+?class="picthumb"><a href="(?P<bookimg>.+?)" rel="external nofollow" target="_blank"', htmlcontent); if(bookimg): bookimg = bookimg.group("bookimg"); print "封面图片:",bookimg; file_object.write('封面图片:'+bookimg+'r'); #<li><a href="http://xz6.jb51.net:81/201512/books/JavaWeb100(jb51.net).rar" target="_blank">酷云中国电信下载</a></li> downurl1 = re.search('<li><a href="(?P<downurl1>.+?)" rel="external nofollow" target="_blank">酷云中国电信下载</a></li>', htmlcontent); if(downurl1): downurl1 = downurl1.group("downurl1"); print "下载地址1:",downurl1; file_object.write('下载地址1:'+downurl1+'r'); sys.stdout.write('rFetching ' + title + '...n') title = title.replace(' ', ''); title = title.replace('/', ''); saveFile = '/Users/superl/Desktop/pythonbook/'+title+'.rar'; if os.path.exists(saveFile): print "该文件已经下载了!"; else: urllib.urlretrieve(downurl1, saveFile, reporthook=report); sys.stdout.write("rDownload complete, saved as %s" % (saveFile) + 'nn') sys.stdout.flush() file_object.write('文件下载成功!r'); else: print "下载地址1不存在"; file_error.write(url+'r'); file_error.write(title+"下载地址1不存在!文件没有自动下载!r"); file_error.write('r'); #<li><a href="http://pan.baidu.com/s/1pKfVNwJ" rel="external nofollow" target="_blank">百度网盘下载2</a></li> downurl2 = re.search('</a></li><li><a href="(?P<downurl2>.+?)" rel="external nofollow" target="_blank">百度网盘下载2</a></li>', htmlcontent); if(downurl2): downurl2 = downurl2.group("downurl2"); print "下载地址2:",downurl2; file_object.write('下载地址2:'+downurl2+'r'); else: #file_error.write(url+'r'); print "下载地址2不存在"; file_error.write(title+"下载地址2不存在r"); file_error.write('r'); file_object.write('r'); print "n"; def getBooksUrl(url): htmlcontent = getHtml(url); #<ul class="cur-cat-list"><a href="/books/438381.html" rel="external nofollow" class="tit"</ul></div><!--end #content --> urls = re.findall('<a href="(?P<urls>.+?)" rel="external nofollow" class="tit"', htmlcontent); for url in urls: url = "//www.jb51.net"+url; print url+"n"; file_object.write(url+'r'); getBookInfo(url) #print "url->", url if __name__=="__main__": file_object = open('/Users/superl/Desktop/python.txt','w+'); file_error = open('/Users/superl/Desktop/pythonerror.txt','w+'); pagenum = 3; for pagevalue in range(1,pagenum+1): listurl = "//www.jb51.net/ books/list476_%d.html"%pagevalue; print listurl; file_object.write(listurl+'r'); getBooksUrl(listurl); file_object.close(); file_error.close();</pre><p align=center>另外有需要云服务器可以了解下创新互联scvps.cn,海内外云服务器15元起步,三天无理由+7*72小时售后在线,公司持有idc许可证,提供“云服务器、裸金属服务器、<a href="https://www.cdcxhl.com/gaofang/" target="_blank">高防服务器</a>、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案,具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势,专为企业上云打造定制,能够满足用户丰富、多元化的应用场景需求。</p> <br> 网站名称:使用python采集脚本之家电子书资源并自动下载到本地的实例脚本-创新互联 <br> 本文URL:<a href="http://kswsj.cn/article/dijhco.html">http://kswsj.cn/article/dijhco.html</a> </div> </div> <div class="other"> <h3>其他资讯</h3> <ul> <li> <a href="/article/eghesh.html">浏览pv是什么意思,网站上的PV和UV分别是什么意思</a> </li><li> <a href="/article/egheje.html">微信公众号如何建立,如何创建微信公众号</a> </li><li> <a href="/article/egheei.html">百度商桥织梦</a> </li><li> <a href="/article/eghede.html">怎么注册免费网站发布产品信息</a> </li><li> <a href="/article/egheci.html">html怎么设置超链接字体的颜色</a> </li> </ul> </div> </div> <div class="line"></div> <!--底部--> <footer id="5"> <div class="foot1 container"> <div class="list"> <div class="item"> <a href="javascript:;"> <span class="ico1"><i class="iconfont"></i><img src="/Public/Home/img/ewm.png" alt=""></span> <strong>关注我们</strong> </a> </div> <div class="item"> <a href="" target="_blank"> <span><i class="iconfont"></i></span> <strong>索要报价</strong> </a> </div> <div class="item"> <a href="" target="_blank"> <span><i class="iconfont"></i></span> <strong>我要咨询</strong> </a> </div> <div class="item"> <a href="" target="_blank"> <span><i class="iconfont"></i></span> <strong>找到我们</strong> </a> </div> <div class="item"> <a href="" target="_blank"> <span><i class="iconfont"></i></span> <strong>投诉建议</strong> </a> </div> </div> <div class="tel"> <dl> <tel><a href="tel:400-028-6601" target="_blank">400-028-6601</a></tel><br> <span>也许您需要专业的服务,欢迎来电咨询</span> </dl> <dl> <tel><a href="tel:18980820575" target="_blank">18980820575</a></tel><br> <span>您的需求,是我们前进的动力</span> </dl> </div> </div> <div class="friend"> <div class="container"> <span class="tit">友情链接:</span> <div class="inner"> <a href="https://www.cdcxhl.com/tuoguan/" target="_blank">成都IDC机房托管</a><a href="https://www.cdcxhl.com/jigui/" target="_blank">服务器机柜租用</a><a href="https://www.cdcxhl.com/tuoguan/yaan/" target="_blank">雅安云锦天府</a><a href="https://www.cdcxhl.com/shoulu/" target="_blank">免费收录网站</a><a href="https://www.cdcxhl.com/link/" target="_blank">卖友情链接</a><a href="https://www.cdcxhl.com/hangyead/" target="_blank">一元广告</a><a href="https://www.cdcxhl.com/shoulu/" target="_blank">网站快速收录</a><a href="https://www.cdcxhl.com/ruanwen/" target="_blank">软文发布</a><a href="https://www.cdcxhl.com/yunying.html" target="_blank">网站托管</a><a href="https://www.cdcxhl.com/link/" target="_blank">卖链接</a><a href="https://www.cdcxhl.com/gaiban/chengdu.html" target="_blank">成都网站改版</a><a href="https://www.cdcxhl.com/app.html" target="_blank">app软件开发</a><a href="https://www.cdcxhl.com/xiaochengx.html" target="_blank">微信小程序开发公司</a><a href="https://www.cdcxhl.com/sosuo.html" target="_blank">网站排名优化</a><a href="https://www.cdcxhl.com/ruanwen/" target="_blank">软文营销</a><a href="https://www.cdcxhl.com/mobile.html" target="_blank">手机网站制作</a><a href="https://www.cdcxhl.com/security/" target="_blank">成都等保测评</a><a href="https://www.cdcxhl.com/" target="_blank">成都网站设计公司</a> </div> </div> </div> <div class="foot"> <div class="container"> <div class="footNav"> <h3>网站建设</h3> <a href="https://www.cdcxhl.com/" target="_blank">成都网站建设</a><a href="https://www.cdcxhl.com/xiangyingshi.html" target="_blank">响应式网站建设</a><a href="https://www.cdxwcx.com/wangzhan/mobile.html" target="_blank">成都手机网站建设</a> </div> <div class="footNav"> <h3>服务器托管</h3> <a href="https://www.cdcxhl.com/tuoguan/yidong/" target="_blank">移动服务器托管</a><a href="https://www.cdcxhl.com/idc/cqstsanx.html" target="_blank">重庆水土三线托管</a><a href="https://www.xwcx.net/" target="_blank">成都机柜租用</a> </div> <div class="footNav"> <h3>网站制作</h3> <a href="http://www.cxhlcq.com/zhizuo/" target="_blank">重庆网站制作</a><a href="http://www.cxjianzhan.com/" target="_blank">成都网站制作</a><a href="http://www.cdxwcx.cn/bj/" target="_blank">网站制作报价</a> </div> <div class="footNav"> <h3>企业服务</h3> <a href="https://www.cdcxhl.com/service/gongsibiangeng.html" target="_blank">工商变更</a><a href="https://www.cdcxhl.com/mianfei/jianshe/chengdu.html" target="_blank">免费网站建设</a><a href="https://www.cdcxhl.com/mianfei/jianzhan/" target="_blank">免费建站模板</a> </div> <div class="fr ecode"> <div class="fl"> <img src="/Public/Home/img/ewm.jpg"> <p>关注企业微信</p> </div> <div class="fr slogan"> <p class="icon"> <a class="ph" href=""><i class="iconfont"></i></a> <a class="qq" href="tencent://message/?uin=1683211881&Site=&Menu=yes"><i class="iconfont"></i></a> </p> <p> <i>想要找 </i> <a href="">小程序开发</a>、<a href="">APP开发</a>、 <a href="">营销型网站建设</a>、<a href="">网站建设</a>、 <i><a href="">网站定制开发</a></i> ,就选<a href="">创新互联</a> </p> </div> </div> </div> <div class="bottom container"> <p class="fl"> 版权所有:成都创新互联科技有限公司 备案号:<a href="https://beian.miit.gov.cn/" target="_blank" rel="nofollow">蜀ICP备19037934号</a> 服务热线:028-86922220 </p> <p class="fr"> <a href="https://www.cdxwcx.com/" target="_blank">成都网站建设</a>: <a href="https://www.cdcxhl.com/" target="_blank">创新互联</a> </p> </div> </div> </footer> <!--在线咨询--> <div class="fot"> <ul> <li> <a href="https://p.qiao.baidu.com/cps/mobileChat?siteId=11284691&userId=6256368&type=1&reqParam=%20{%22from%22:0,%22sessionid%22:%22%22,%22siteId%22:%2211284691%22,%22tid%22:%22-1%22,%22userId%22:%226256368%22,%22ttype%22:1,%22siteConfig%22:%20{%22eid%22:%226256368%22,%22queuing%22:%22%22,%22siteToken%22:%226ce441ff9e2d6bedbdfc2a4138de449e%22,%22userId%22:%226256368%22,%22isGray%22:%22false%22,%22wsUrl%22:%22wss://p.qiao.baidu.com/cps3/websocket%22,%22likeVersion%22:%22generic%22,%22siteId%22:%2211284691%22,%22online%22:%22true%22,%22webRoot%22:%22//p.qiao.baidu.com/cps3/%22,%22bid%22:%22160142915792139572%22,%22isSmallFlow%22:0,%22isPreonline%22:0,%22invited%22:0%20},%22config%22:%20{%22themeColor%22:%224d74fa%22%20}%20}&appId=&referer=&iswechat=0&expectWaiter=-1&openid=null&otherParam=null&telephone=null&speedLogId=null&eid=null&siteToken=6ce441ff9e2d6bedbdfc2a4138de449e" target="_blank"> <img src="/Public/Home/img/fot1.png" alt=""> <p>在线咨询</p> </a> </li> <li> <a href="tel:18980820575" target="_blank"> <img src="/Public/Home/img/fot2.png" alt=""> <p>拨打电话</p> </a> </li> </ul> </div> </body> </html> <script> $(".con img").each(function(){ var src = $(this).attr("src"); //获取图片地址 var str=new RegExp("http"); var result=str.test(src); if(result==false){ var url = "https://www.cdcxhl.com"+src; //绝对路径 $(this).attr("src",url); } }); window.onload=function(){ document.oncontextmenu=function(){ return false; } } </script>