博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
python爬虫——爬取百度文库所有内容
阅读量:7111 次
发布时间:2019-06-28

本文共 2532 字,大约阅读时间需要 8 分钟。

  • 转载自 GitHub 的 Jack-Cherish 大神

代码实现

import requests import re import json import os session = requests.session() def fetch_url(url):  return session.get(url).content.decode('gbk') def get_doc_id(url):  return re.findall('view/(.*).html', url)[0] def parse_type(content):  return re.findall(r"docType.*?\:.*?\'(.*?)\'\,", content)[0] def parse_title(content):  return re.findall(r"title.*?\:.*?\'(.*?)\'\,", content)[0] def parse_doc(content):  result = ''  url_list = re.findall('(https.*?0.json.*?)\\\\\x22}', content)  url_list = [addr.replace("\\\\\\/", "/") for addr in url_list]  for url in url_list[:-5]:  content = fetch_url(url)  y = 0  txtlists = re.findall('"c":"(.*?)".*?"y":(.*?),', content)  for item in txtlists:  if not y == item[1]:  y = item[1]  n = '\n'  else:  n = ''  result += n  result += item[0].encode('utf-8').decode('unicode_escape', 'ignore')  return result def parse_txt(doc_id):  content_url = 'https://wenku.baidu.com/api/doc/getdocinfo?callback=cb&doc_id=' + doc_id  content = fetch_url(content_url)  md5 = re.findall('"md5sum":"(.*?)"', content)[0]  pn = re.findall('"totalPageNum":"(.*?)"', content)[0]  rsign = re.findall('"rsign":"(.*?)"', content)[0]  content_url = 'https://wkretype.bdimg.com/retype/text/' + doc_id + '?rn=' + pn + '&type=txt' + md5 + '&rsign=' + rsign  content = json.loads(fetch_url(content_url))  result = ''  for item in content:  for i in item['parags']:  result += i['c'].replace('\\r', '\r').replace('\\n', '\n')  return result def parse_other(doc_id):  content_url = "https://wenku.baidu.com/browse/getbcsurl?doc_id=" + doc_id + "&pn=1&rn=99999&type=ppt"  content = fetch_url(content_url)  url_list = re.findall('{"zoom":"(.*?)","page"', content)  url_list = [item.replace("\\", '') for item in url_list]  if not os.path.exists(doc_id):  os.mkdir(doc_id)  for index, url in enumerate(url_list):  content = session.get(url).content  path = os.path.join(doc_id, str(index) + '.jpg')  with open(path, 'wb') as f:  f.write(content)  print("图片保存在" + doc_id + "文件夹") def save_file(filename, content):  with open(filename, 'w', encoding='utf8') as f:  f.write(content)  print('已保存为:' + filename) def main():  url = input('请输入要下载的文库URL地址_')  content = fetch_url(url)  doc_id = get_doc_id(url)  type = parse_type(content)  title = parse_title(content)  if type == 'doc':  result = parse_doc(content)  save_file(title + '.txt', result)  elif type == 'txt':  result = parse_txt(doc_id)  save_file(title + '.txt', result)  else:  parse_other(doc_id) if __name__ == "__main__":  main()
你可能感兴趣的文章
Java 深拷贝和浅拷贝
查看>>
生产环境下Tengine keepalived tomcat 安装和配置
查看>>
SVN Can't open file '…/.svn/tmp/text-base/file.svn-base' Clean up Locked
查看>>
华为交换机SNMP故障处理
查看>>
libevent入门教程:Echo Server based on libevent
查看>>
我的友情链接
查看>>
PHP加速、优化、编译 eAccelerator
查看>>
postgresql重置序列SEQUENCE
查看>>
如何在sqlite3中插入中文字符
查看>>
51单片机课程设计:基于ADXL345的三轴偏移值显示(可用于加速度测量)
查看>>
我的友情链接
查看>>
Exchange Server 2010传输规则技巧一
查看>>
我的友情链接
查看>>
android studio更改module名字
查看>>
pam_listfile.so
查看>>
***
查看>>
如何实现对EX4200的远程管理
查看>>
Spring Java-based容器配置
查看>>
复制一个系统和安装压缩和scim输入法的安装。
查看>>
String,StringBuffer与StringBuilder的区别??
查看>>