摘要：本系列以為基礎(chǔ)是的標(biāo)準(zhǔn)網(wǎng)絡(luò)請(qǐng)求庫。包含了網(wǎng)絡(luò)數(shù)據(jù)請(qǐng)求，處理改變請(qǐng)求頭和用戶代理，重定向，認(rèn)證等的函數(shù)。超時(shí)單位為秒?yún)?shù)必須是的實(shí)例返回值返回一個(gè)可以作為的對(duì)象。返回的個(gè)部分，分別是機(jī)制網(wǎng)絡(luò)位置路徑路徑段參數(shù)查詢片段。

本系列以python3.4為基礎(chǔ)
urllib是Python3的標(biāo)準(zhǔn)網(wǎng)絡(luò)請(qǐng)求庫。包含了網(wǎng)絡(luò)數(shù)據(jù)請(qǐng)求，處理cookie,改變請(qǐng)求頭和用戶代理，重定向，認(rèn)證等的函數(shù)。
urllib與urllib2?:python2.x用urllib2,而python3改名為urllib,被分成一些子模塊：urllib.request,urllib.parse,urllib.error,urllib.robotparser.盡管函數(shù)名稱大多和原來一樣，但是使用新的urllib庫時(shí)需要注意哪些函數(shù)被移動(dòng)到子模塊里了。

HTTP版本：HTTP/1.1，包含Connection:close 頭

特別常用的函數(shù):urllib.request.urlopen()

同類型開源庫推薦:requests

urllib:用來處理網(wǎng)絡(luò)請(qǐng)求和操作url。有以下子模塊

urllib.request 打開后讀取url內(nèi)容

urllib.error 包含由urllib.request拋出的異常類

urllib.parse 解析URL

urllib.robotparser 解析robots.txt files

from urllib.request import urlopen
html=urlopen("https://www.baidu.com")
print(html.geturl(),html.info(),html.getcode(),sep="
")
print(html.read().decode("UTF-8"))

from urllib import request
with request.urlopen("https://api.douban.com/v2/book/2129650") as f:
    data = f.read()
    print("Status:", f.status, f.reason)
    for k, v in f.getheaders():
        print("%s: %s" % (k, v))
    print("Data:", data.decode("utf-8"))

from urllib import request
req = request.Request("http://www.douban.com/") #設(shè)置請(qǐng)求頭
req.add_header("User-Agent", "Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25")
with request.urlopen(req) as f:
    print("Status:", f.status, f.reason)
    for k, v in f.getheaders():
        print("%s: %s" % (k, v))
    print("Data:", f.read().decode("utf-8"))
import urllib.request
data = parse.urlencode([ #進(jìn)行url編碼參數(shù)
    ("username", "xby")]
req = urllib.request.Request(url="https://www.baidu.com",
                     data=data)
with urllib.request.urlopen(req) as f:
    print(f.read().decode("utf-8"))
from urllib import request, parse
print("Login to weibo.cn...")
email = input("Email: ")
passwd = input("Password: ")
login_data = parse.urlencode([ #進(jìn)行url編碼參數(shù)
    ("username", email),
    ("password", passwd),
    ("entry", "mweibo"),
    ("client_id", ""),
    ("savestate", "1"),
    ("ec", ""),
    ("pagerefer", "https://passport.weibo.cn/signin/welcome?entry=mweibo&r=http%3A%2F%2Fm.weibo.cn%2F")
])
req = request.Request("https://passport.weibo.cn/sso/login") 
req.add_header("Origin", "https://passport.weibo.cn")
req.add_header("User-Agent", "Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25")
req.add_header("Referer", "https://passport.weibo.cn/signin/login?entry=mweibo&res=wel&wm=3349&r=http%3A%2F%2Fm.weibo.cn%2F")
with request.urlopen(req, data=login_data.encode("utf-8")) as f:
    print("Status:", f.status, f.reason)
    for k, v in f.getheaders():
        print("%s: %s" % (k, v))
    print("Data:", f.read().decode("utf-8"))
urllib.request
urllib.request.urlopen(url, data=None, [timeout, ]*, cafile=None, capath=None, cadefault=False, context=None)

url參數(shù)可以是字符串或者urllib.request.Request對(duì)象

data參數(shù)必須是字節(jié)形式。可以通過from urllib import parse parse.urlencode()來處理得到。如果沒有提供dat參數(shù)則為GET請(qǐng)求，否則為POST請(qǐng)求。

[tomeout,]超時(shí)單位為秒

context參數(shù)必須是ssl.SSLContext的實(shí)例

返回值：返回一個(gè)可以作為contextmanager的對(duì)象。它有一些方法和屬性：
geturl()
info()-元數(shù)據(jù)信息，比如headers
getcode()-http響應(yīng)碼，比如200
read()-獲取內(nèi)容，字節(jié)形式
status
reason

對(duì)于Http(s)請(qǐng)求，返回的一個(gè)http.client.HTTPResponse對(duì)象。常用方法getheaders(),read()
對(duì)于ftp,file請(qǐng)求，返回一個(gè)urllib.response.addinfourl對(duì)象

可能拋出的異常urllib.error.URLError,urllib.error.HTTPError

class urllib.request.Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None)
通過這個(gè)對(duì)象我們可以設(shè)置請(qǐng)求數(shù)據(jù)，添加請(qǐng)求頭，同時(shí)可以獲取一些url信息：比如協(xié)議類型，主機(jī)。也可以設(shè)置代理Request.set_proxy(host, type)

class urllib.request.OpenerDirector以及關(guān)聯(lián)的urllib.request.install_opener(opener),urllib.request.build_opener([handler, …])
方法：OpenerDirector.add_handler(handler) ，這個(gè)handler對(duì)象必須繼承urllib.request.BaseHandler，常見的有
urllib.request.BaseHandler -基類
urllib.request.HTTPDefaultErrorHandler
urllib.request.HTTPRedirectHandler
urllib.request.HTTPCookieProcessor
urllib.request.ProxyHandler
urllib.request.HTTPBasicAuthHandler
urllib.request.HTTPSHandler
例子：

import urllib.request
# Create an OpenerDirector with support for Basic HTTP Authentication...
auth_handler = urllib.request.HTTPBasicAuthHandler()
auth_handler.add_password(realm="PDQ Application",
                          uri="https://mahler:8092/site-updates.py",
                          user="klem",
                          passwd="kadidd!ehopper")
opener = urllib.request.build_opener(auth_handler)
# ...and install it globally so it can be used with urlopen.
urllib.request.install_opener(opener)
urllib.request.urlopen("http://www.example.com/login.html")

proxy_handler = urllib.request.ProxyHandler({"http": "http://www.example.com:3128/"})
proxy_auth_handler = urllib.request.ProxyBasicAuthHandler()
proxy_auth_handler.add_password("realm", "host", "username", "password")
 
opener = urllib.request.build_opener(proxy_handler, proxy_auth_handler)
# This time, rather than install the OpenerDirector, we use it directly:
opener.open("http://www.example.com/login.html")

可能拋出的異常urllib.error.URLError,urllib.error.HTTPError
exception urllib.error.URLError :有以下屬性：reason
exception urllib.error.HTTPError 它是URLError的一個(gè)子類，有以下屬性：
code
reason
headers

from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
req = Request("http://www.baidu.com/")
try:
    response = urlopen(req)
except HTTPError as e:
    print("Error code: ", e.code)
except URLError as e:
    print("We failed to reach a server.")
    print("Reason: ", e.reason)
else:
    print("good!")
print(response.read().decode("utf8"))

urllib.parse.urlparse函數(shù)會(huì)將一個(gè)普通的url解析為6個(gè)部分，返回的數(shù)據(jù)類型為ParseResult對(duì)象，通過訪問其屬性可以獲取對(duì)應(yīng)的值。
同時(shí)，它還可以將已經(jīng)分解后的url再組合成一個(gè)url地址（通過urlunparse(parts)）。返回的6個(gè)部分，分別是：scheme(機(jī)制)、netloc(網(wǎng)絡(luò)位置)、path(路徑)、params(路徑段參數(shù))、query(查詢)、fragment(片段)。

urllib.parse.urlencode(query, doseq=False, safe=" ", encoding=None, errors=None),注意：query參數(shù)是一個(gè)序列對(duì)象

urllib.request.urlretrieve(url,savefilepath)