howto-urllib2.pdf
(
133 KB
)
Pobierz
HOWTOFetchInternetResources
Usingurllib2
Release2.7.6
GuidovanRossum
FredL.Drake,Jr.,editor
March 17, 2014
PythonSoftwareFoundation
Email:docs@python.org
Contents
1Introduction
ii
2FetchingURLs
ii
2.1
Data
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iii
2.2
Headers
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iii
3HandlingExceptions
iv
3.1
URLError
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iv
3.2
HTTPError
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
iv
Error Codes
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
v
3.3
Wrapping it Up
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vi
Number 1
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vi
Number 2
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vii
4infoandgeturl
vii
5OpenersandHandlers
vii
6BasicAuthentication
viii
7Proxies
ix
8SocketsandLayers
ix
9Footnotes
ix
Index
xi
Author
Michael Foord
Note: There is an French translation of an earlier revision of this HOWTO, available at
urllib2 - Le Manuel
manquant
.
1Introduction
RelatedArticles
You may also find useful the following article on fetching web resources with Python:
•
Basic Authentication
A tutorial onBasicAuthentication, with examples in Python.
urllib2is a
Python
module for fetching URLs (Uniform Resource Locators). It offers a very simple interface,
in the form of theurlopenfunction. This is capable of fetching URLs using a variety of different protocols. It
also offers a slightly more complex interface for handling common situations - like basic authentication, cookies,
proxies and so on. These are provided by objects called handlers and openers.
urllib2 supports fetching URLs for many “URL schemes” (identified by the string before the ”:” in URL - for
example “ftp” is the URL scheme of “
ftp://python.org/
”
) using their associated network protocols (e.g.
FTP,
HTTP). This tutorial focuses on the most common case, HTTP.
For straightforward situationsurlopenis very easy to use. But as soon as you encounter errors or non-trivial cases
when opening HTTP URLs, you will need some understanding of the HyperText Transfer Protocol. The most
comprehensive and authoritative reference to HTTP is
RFC2616
.
This is a technical document and not intended
to be easy to read. This HOWTO aims to illustrate usingurllib2, with enough detail about HTTP to help you
through. It is not intended to replace the
urllib2
docs, but is supplementary to them.
2FetchingURLs
The simplest way to use urllib2 is as follows:
import
urllib2
response
=
urllib2
.
urlopen(
’http://python.org/’
)
html
=
response
.
read()
Many uses of urllib2 will be that simple (note that instead of an ‘http:’ URL we could have used an URL start-
ing with ‘ftp:’, ‘file:’, etc.).
However, it’s the purpose of this tutorial to explain the more complicated cases,
concentrating on HTTP.
HTTP is based on requests and responses - the client makes requests and servers send responses. urllib2 mirrors
this with a
Request
object which represents the HTTP request you are making. In its simplest form you create
a Request object that specifies the URL you want to fetch. Calling
urlopen
with this Request object returns a
response object for the URL requested. This response is a file-like object, which means you can for example call
.read()
on the response:
import
urllib2
req
=
urllib2
.
Request(
’http://www.voidspace.org.uk’
)
response
=
urllib2
.
urlopen(req)
the_page
=
response
.
read()
Note that urllib2 makes use of the same Request interface to handle all URL schemes. For example, you can make
an FTP request like so:
req
=
urllib2
.
Request(
’ftp://example.com/’
)
In the case of HTTP, there are two extra things that Request objects allow you to do: First, you can pass data to be
sent to the server. Second, you can pass extra information (“metadata”)aboutthe data or the about request itself,
to the server - this information is sent as HTTP “headers”. Let’s look at each of these in turn.
2.1Data
Sometimes you want to send data to a URL (often the URL will refer to a CGI (Common Gateway Interface)
script
1
or other web application). With HTTP, this is often done using what’s known as aPOSTrequest. This is
often what your browser does when you submit a HTML form that you filled in on the web. Not all POSTs have to
come from forms: you can use a POST to transmit arbitrary data to your own application. In the common case of
HTML forms, the data needs to be encoded in a standard way, and then passed to the Request object as the
data
argument. The encoding is done using a function from the
urllib
librarynotfrom
urllib2
.
import
urllib
import
urllib2
url
=
’http://www.someserver.com/cgi-bin/register.cgi’
values
=
{
’name’
:
’MichaelFoord’
,
’location’
:
’Northampton’
,
’language’
:
’Python’
}
data
=
urllib
.
urlencode(values)
req
=
urllib2
.
Request(url,data)
response
=
urllib2
.
urlopen(req)
the_page
=
response
.
read()
Note that other encodings are sometimes required (e.g. for file upload from HTML forms - see
HTML Specifica-
tion, Form Submission
for more details).
If you do not pass the
data
argument, urllib2 uses aGETrequest. One way in which GET and POST requests
differ is that POST requests often have “side-effects”: they change the state of the system in some way (for
example by placing an order with the website for a hundredweight of tinned spam to be delivered to your door).
Though the HTTP standard makes it clear that POSTs are intended toalwayscause side-effects, and GET requests
neverto cause side-effects, nothing prevents a GET request from having side-effects, nor a POST requests from
having no side-effects. Data can also be passed in an HTTP GET request by encoding it in the URL itself.
This is done as follows:
>>>
import
urllib2
>>>
import
urllib
>>>
data
=
{}
>>>
data[
’name’
]
=
’SomebodyHere’
>>>
data[
’location’
]
=
’Northampton’
>>>
data[
’language’
]
=
’Python’
>>>
url_values
=
urllib
.
urlencode(data)
>>>
print
url_values
#Theordermaydiffer.
name=Somebody+Here&language=Python&location=Northampton
>>>
url
=
’http://www.example.com/example.cgi’
>>>
full_url
=
url
+
’?’
+
url_values
>>>
data
=
urllib2
.
urlopen(full_url)
Notice that the full URL is created by adding a
?
to the URL, followed by the encoded values.
2.2Headers
We’ll discuss here one particular HTTP header, to illustrate how to add headers to your HTTP request.
Some websites
2
dislike being browsed by programs, or send different versions to different browsers
3
. By default
urllib2 identifies itself as
Python-urllib/x.y
(where
x
and
y
are the major and minor version numbers of
the Python release, e.g.
Python-urllib/2.5
), which may confuse the site, or just plain not work. The way a
1
For an introduction to the CGI protocol see
Writing Web Applications in Python
.
2
Like Google for example. Theproperway to use google from a program is to use
PyGoogle
of course. See
Voidspace Google
for some
examples of using the Google API.
3
Browser sniffing is a very bad practise for website design - building sites using web standards is much more sensible. Unfortunately a lot
of sites still send different versions to different browsers.
browser identifies itself is through the
User-Agent
header
4
. When you create a Request object you can pass a
dictionary of headers in. The following example makes the same request as above, but identifies itself as a version
of Internet Explorer
5
.
import
urllib
import
urllib2
url
=
’http://www.someserver.com/cgi-bin/register.cgi’
user_agent
=
’Mozilla/4.0(compatible;MSIE5.5;WindowsNT)’
values
=
{
’name’
:
’MichaelFoord’
,
’location’
:
’Northampton’
,
’language’
:
’Python’
}
headers
=
{
’User-Agent’
:user_agent}
data
=
urllib
.
urlencode(values)
req
=
urllib2
.
Request(url,data,headers)
response
=
urllib2
.
urlopen(req)
the_page
=
response
.
read()
The response also has two useful methods. See the section on
info and geturl
which comes after we have a look at
what happens when things go wrong.
3HandlingExceptions
urlopenraises
URLError
when it cannot handle a response (though as usual with Python APIs, built-in excep-
tions such as
ValueError
,
TypeError
etc. may also be raised).
HTTPError
is the subclass of
URLError
raised in the specific case of HTTP URLs.
3.1URLError
Often, URLError is raised because there is no network connection (no route to the specified server), or the specified
server doesn’t exist. In this case, the exception raised will have a ‘reason’ attribute, which is a tuple containing an
error code and a text error message.
e.g.
>>>
req
=
urllib2
.
Request(
’http://www.pretend_server.org’
)
>>>
try
:urllib2
.
urlopen(req)
...
except
URLError
as
e:
...
print
e
.
reason
...
(4,’getaddrinfofailed’)
3.2HTTPError
Every HTTP response from the server contains a numeric “status code”. Sometimes the status code indicates
that the server is unable to fulfil the request. The default handlers will handle some of these responses for you
(for example, if the response is a “redirection” that requests the client fetch the document from a different URL,
urllib2 will handle that for you). For those it can’t handle, urlopen will raise an
HTTPError
. Typical errors
include ‘404’ (page not found), ‘403’ (request forbidden), and ‘401’ (authentication required).
See section 10 of RFC 2616 for a reference on all the HTTP error codes.
The
HTTPError
instance raised will have an integer ‘code’ attribute, which corresponds to the error sent by the
server.
4
The user agent for MSIE 6 is‘Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1;SV1;.NETCLR1.1.4322)’
5
For details of more HTTP request headers, see
Quick Reference to HTTP Headers
.
ErrorCodes
Because the default handlers handle redirects (codes in the 300 range), and codes in the 100-299 range indicate
success, you will usually only see error codes in the 400-599 range.
BaseHTTPServer.BaseHTTPRequestHandler.responses
is a useful dictionary of response codes in
that shows all the response codes used by RFC 2616. The dictionary is reproduced here for convenience
#Tablemappingresponsecodestomessages;entrieshavethe
#form{code:(shortmessage,longmessage)}.
responses
=
{
100
:(
’Continue’
,
’Requestreceived,pleasecontinue’
),
101
:(
’SwitchingProtocols’
,
’Switchingtonewprotocol;obeyUpgradeheader’
),
200
:(
’OK’
,
’Requestfulfilled,documentfollows’
),
201
:(
’Created’
,
’Documentcreated,URLfollows’
),
202
:(
’Accepted’
,
’Requestaccepted,processingcontinuesoff-line’
),
203
:(
’Non-AuthoritativeInformation’
,
’Requestfulfilledfromcache’
),
204
:(
’NoContent’
,
’Requestfulfilled,nothingfollows’
),
205
:(
’ResetContent’
,
’Clearinputformforfurtherinput.’
),
206
:(
’PartialContent’
,
’Partialcontentfollows.’
),
300
:(
’MultipleChoices’
,
’Objecthasseveralresources--seeURIlist’
),
301
:(
’MovedPermanently’
,
’Objectmovedpermanently--seeURIlist’
),
302
:(
’Found’
,
’Objectmovedtemporarily--seeURIlist’
),
303
:(
’SeeOther’
,
’Objectmoved--seeMethodandURLlist’
),
304
:(
’NotModified’
,
’Documenthasnotchangedsincegiventime’
),
305
:(
’UseProxy’
,
’YoumustuseproxyspecifiedinLocationtoaccessthis’
’resource.’
),
307
:(
’TemporaryRedirect’
,
’Objectmovedtemporarily--seeURIlist’
),
400
:(
’BadRequest’
,
’Badrequestsyntaxorunsupportedmethod’
),
401
:(
’Unauthorized’
,
’Nopermission--seeauthorizationschemes’
),
402
:(
’PaymentRequired’
,
’Nopayment--seechargingschemes’
),
403
:(
’Forbidden’
,
’Requestforbidden--authorizationwillnothelp’
),
404
:(
’NotFound’
,
’NothingmatchesthegivenURI’
),
405
:(
’MethodNotAllowed’
,
’Specifiedmethodisinvalidforthisserver.’
),
406
:(
’NotAcceptable’
,
’URInotavailableinpreferredformat.’
),
407
:(
’ProxyAuthenticationRequired’
,
’Youmustauthenticatewith’
’thisproxybeforeproceeding.’
),
408
:(
’RequestTimeout’
,
’Requesttimedout;tryagainlater.’
),
409
:(
’Conflict’
,
’Requestconflict.’
),
410
:(
’Gone’
,
’URInolongerexistsandhasbeenpermanentlyremoved.’
),
411
:(
’LengthRequired’
,
’ClientmustspecifyContent-Length.’
),
412
:(
’PreconditionFailed’
,
’Preconditioninheadersisfalse.’
),
413
:(
’RequestEntityTooLarge’
,
’Entityistoolarge.’
),
414
:(
’Request-URITooLong’
,
’URIistoolong.’
),
Plik z chomika:
chomiczekipliczek
Inne pliki z tego folderu:
c-api.pdf
(802 KB)
distutils.pdf
(412 KB)
extending.pdf
(393 KB)
faq.pdf
(451 KB)
howto-argparse.pdf
(113 KB)
Inne foldery tego chomika:
diveintopython-5.4
Grafika
Zgłoś jeśli
naruszono regulamin