I was trying to write a python script which would login to a remote web page, and then request some data. The login was a standard HTML form, so I had to POST to a url with the appropriate form fields.

Using python httplib, everytime I did the POST for the login, I was getting the login page HTML returned again, with a 200 status. To solve this I switched to the httplib2 library - I couldn't get this to work until I switched to this.

Once I had login working, I had trouble requesting the data url. I believe this was to do with the .NET server I was communicating with using cookies to enforce the login session. As I didn't pass the cookie into my data request, it didn't think I was authenticated.

Here's the code that I came up with:

import httplib2, urllib

lLoginUrl="https://www.server.co.uk/login.asp"
lDataUrl="https://www.server.co.uk/data.asp?from=01/01/2010&to=24/08/2010&csv=1"

lLoginParams={'username':'username','password':'secret'}
lLoginHeaders = {'Content-type': 'application/x-www-form-urlencoded'}

http = httplib2.Http()
lResponse, lContent = http.request(lLoginUrl, 'POST', headers=lLoginHeaders, body=urllib.urlencode(lLoginParams))
print lResponse.status

lDataHeaders = {}
lDataHeaders['Cookie'] = lResponse['set-cookie']
lResponse, lContent = http.request(lDataUrl, 'GET', headers=lDataHeaders)
print lResponse.status

print lContent

This code first of all does a POST to a login url. It then gets the cookies from the login response, and passes them into the GET for the data.

References

• http://code.google.com/p/httplib2/
• http://www.testingreflections.com/node/view/5919