Login to a website for scraping

I am trying to login to my brokerage account using Python so I can monitor market movement in real time. I followed a tutorial referenced in the below code.

# -*- coding: utf-8 -*-

import requests
from lxml import html
from urllib.request import Request, urlopen
from getpass import getpass

login_url = "https://auth.tdameritrade.com/auth?response_type=code&client_id=TDARETAILWEB%40AMER.OAUTHAP&redirec"

#--http://kazuar.github.io/scraping-tutorial/
userName = getpass( 'Username: ' )
password = getpass( 'Password: ' )

session_requests = requests.session()

result = session_requests.get(login_url)
tree = html.fromstring(result.text)
authenticity_token = list(set(tree.xpath("//input[@name='_csrf']/@value")))[0]

print(result)

payload = {
	"su_username": "userName", 
	"su_password": "password", 
	"_csrf": "authenticity_token"
}

result = session_requests.post(
	login_url, 
	data = payload, 
	headers = dict(referer=login_url)
)

print(result)

The first print(result) gives <Response [200]> while the second gives <Response [404]>. I noticed “OAUTH” in login url (I did not provide the full login URL). Do I need to use a Python Library that can handle OAUTH Authentication? Please recommend a resource I can study to accomplish this login task.

You are using string values instead of the actual variables in your payload:

Just remove the string quotes around userName, password, and authenticity_token, and you should be fine.

1 Like