Daniel

In this part (which is, probably thankfully, much shorter than the last one,) I'm going to use the code we developed last time to actually access the real-time quote service available to registered E*Trade customers. Onward!

In order to access the real-time quote feature of the E*Trade Canada site, we need to follow the same steps we did last time, when trying to emulate the login process:

• Determine what data is being sent with the form request
• Mimc that request in our code
• Parse the response appropriately

After logging into the E*Trade Canada site, we fire up Live HTTP Headers, enter in a desired quote, submit it, and observe the results:

What does all this tell us? Just as before, we see what URL the form is acessing, how it is doing so, and what the content being posted is. We are now almost ready to use this information to properly emulate the real-time quote feature.

One wrinkle in this case is that while the form has a dropdown listing the available exchanges by using their full names, the data actually being passed identifies the selected exchange by a short-hand code (in this case "O" indicates teh NASDAQ exchange.) To properly account for this, we must inspect the form's source, and integrate the available options into our own code:

With that done, we simply follow the process set forth in the first part of this tutorial for posting to a website (with the proper cookies attached):

// Prepare the request
string content = QUOTE_CONTENT.Replace(SYMBOL_PLACEHOLDER,symbol);
content = content.Replace(EXHANGE_PLACEHOLDER,exchange);

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(QUOTE_HOST + QUOTE_PATH);
req.ContentType = FORM_CONTENT_TYPE;
req.Method = POST_METHOD;
req.ContentLength = content.Length;
req.CookieContainer = cookies;

// Send the request
StreamWriter sw = new StreamWriter(req.GetRequestStream());
sw.Write(content);
sw.Flush();
sw.Close();

The final step is reading the response, and parsing it for the desired data:

string responseContent = "";
using (HttpWebResponse res = req.GetResponse())
{
    using (StreamReader sr = new StreamReader(res.GetResponseStream()))
    {
        responseContent = sr.ReadToEnd();
    }
}

// Parse the response for the quote data
QuoteData qd = ParseQuoteData(responseContent.ToUpper(),symbol.ToUpper());

That's it! Any remaining difficulties in terms of using the real-time quote feature are confined to extracting the quote from the returned HTML, a process that is, though somewhat tedious, not altogether that difficult.

Complete code for the EtradeConnectionCanada class (along with the skeleton of a very simple WCF service that wraps it): EtradeService.zip

That's all we need to know for this part. We can now move over to our code, and start filling in the details of our initial HttpWebRequest object. First, we need to specify the correct URL, then specify that it should be a POST. We need to add the correct Content-type header, and finally, we need to set the correct data to send. The easiest way to handle the data string is to make a parameterized version into a constant, and then replace the userid and password parameters as necessary. This is what our code looks like so far:

public bool login(String userid, String password)
{
    // Setup the content string
    string content = LOGIN_CONTENT.Replace(USERID_PLACEHOLDER,userid);
    content = content.Replace(PASSWORD_PLACEHOLDER,password);

    // Setup the request
    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(LOGIN_HOST + LOGIN_PATH);
    req.ContentType = FORM_CONTENT_TYPE;
    req.ContentLength = content.Length;
    req.Method = POST_METHOD;
    req.AllowAutoRedirect = false;   // We need to handle the redirect ourselves

We now have to examine the server's response to the initial login request, to see what happens:

Looking at the headers, we learn a few things. The first is that we're being redirected (that's what 302 means.) Note the presence of the "Location" header, which tells us where our next destination is. Further, we see that there are two "Set-Cookie" headers. These hold the key to mimicing a browser's behaviour.

If we had not turned off auto-redirection for the request, then the .Net class would have happily followed the redirect to the next location without setting the cookies appropriately. Thus the server assumes our request is ill-formatted, and will boot us back to the beginning. Thus by handling the redirection ourselves, we can parse the headers correctly, and thus have the proper information in the subsequent request:

using (HttpWebResponse res = (HttpWebResponse)req.GetResponse())
{
    // Check if it's 302'ing us
    if (res.StatusCode == HttpStatusCode.Found)
    {
        string cookies = res.Headers.GetValues(SET_COOKIE_HEADER);
        foreach (string cookie in cookies)
            this.cookieCollection.Add(ParseCookie(cookie));
    }

The cookieCollection member is .Net CookieCollection object. HttpWebRequest objects have a property called CookieContainer of the same type, so once we populate our copy appropriately, we can use it with all of our subsequent requests to the site. In fact, we are now ready to follow the redirect:

string location = res.Headers.Get(LOCATION_HEADER);
req = (HttpWebRequest)WebRequest.Create(LOGIN_HOST + location);

// Set the cookie container
req.CookieContainer = this.CookieCollection;

// Getting the response submits the request
using (HttpWebResponse redirRes = (HttpWebResponse)req.GetResponse())
{
    // Get the cookies as above; note that if a cookie
    // is Added that already exists, then its value
    // will just be overwritten with the updated data
   
    // ... At this point, we'd check for login success, etc.

There are a number of interesting points here, but the most important is the use of the CookieContainer property; setting it ensures that the correct authentication information is sent to the server each time. Let us look at what that actually is:

Once again, we can see exactly what is going on. The top headers are for the GET request for the page to which we were redirected. Though not visible (since they're off the screen,) the cookies set in the previous response are appended at the end of the "Cookie" header. In the response headers, we get the final cookie we need, the SMSESSION cookie.

Having verified our login information, SiteMinder generates a unique session id, and marks it in the DB as logged in. Thus every time we send that cookie, SiteMinder can verify its logged in status. Note also that the previous two SiteMinder cookies are set back to empty, to indicate the login process has come to an end.

This is the end of Part I of this tutorial. At this point, we have successfully filled in a login form, and followed the steps required to ensure that we can use a successful login to utilize otherwise inaccessible features of the site. In the next part, I'll go through an example of doing just that. Stay tuned!

Code: EtradeCon.zip