With VS2008 released this week, I should be diving into LINQ, WCF, and other cool stuff in the .NET CF 3.5. But it's already the end of the week, and it's time for the bigger picture. I want to say a few words about the apps that I'm missing, and all the great apps that I want to write. I use my WM device(s) every day, and there are so many things that I want to do, that I can't. Maybe I can, but it's still very difficult. Let me take a few examples. I was in my car the other day, and I wanted to look up the number to a friend. I started my browser, I searched, I got the link to the "white pages" site, I entered the name, ... I finally got the number, but it took too long, and I was already somewhat frustrated.
I talked to a mobile veteran the other day, and I asked him what he misses the most, and his instant response was "a good mobile browser". I believe that he's right, there can definitely go more development into browser technology. But I think that the real problem is that browsing is no good on a small screen with low bandwidth. I think (know) there's another solution. As a developer, I have the liberty to write my own apps when I need some functionality on my device, and I do it with the best development tools in the world (that was just released this week). When there is a Web Service available with reasonable pricing, I just add a few controls to a form along with a Web Reference and a few lines of glue code, and I got my personal service in place (usually in a couple of minutes). There are great directory sites to find Web Services like XMmethods, RemoteMethods, APIfinder, etc, and companies like ServiceObjects, StrikeIron, CDYNE, WebservcieX, etc, are offering commercial Web Services. But even if the pricing is getting more reasonable (some services are a few cents per transaction), the number of available services are still very limited.
However, if there's no Web Service available, my only option is to get the information directly from the Web using a technique called "scraping". It's really nothing advanced, just the manipulation of the HTTP requests and responses that the browser natively handles from code. For example, let's say I want to put my earlier frustration when looking for my friend's number to an end. I would go out and look for a nice site for looking up numbers, like whitepages, and after analyzing the requests and responses (many times the "view source" in IE is sufficient, but there are also great tools like Fiddler), I write code similar to this...
// Get search result page
string url = string.Format("http://www.whitepages.com/search/FindPerson?who={0}&where={1}", whoTextBox.Text, whereTextBox.Text);
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
StreamReader responseReader = new StreamReader(request.GetResponse().GetResponseStream());
string responseData = responseReader.ReadToEnd();
responseReader.Close();
// More than one?
int i = 0;
if((i = responseData.IndexOf("results_multiple_widget_matching")) > -1)
{
i = responseData.IndexOf("<strong>", i) + 8;
string matches = responseData.Substring(i, responseData.IndexOf("\n", i) - i);
MessageBox.Show(string.Format("{0} matches, refine search!", matches));
}
else
{
// Any
if(responseData.IndexOf("class=\"fn n\">") < 0)
MessageBox.Show("No matches, try again!");
else
{
// Only one
string s = extractValue(responseData, "fn n"); // name
s += "\r\n" + extractValue(responseData, "street-address"); // street
s += "\r\n" + extractValue(responseData, "locality"); // city
s += ", " + extractValue(responseData, "region"); // state
s += " " + extractValue(responseData, "postal-code"); // zip
s += "\r\n" + extractValue(responseData, "tel"); // phone
resultTextBox.Text = s;
}
}
...and on the right you see the code in action (well, Don Box is not exactly my friend, but me and Andy sat down and talked to him after an event once, and that should count for something ;-)). The code for the private method looks like this...
private string extractValue(string s, string name)
{
string valueDelimiter = "class=\"" + name + "\">";
int valuePosition = s.IndexOf(valueDelimiter);
if(valuePosition < 0)
return string.Empty;
int startPosition = valuePosition + valueDelimiter.Length;
int endPosition = s.IndexOf("<", startPosition);
return s.Substring(startPosition, endPosition - startPosition);
}
...and even if helpers like this can be very useful, I clearly recommend the use of regex for more advanced scraping. Note that this approach still needs to download the whole page just to get to the data, and if the above code is running on the WM device the response time will be the same as accessing the site through the browser. That is why this code should be wrapped into a Web Service on a server somewhere, and then the WM device can access the Web Service with the few lines of code that I mentioned above for an already existing Web Service. Making the Web Service responsible for the actual scraping is also better because the clients doesn't need to be updated if there are changes to the source site.
This way, I can now easily look up both addresses and phone numbers to my friends, and it would be easy to add more functionality like calling the found phone number, etc. I should mention that whitepages has added a great mobile version of their site that you enter automatically when you go to their site with a WM device (read more about it, and see a demo), so they are no longer the best example of a service to scrape, but still, their mobile site takes much more bandwidth than the Web Service approach I described above. I don't know about you, but I prefer the clean look in the screenshot above compared to a cluttered web page.
There are so many simple services like this that I still miss, and to name a few apart from the above mentioned phone number lookup, I would like info about flights (delays), cinema search/booking, package tracking, etc, etc, etc. I guess an instant reverse phone number lookup would be a killer for anyone that wants to know who is calling right now, and the number is not in Contacts. What services are you missing?
Subscribe