INDUSTRY TRENDS

The Internet is evolving (understatement of the week). Not just changing, as that is constant, but more extreme along the lines of Neanderthal man walking upright.INDUSTRY TRENDS

The Internet is evolving (understatement of the week). Not just changing, as that is constant, but more extreme along the lines of Neanderthal man walking upright. This new species is called “Web Services.”

Defined, Web Services are programmatic APIs accessible through Internet standards. They are also billed as the third generation of the Internet. In this industry: the first generation involved dial-up bulletin board systems and newsgroups and the second generation we currently live in with HTML and file-based content accessible through a browser like Internet Explorer. In the new world, HTML becomes secondary, and exchanging XML data is now king (more on this later). To help us understand this concept, let me walk through a possible Web Service scenario contrasted to the current standards.

There are a lot of services being provided on the Web right now through hyperlinks, and the use of these services can be easily seen on a majority of adult sites. Payment processing links are to a small number of sites. The same live video feed links are on every site. The same security service pages are seen again and again.

Now lets see what is good and bad about this. Good: your primary business is not any of those services, so you should leave it up to the guys that do make those services their primary business. Bad: you worked hard to get this person to your site in the first place and now you are effectively sending them to another site. Bad: you worked hard to make your site look great, and now you are linking to a frame or popup window that looks nothing like your site at all. Bad: if you want to change services, you have to learn how this new provider works and will have to re-write code to integrate with them. Web Services can solve all these problems… are you interested now?

To reiterate, the drawing above lays out the above scenario:
1) A person is browsing your Website.
2) They hit a hyperlink that updates a frame or pops up a window to another site.
3) The person is now browsing the other Website

Now let us look at this same scenario using Web Services. The problem above occurs because each of the Web sites is giving too much info. YourWebsite.com ends up handing over its customer to TheirWebsite.com to receive whatever services. While TheirWebsite.com ends up returning their graphics and HTML. All that really needs to be exchanged is data. YourWebsite.com collects whatever info is needed from the user using its own pages. That data is then sent to TheirWebsite.com, which processes that data, and returns the result (e.g. credit card approved, a video link, user login accepted). YourWebsite.com then shows the result using its own pages. So the customer has never left your site or even knows TheirWebsite.com exists, and all the pages viewed have the same look-and-feel as the rest of your site. Also, if the service providers would agree to accept and return the data using the same format (called Schemas), then you could change providers by simply changing the URL which you send/receive the data to/from.

The picture shows the Web Service model:
1) A person is browsing your Website
2) They hit a hyperlink back to your site, which requests data from a Web Service provided by another site
3) Your site processes the results and displays the page to your customer

Technically, the same sort of model is possible through what is called ‘screen scraping’. Screen scraping is the practice of a server making a request for a Web page from another server, but instead of displaying that data in a Web browser, the HTML is parsed as text pulling out the appropriate info, and then that info is displayed using a different HTML format. The problem with screen scraping is that it breaks easily. If the site that you are scraping from changes their HTML format, then your program is likely to break and you will have to rewrite that code. Also, if you want to screen scrape a different site, then your code will have to change based on the way the new site designed their pages.

Screen scraping HTML is unnecessary for Web Services, and XML is passed back and forth as a contract between servers instead. The key point being a Web Service just returns data, and has no presentation information, so that it can be displayed differently for each site and for each device (e.g. wireless Web or Pocket PC devices).

It is beyond the scope of this article to actually teach the technologies involved for creating and consuming Web Services. There are many articles and books available that have already been written to adequately cover that topic, with the current leaders being Microsoft and IBM. Some simple articles I wrote over a year ago are Creating A Web Service and Accessing A Web Service.

Instead, I will move on to describe some of the existing Web Services that are already out there. But first, another key point to note regarding the technologies involved for Web Services is that they are based on standards. The W3C standards committee, working with all of the large software companies, are making sure that everybody’s Web Services written in any language, running on any server, will work with each other! For example, if my site is written in PHP, then I can still use a Web Service that was written using ASP, Java, ASP.NET, etc… and vice versa.

There are literally hundreds if not thousands of Web Services that are already live. Most of these are trial projects that people are experimenting with to learn this new Internet model. As far as production quality Web Services, the two most noteworthy are Microsoft MapPoint .NET and Google’s Search API. So the question is “what do they do?” and “how can I corrupt them?”

The Microsoft MapPoint .NET Web Service is for general mapping needs. It can be used to verify addresses, find driving locations, generate maps, etc… Much like MapQuest, but without the HTML to go along with it. For the adult world, I used this service to create a Web application that was a strip club locater. Basically, the application asks a user to enter a zip code. That zip code is then sent to the MapPoint Web Service, along with the request for strip clubs that are nearby. The Web Service then returns the image as a base64 encoded string that I convert to an image and display in my own application. A Web Service was required here because the application was geared towards Pocket PC devices. I could have just linked to MapQuest, but that sends them away from my site, and the rest of the HTML and graphics that they send would look horrible on the small screen of a PocketPC device, effectively making it unusable.

The screen capture above shows the results of a call to the MapPoint .NET Web Service displayed in my Pocket PC Web application. The zip code entered is marked with a blue pin in the upper-right corner, and two nearby clubs are marked with red pins at the bottom of the screen (one being partially covered by copyright info.).

The Google Search API is for querying the Google search engine to find links based on keywords and other search criteria. It can be used to retrieve cached pages, find certain media types (images/videos), and even has a flag to block adult content! I created my own search engine site geared towards adult security and videos. It uses its own spider to crawl the Web looking for appropriate links (using the screen scraping technique described above). One thing it tracks down is adult password sites. Using its own logic it was able to find links to about 400 password pages over months of use; but by integrating the Google API to go against the much larger database of links, it was able to find about 100 more password pages within a couple of hours. Worth noting that the Google Web Service was written in Java hosted on Unix/Linux servers, and is being used by my C# program running on Windows.

The obvious adult Web Services that should emerge are already provided in HTML form. Credit card processing: instead of linking to their page, you collect the standard info in your own simple form page and submit that info to their Web Service to process that card. Security: instead of a 3rd party blocked IP/account page being displayed, you should just retrieve a true/false answer from the security Web Service and then can display your own blocked or members page accordingly. Premium video links: instead of popping up their window, you request a video link or list of available videos, and then tie that link to the embedded video player in your own page. With the creativity shown in this industry, I am sure that there will be many more compelling Web Services created in the upcoming years.

Enough theory… let’s get our hands dirty with two Web Service that I have already created so that we can get an even better idea. My adult search engine site focuses on two tasks: 1) Finding adult passwords that are listed on the Web (no proxy servers are scanned), and 2) Finding videos that are posted from movie sites with absolute links (relative links are ignored). The search engine has its own Website interface, but it also exposes this functionality as Web Services.

The Password Web Service takes a domain name as input (e.g. YourWebsite.com). It then searches its archives and returns XML to tell if the search engine has found a username/password for that site; and if so, it returns the username that was found along with a hash of the password. Note: the password is hashed because there is no way to get the actual password from a hash. If it was encrypted (and not hashed) then the original password could be retrieved… and I am pretty sure that you do not want me exposing that functionality, because then there would be much more then 500 password sites out there. With that info, your Website can determine if it has that account from the username, hash the password that you have stored, and if your hash matches the hash returned from the Web Service then disable that account because it has been compromised.

The Movie Web Service is simpler to call, but has an extra feature which is experimental in the Web Service world. To call it, it takes no input parameters; and it just returns the list of latest movies for the last day that the spider was run. These results could then be displayed however you see fit. But to make it even easier to use, it has an associated Web page, which acts as a ‘Web Service faceplate’. A Web Service faceplate is a thin layer in front of a Web Service to make it even more accessible to a larger number of programmers. The concept being that if you access the page instead, it calls the Web Service and then returns a minimal amount of HTML to be displayed in a frame or popup, just like the current de facto standard. But to still get the benefits of common look-and-feel, you can give the page a query string parameter to an XSLT file to transform the data from the Web Service into the HTML that you want it to look like. So accessing this Web Service becomes as simple as creating a custom XSLT file and adding that as a query string to the faceplate page. Then on my server, the faceplate page picks up the XSLT file from your server, transforms the XML from the Web Service, and renders the appropriate HTML. My server does end up returning HTML, but it ends up looking just like your site, so the user did view a page from my server, but never knew it. Also, in this manner, my web service can render differently looking pages for thousands of different sites and the code for the faceplate page or Web Service never has to change. I’m not keeping good logs yet, but the movie Web Service is currently being used by a handful of sites.

To recap, this article has given an introduction to Web Services, examples of how current Web Services can be used for the adult industry, and pointers on how to play around with the Web Services that I have created. I hope that I have adequately demonstrated how the adoption of Web Services will allow us to create more powerful sites using other site’s Web Services, in a more professional manner keeping our own look-and-feel, and solving some of the technical problems of this industry to give our users a better experience.

Casey Chesnut (casey@brains-N-brawn.com) is an independent consultant specializing in bleeding edge technologies. These currently include Web Services, wireless, mobile, and speech application development. In his spare time he works on revolutionizing porn and runs his own Web server to support his coding habit. His list of advanced development articles, demo applications (adult and mainstream), and live video feed can be found at brains-N-brawn.com LLC.