The notion of a Web service often includes the use of an XML-based protocol, such as SOAP or REST, in order to extract information from a service provider. Many useful bits of data for creating mashups, however, are accessible only by extracting that data directly from a Web site's HTML output.

Such screen-scraping has recently been made easier by a set of new SwingX APIs, SwingX-WS. Richard Bair's recent java.net article, Web Swinging, describes the benefits of that API in comparison to other HTTP client APIs:

The org.jdesktop.http package of the SwingX-WS project provides a set of higher-level constructs for interacting with HTTP-based servers. These classes were written for two reasons: first, because [Apache] HttpClient needs a higher-level framework for simplifying the common case; second, because SwingX-WS shouldn't be tied by API to a third-party library, especially one that is likely to be succeeded by another project in the not-too-distant future.

Bair illustrates the ease with which the new APIs allows the retrieval of an HTML page:

Session s = new Session();
Response r = s.get("http://www.java.net");
System.out.println(r.getBody());

...

Session represents an HTTP session from the client's perspective. For example, if I were implementing a tabbed web browser, I would have one Session per tab. Each Session maintains its own cookie policy. Each may also support several simultaneous connections. Each Session also maintains its own password/authentication state.

Another new class in this API is an implementation of Ajax's well-known XMLHttpRequest:

In the SwingX-WS project, we recently introduced the XmlHttpRequest class. This class is based on the W3C working draft specification www.w3.org/TR/XMLHttpRequest, and is similar to the XMLHttpRequest object found in web browsers today... In addition to enjoying widespread use, XmlHttpRequest is incredibly easy to use:

final XmlHttpRequest req = new XmlHttpRequest();
req.addOnReadyStateChangedListener(new PropertyChangeListener() {
    public void propertyChange(PropertyChangeEvent evt) {
        if (XmlHttpRequest.ReadyState.LOADED == evt.getNewValue()) {
            //update my Swing GUI here
            textArea.setText(req.getResponseText());
        }
    }
});
req.open(Method.GET, "http://www.java.net");
//called from the Swing event handling code,
// this starts the background process
req.send();

The SwingX-WS project also supports working with HTML forms:

It defines the model portion of the HTML form element. For example, there is no id or name property in the Form interface, because they have more to do with HTML than the form concept itself.

The Form interface has methods for retrieving the "action" (the URL to navigate to when the form is submitted) and the HTTP method to use on submit. A Form is composed of a set of Inputs. Each Input represents an HTML input element. An Input has a name and value. The name property is read only, but the value property is read/write and takes a String... HTML forms also support the concept of a group of inputs tied together by name (these are represented as radio buttons in the form), and a combo-box- or list-type selection component. The RadioInput and Select interfaces are used to represent these constructs.

Finally, a set of classes in SwingX-WS make it easier to work with DOM and XPath. Bair illustrates the org.jdesktop.dom API with an example:

Session s = new Session();
Response r = s.get("http://csszengarden.com");
SimpleDocument dom =
    SimpleDocumentBuilder.simpleParse(r.getBody());
for (Node n : dom.getElements("//a")) {
    String href = dom.getString("@href", n);
    String text = n.getTextContent();
    System.out.println(text + "(" + href + ")");
}

While many sites still don't offer Web services, you can extract data from any publicly available site using these new APIs. What do you think of using this technique in real-world—as opposed to toy or demo—applications?