Adriaan de Jonge

Available in stores now: Essential App Engine

2011-11-03T14:19:00.002+01:00

My first book is now available in book stores: Essential App Engine. :-)

For more blog posts on this subject, see my other weblog that is devoted to the App Engine.

OrientDB: Connecting, writing and reading data

2011-07-10T21:46:00.002+02:00

In my last post, I made a quick comparison of the performance of OrientDB, MongoDB and CouchDB. Since then, I received some requests for source code. Readers seemed most interested in OrientDB. This might be explained by the fact that there already is a lot of information available on MongoDB. The attention for MongoDB is understandable. It is a great and efficient product.

If you like MongoDB, you might also like OrientDB. It shares many of the characteristics and adds extra features, like a friendly REST interface and a GraphDB. As a first step, giving some extra attention to OrientDB, here is the source code for the last post. It is a bit crude, but gives a good impression of the simplicity of working with OrientDB in its most basic form.

This code reads a comma (semicolon) separated text file (CSV), breaks it up without taking care of escaped characters (semicolons within the text fields) and store all fields in OrientDB. It starts reading the first line of the CSV to serve as field names. It translates the field names from PascalCasing to camelCasing and removes a vendor-specific prefix. This step might need some changes for your own CSV file. I cannot provide my test data for legal reasons.

After that, it reads every line from the file and stores all the values under the field names from the first row. If there is an extra semicolon on that line, there will be errors in the data. In a more realistic example, such errors could be detected by comparing the split line with the number of field names from the first line. The code in this example should be used with a clean CSV file to prevent these errors.

package eu.adriaandejonge.orient;

import java.io.BufferedReader;

import java.io.File;

import java.io.FileReader;

import com.orientechnologies.orient.core.

db.document.ODatabaseDocumentTx;

import com.orientechnologies.orient.core.

record.impl.ODocument;

public class WriteOrient {

public static void main(String[] args) {

try {

long start = System.currentTimeMillis();

File file =

new File("E:/Development/uitjes.csv");

FileReader reader = new FileReader(file);

BufferedReader bufferedReader = new BufferedReader(reader);

String firstLine = bufferedReader.readLine() + "";

String[] columns = firstLine.split(";");

int length = columns.length;

for (int i = 0; i < length; i++) {

columns[i] = columns[i].replaceAll("w3s_", "");

columns[i] =

columns[i].substring(0, 1).toLowerCase() +

columns[i].substring(1);

}

ODatabaseDocumentTx db =

new ODatabaseDocumentTx("local:/tmp/demo").create();

int cnt = 0;

String line;

while ((line = bufferedReader.readLine()) != null) {

cnt++;

if(cnt % 100 == 0) System.out.println("cnt=" + cnt);

ODocument uitje = new ODocument(db, "uitje");

String[] values = line.split(";");

for (int i = 0; i < length; i++) {

if (i < values.length)

uitje.field(columns[i], values[i]);

}

uitje.save();

}

db.close();

System.out.println("DONE in " +

(System.currentTimeMillis() - start) + "ms");

} catch (Exception e) {

e.printStackTrace();

}

To estimate the amount of code needed to communicate with OrientDB, you should focus on the bold lines. The rest of the code only serves to read the CSV files. The bold code is comparable to code for similar NoSQL databases, like MongoDB and CouchDB. Even though there is no standardized API for these databases yet, you do not have to worry about lock-in too much. As long as you isolate the database specific code, you can easily migrate to a different datastore as long as it shares the same characteristics. OrientDB, MongoDB and CouchDB can all be characterized as NoSQL document storages that are particularly well suited for storing JSON documents with nested key-value pairs.

Reading data from OrientDB is somewhat similar. You can do a lot more than demonstrated in the code example. Querying and reading specific fields to name the most basic examples. What the code demonstrates, is that if you simply want to serve JSON documents to the outside world for client side processing, you don't need to write a lot of code.

package eu.adriaandejonge.orient;

import java.io.FileWriter;

import java.io.IOException;

import com.orientechnologies.orient.core.

db.document.ODatabaseDocumentTx;

import com.orientechnologies.orient.core.

record.impl.ODocument;

public class ReadOrient {

public static void main(String[] args) {

try {

tryOrient();

} catch (Exception e) {

e.printStackTrace();

}

private static void tryOrient() throws IOException {

long startTime = System.currentTimeMillis();

ODatabaseDocumentTx db =

new ODatabaseDocumentTx("local:/tmp/demo")

.open("admin", "admin");

readCollection(db);

System.out.println("DONE in " +

(System.currentTimeMillis() - startTime) + "ms");

}

private static void readCollection(ODatabaseDocumentTx db)

throws IOException {

int count = 0;

FileWriter fileWriter = new FileWriter("E:/orient.txt");

for(ODocument doc : db.browseClass("uitje")) {

count++;

fileWriter.write(doc.toJSON() + "\n");

}

System.out.println("# " + count);

}

To test these examples, you need to set up a local instance of OrientDB with a default set up. Also, you need to copy the JAR files called orientdb-core.jar and orient-commons.jar to your /lib folder. When connecting to a remote server, you require two additional JARs, orientdb-client.jar and orientdb-enterprise.jar. More details on the libraries required to connect can be found in the OrientDB documentation.

This is just a small first step towards an actual application. Let me know what you think of it. Suggestions for improvement and follow-up posts are welcome.

MongoDB, CouchDB and OrientDB - a quick comparison

2011-05-19T22:00:00.005+02:00

There are many NoSQL databases, each with their own specific purpose and characteristics. This week, my colleagues and I have been looking for a database that helps store a set of simple JSON documents without a lot of hassle and overhead.

Although Cassandra and Hadoop are fascinating products with lots of potential, they seemed to be designed to solve different problems. The long list quickly narrowed down to MongoDB, CouchDB and OrientDB. This was more due to time constraints than a lack of choice. So that leaves many products in the NoSQL world for follow-up posts!

The quick comparison consisted of three questions:

How easy is it to set up the server and connect to it using a Java API?
How fast can you store 9300 records in JSON format?
How fast can you read 9300 records in JSON format?

Please note that this is not a representative test. It is just a quick scan investigating the first results not hindered by any background knowledge and not performing the optimizations like an expert would. This means that the end result only says something about the quality out of the box, not about the ultimate limitations of the product. It must also be mentioned that all tests were performed on Windows XP, 32 bits. Perhaps this is not the best suitable environment for NoSQL.

These are our findings:

OrientDB

The OrientDB is relatively simple to install. The client API consists of two JARs - less than 850KB in total. The only hurdle in setting up a connection is determining what the connection string should be. It turns out that you need to explicitly create a database before you can connect to it. That is not unreasonable of course.

Writing 9300 records: ± 2,640ms (2nd place in this post)

Reading 9300 records: ± 5,157ms (2nd place in this post)

CouchDB

Installing CouchDB is painful from the start. After downloading, you need to make the server yourself. Or you can download an "unstable" installation package (just quoting the website). Setting up the client required the installation of nine JARs, 1.5MB in total. The CouchDB website suggests using HttpClient 4.0 which results in a ClassNotFoundException. After downgrading to HttpClient 3.1, the client starts working.

Writing 9300 records: ± 45,340ms (3rd place in this post)

Reading 9300 records: ± 196,985ms (3rd place in this post)

MongoDB

MongoDB is one of the most pleasant servers I have ever installed. The only hurdle during startup is that you need to create a /data/db directory on your hard drive. After that, anything is automated for you. The daemon starts up in an eye blink. Databases and collections are automatically created when necessary. The client API consists of a single JAR of 240KB and provides a clean and simple API.

Writing 9300 records: ± 1406ms (1st place in this post)

Reading 9300 records: ± 2140ms (1st place in this post)

Conclusion

In terms of ease of use and response times in the out-of-the-box, unconfigured, unoptimized installation, MongoDB clearly wins. And CouchDB loses in all possible ways.

OrientDB should not be disregarded too quickly though. It offers more interfaces and functionality out of the box than MongoDB. For example, for a user-friendly REST interface, MongoDB relies on 3rd party add ons while OrientDB offers a nice one out of the box. Another fundamental difference is that OrientDB also offers GraphDB functionality in addition to the document storage. Depending on the requirements, that may justify accepting the performance penalty which is significant but not dramatical.

If you're looking for absolute efficiency in terms of both performance and ease of use, go with MongoDB!

And there is still a lot more to find out about these products beyond the simple tests in this post. Keep an eye open for follow-up posts!

Google Search Appliance for Structured Data

2010-12-07T23:57:00.001+01:00

Today was the go-live for ANWB's new Car Portal. The main application we call "Search & Compare". Even if you don't speak dutch, you should be able to try it on http://www.anwb.nl/auto and click on the banner in the upper left of the screen. (or go directly to this link)

The technology for this is based on the Google Search Appliance (GSA). It contains approximately 200,000 cars with many detailed metadata fields to search on. Most people only consider the GSA for unstructured text search. The new Search & Compare application proves that GSA also works well for structured queries. For more documentation on how to do this, see this link on Google's site.

One of the advantages of GSA is easy maintenance and administration. You don't have to think too much about complicated configurations specific for indexing and search solutions (like you need for Lucene). All you have to do, is to create an XML content feed containing lots of meta fields as described in Google's documentation.

Then the last step is to create your own front end. I have to admit: that does require a lot of work. With just an XSLT running inside the GSA, you won't get the results we get in our Search & Compare application. What you can do, is ask the GSA for output in XML by adding &output=xml or &output=xml_no_dtd to the GET request. Then you can use XSLT in your own front-end to create your own screens!

Answering mail #12

2010-07-17T19:33:00.002+02:00

States Mphinyane wrote:

I was recently on the IBM site and you were listed as one of the expects on XML and that seems to be area that i have serious problems with.

The Problem

I am an intern at Botswana Harvard Partnership which is a research institute in matters relating to HIV/AIDS. Currently we are running a study in noe village just outside the capital city. In this study, we use GPS devices so that we can know to which households we have been to. Our job is to pull the way points off of these GPS's and the data comes out the form of an XML file.We do so by using a software called GPS babel which are installed on Ubuntu.We compile all these XML and to come with one XML file to show us the coverage of the study so far because this software has a google maps functionality. Now we do not want to keep appending new XML code each the GPS's come back from the field becaues it can get very tricky and confusing working with thousands of lines of code. We want edit the XML file for each device so that the researchers can return directly to the households they had been to previously.We want to be able to trace our way back to each household. I don't even know if i'm making any sense but that's the best way i can explain it.If you can manage you can give me a call at 76XXXXXX,but of course you would need an extension +267. Thank you in advance.

Hello States,

I am not entirely sure if I understand the question well. This is what I got from the mail:

1. You have multiple people, each with his/her own GPS device

2. The GPS device collects data where you have been

3. The GPS device allows you to revisit places you have been earlier

4. Each person only revisits places he/she has been before, not places visited by a colleague

5. GPS devices can export data in a format readable by GPSBable (not sure which format and whether it is XML or not)

6. GPSBable outputs KML (most probably), readable by Google Maps

7. The GPSBable translations are added to one large KML file by hand

8. The KML file is growing so large that editing by hand is no longer desirable

Am I right so far? (I have the most doubts at point 4.)

Now, if I understand well, you are looking for a way where you can just input individual, smaller XML (probably KML) files to a data repository and have the large all-in-one KML file generated automatically. Correct?

In this case, I would recommend using eXist-db, an open source, native XML database available at http://exist.sourceforge.net/

Using this product, you could add the XML files using a WebDAV connection and query for the large overview file using XQuery.

Then where I am not sure: do you also need to transfer the stored KML files back to the GPS devices? Or is that unnecessary because of point 4.?

Let me know if I understood well and let me know if you need more pointers on using eXist. If you think it could help, please let me know more about the GPS devices and the data format they export.

Kind regards,

Adriaan.

IBM developerWorks: "Query XML documents outside an XML database"

2010-07-13T21:06:00.002+02:00

IBM published my latest article today:

Query XML documents outside an XML database
www.ibm.com
Processing XML in Java usually requires a lot of code and overhead. If you use XQuery, you can do a lot more with a lot less code, even when the XML is stored outside of XML databases. Learn how to use XQuery with Java technology by extracting the hidden information from XML-based Maven POM files.

Go to http://www.ibm.com/developerworks/library/x-xquerymaven/index.html

Other voices: App Engine Cold Start Tips

2010-06-02T21:04:00.002+02:00

The Google App Engine starts and stops web applications quite frequently to add resources or to free resources for other applications. App Engine developers should change the way they write their code to optimize the startup time. I'm working on a more comprehensive guide on how to do this. This will take some time to write and publish though.

In the meantime, there are resources on the internet providing tips on how to reduce cold startup times for the App Engine. For example, take a look at this post on the Answer Cow web log.

Running Saxon HE on Google App Engine

2010-06-02T00:01:00.005+02:00

(Modified. Thanks to Archie Cobbs for the tip)

About a year ago I tested Saxon B version 8 on the Google App Engine. This worked (and still works) perfectly. You can implement XSLT 2.0 and XQuery on Google's hardware.

Later on, Saxon moved to version 9 and the naming changed. I tried the Home Edition on the App Engine and got the following error:

java.lang.SecurityException: SHA1 digest error for net/sf/saxon/s9api/SaxonApiException.class

Usually I think this should mean that the SHA-1 key is invalid. (I'm not exactly an expert on the subject of security). The easiest way around this error it to remove some files from the /META-INF of the saxon9he.jar file. If you remove SAXONICA.SF and SAXONICA.RSA and upload the file again, the library works.

[...]removed[...]

I Googled for the error message and there seems to be another App Engine project with the same error: the Vosao CMS. They say they fixed the error. I checked and yes: they also just removed the SHA-1 files from the JAR...

http://code.google.com/p/vosao/issues/detail?id=142

[...]removed[...]

As noted in a comment by Archie Cobbs, the cause of this problem could be the JAR precompilation feature of the App Engine. This means that signing JARs does not work in general unless you turn off precompilation. This is a security vs. performance trade-off.

Answering mail #11

2010-05-31T21:58:00.004+02:00

Yves Richard wrote:

Adriaan,

Just read your article “Comparing XML database approaches”. Nice article. Just wondering your thoughts on “NXD’s” nowadays. Seems like it’s still a niche market but is it dying?

We have our own grammar called CAAML (http://code.google.com/p/caaml/). We are basically a data warehouse for everything avalanche in Canada with recent interest from European countries and the US. I have implemented a small web application on top of the eXist solution which seems to be working great. I am not getting much support from my more experienced peers though. For me its quite clear that NXD is a very good fit. What am I missing?

Thank you for your input.

Sincerely,

Yves Richard - IT Manager

Hello Yves,

Thank you for your e-mail. It is nice to see your work on CAAML. This looks like an interesting and well-designed XML format. Also nice to hear you are experimenting with eXist.

I'm not sure I can answer your questions reliably. A lot of these things are personal opinions and personal preference. Nevertheless, here is what I think:

I do not think the market for XML databases (native or hybrid) is dying. Some business domains are ahead of others in the adoption of XML technologies. For example, publishing companies use more and more XML to facilitate concepts like "single source publishing". If you write content once and store it in XML, you can publish to paper, the internet, mobile devices or anything else.

The majority of webservices (not only WSDL/SOAP but also REST) use XML as exchange format. Webservices are so widely adopted, it is hard to imagine a world without them.

I think more and more people are slowly realizing that if your input is XML and your output is XML (XHTML), then it is inefficient to translate XML to objects or relational structures in between. On the other hand it seems many software developers are trying to eliminate XML from their code as much as possible because of personal preference.

I'm not sure how to resolve differences in personal preferences between several team members. Showing working examples like you're doing with the small web application should be the best way. I don't think you're missing something.

I hope that showing concise, efficient XQuery code and pointing out the savings in implementation time can help convince your peers to give the technology a chance.

Good luck with your project!

Kind regards,

Adriaan.

Moving jobs using Strikethrough Fonts

2009-09-22T20:44:00.001+02:00

http://www.antw~~oordvoor~~b~~edrijven~~.nl

IFrames are like heroin...

2009-09-20T13:55:00.004+02:00

For some people IFrames seem to be the best invention since sliced bread. IFrames can best be classified as an Integration Pattern. As such, it has the following characteristics:

Quick visual integration of content from other web sites
Loading on the client side (zero extra bandwidth usage on the server)
Asynchronous loading of page elements
Styling at the source
Maintenance on the IFrame source independent of the window in which it is integrated

In other words:

It's cheap
It's fast
It's just those puritan technical guys who are always against this great solution!!! :-(

Is it just because they are puritan, or is it because they like to spend more hours working on a more expensive solution. That would mean higher bills or avoiding unemployment...

Technical employees are only working in their own best interest!

Is that true?

I think the truth is that people who like IFrames are addicts. They are addicted to a quick and dirty solution that solves their short-term needs and leads to illness of the system on the long run.

How does the system get sick of IFrames?

In many ways, both technical and practical.

Let's start with the practical disadvantages of IFrames:

Corporate branding and styling tends to change every few years. For web sites, these styles change even quicker than their paper counterparts.

For a well-written web site using proper XHTML and CSS, style changes only have impact on the CSS files of the site. If written properly, this is a single (set of) file(s) used for the whole site.

Using IFrames however, you have to change the styles of every IFrame separately. Depending on how long you have been addicted, this could mean changing the styles of maybe a 100 or more content/service providers.

Or you could not change them and have an inconsistent look and feel mixing new styles with remainings of the old styling in IFrames.

What happens when the site providing the IFrame content is down?

Usually this means awkward browser errors within the IFrame on the site. No fallback, no control and no friendly message from your side.

Does your IFrame provider have the same standards for uptime and performance as you have? You get what you pay for, in this regard.

What is the conversion rate of users within the IFrame?

On your own site, you are using tools like Google Analytics or paid alternatives to measure the success of your site. But you probably don't have a clue of what happens inside the IFrames on your site.

In the best case, the content provider of the IFrame may give you some numbers, but proper analysis of which users got how far in the funnel is not available.

Is your real life shop accessible for people in wheel chairs?

Then why isn't your website accessible for people using text-to-speech readers of braille devices?

In real life it seems harsh to have a policy not to care about wheel chairs. On the seemingly anonymous internet it is quite common to forbid access to a group of people who could benefit most of a medium like internet.

Do you care about security?

IFrames are vulnerable to cross site scripting attacks. Especially IFrames providing paid services are an interesting target to hackers.

Does that seem far-fetched to you?

Think again and take a look at http://ha.ckers.org/xss.html

Or should I have put that link in an IFrame to prevent you from clicking away from this blog?

A few alternatives to IFrames:

- For any purpose: links ("deeplinks" in buzzword jargon)

- For content: Atom Publishing Protocol (or similar non-standard alternatives)

- For services: WSRP (or similar non-standard alternatives)

- Or with a bit more work: Web services; different front ends for different clients

It all starts with a confession: are you an IFrame addict?

Answering mail #10

2008-09-21T21:26:00.003+02:00

Vanina Beraudo wrote:

Hello
My english isn´´t good, but i try speak with you.
Do you know if i can integrate RCP/RAP with XML UI?
Where can i get you demo source sample?
I was looking your blog.It´´Is very interesting.
I will like see runing the demo but in RAP target
Thanks you

Hello Vanina,

I am afraid that the Rich Ajax Platform / Rich Client Platform still require more Java programming than XML specification. I wouldn't be surprised if that would change though. For example, the Eclipse Modeling Framework and the Graphical Modeling Framework can be used to create models from XML Schemas. A powerful application based on EMF/GMF is the SOA Tools BPMN Modeler. (Actually I think this still uses objects in the background, but it is nice anyway ;-))

If you want to create rich AJAX UIs using XML technology, for now, Orbeon is a nice choice. But I can imagine it would be nice to see something similar based on Eclipse technologies, like OSGi. And something that works on both the client and the server side like RAP and RCP. (By the way, XForms do work both on the client and the server side if you have the right plugins... (edit))

The demo I created uses the Jython scripting language to avoid Java development. The Eclipse version on which it was based is a bit outdated now. But you can find it over here.

Sorry I cannot be of any help. Good luck with your efforts on RAP and RCP!

Kind regards,

Adriaan.

Scala: a misguided approach to XML

2008-09-14T22:54:00.006+02:00

In 2001, when Scala was conceived, it might have seemed like a good idea to integrate XML capabilities in a new programming language. XML was emerging technology, Java libraries for XML processing were still young and it seemed like things could be simplified in comparison to DOM and SAX.

But even back then, some XML technologies were already quite powerful and well capable of handling complex problems. The best examples being XPath 1.0 (16 November 1999) and XSLT 1.0 (also 16 November 1999) Their proper usage just wasn't well known yet. Early versions of Cocoon already showed the first ideas of using their power by combining them into XML pipelines.

The fact is: it is not 2001 anymore. Technologies are even more mature. XPath and XSLT are at version 2.0 and XQuery 1.0 is officially a W3C recommendation (all 23 January 2007). While the 1.0 versions were already powerful, the 2.0 versions and XQuery are a lot more mature and backed up by proper research and documentation in a larger set of Recommendation documents.

At the same time in 2008, new dynamic languages are gaining popularity as Java is in an existential crisis heading towards version 7.0. And Scala is ~~one of these languages~~ a language that is becoming more and more popular. And it integrates XML capabilities.

"What could be wrong with adding XML capabilities?" - I hear you ask. This is an XML related weblog after all. What could be wrong with adding XML support?

If you're asking this question, then obviously you haven't seen Scala yet. Or you did, but you're not familiar with proper XML technologies (yet).

A few years ago, it was common to externalize configuration parameters of Java programs XML descriptors. In more recent years, under the pressure of dynamic languages, it became more common to integrate configurations into the code using Annotations and hard-coded values.

And now, using Scala, you can internalize the XML itself, so it becomes easier to output the hard-coded values like this: (Examples taken from scala.xml)

/* examples/phonebook/embeddedBook.scala */
package phonebook

object embeddedBook {

val company = <a href="http://acme.org">ACME</a>
val first = "Burak"
val last = "Emir"
val location = "work"

val embBook =
<phonebook>
<descr>
This is the <b>phonebook</b> of the
{company} corporation.
</descr>
<entry>
<name>{ first+" "+last }</name>
<phone where={ location }>+41 21 693 68 {val x = 60 + 7; x}</phone>
</entry>
</phonebook>;

def main(args: Array[String]) =
Console.println( embBook )

}

But that is not the worst example in this Scala XML manual. Nooooooo!

My real disappointment in this manual started when I read this sentence:

For new developments, it is more straightforward to use the more convenient Scala API rather than the cumbersome XSLT syntax, or (if it really must be XSLT), some Java library.

Personally I do not think of XSLT as cumbersome at all, but I am always open to new ideas and prove my own ideas wrong. So let's take a look at the example for XML transformations that the Scala XML manual came up with: (Examples taken from scala.xml)

object transform {
import scala.xml._ ;
import scala.xml.dtd._ ;
import org.xml.sax.InputSource ;

/* a former version of Scala used regular expression patterns, like
* in the following code. In the future, we plan to reactivate some
* well-behaved regular expressions again
// gimmick: text replacement "scalac" => <code>scalac</code>
def cook(s: String): Seq[Node] = cook1(s) ;
def cook1(s: Seq[Char]):List[Node] = s match {

case Seq( a @ _*, 's','c','a','l','a','c', b @ _* ) =>

Text(cook2( a )) :: <code>scalac</code> :: cook1( b )
case _ => List( Text( cook2( s )))
}
def cook2(s: Seq[Char]): String = {
val r = new StringBuffer();
s.foreach { c:char => val _ = r.append(c); };
r.toString()
}
*/

def transform1(ns: Iterable[Node]): Seq[Node] = {
val zs = new NodeBuffer();
for(val z <- ns) { zs &+ transform( z ) }
zs
}

/** this functions holds "templates" that transform nodes of an input tree
* into an iterable representation of a sequence of nodes of the output
* tree.
*
* It is ok to return a single node, since each node is at the same
* time a singleton sequence. Likewise, the pattern variable x will be
* of type Seq[Node], although here it is always binding a single node.
*/
def transform(n: Node):Iterable[Node] = n match {
case x @ <article>{ ns @ _ * }</article> =>
<source active="ant" title={ (x \ "title" \ "_").toString() }>
<header>
<author>Burak Emir</author>
<keywords>Scala4Ant</keywords>
<style type="text/css"></style>
</header>
<content>
<title><scala/> Ant Task</title>
{ transform1( x \ "_" ) }
</content>
</source>
case x @ <sect1>{ _* }</sect1> =>
<section>{ transform1( x \ "_" ) }</section>
case x @ <title>{ _* }</title> =>
<h>{ x \ "_" }</h>
case x @ <para>{ _* }</para> =>
<p>{ transform1( x \ "_" ) }</p>
case x @ <itemizedlist>{ _* }</itemizedlist> =>
<ul>{ transform1( x \ "_" ) }</ul>
case x @ <listitem>{ _* }</listitem> =>
<li>{ transform1( x \ "_" ) }</li>
case x @ <constant>{ _* }</constant> =>
// an xml:group is a sequence that appears to the scala type system
// as a single node. Here it is used to append a text node with a space
<xml:group><code>{ x \ "_" }</code> </xml:group>
case x @ <programlisting>{ _* }</programlisting> =>
<pre>{ x \ "_" }</pre>
case Elem(namespace, label, attrs, scp, ns @ _*) =>
Elem(namespace, label, attrs, scp, transform1( ns ):_* )
case z =>
z
};

def main(args:Array[String]) = {
if( args.length == 1 ) { // must have one arg
object ConsoleWriter extends java.io.Writer {
def close() = {}
def flush() = Console.flush
def write(cbuf:Array[char], off:int , len:int ): unit = {
var i = 0
while(i < len)
Console.print(cbuf(off + i))
}
}

val src = XML.load(new InputSource( args( 0 ))); //use Java parser

// transform returns an iterable, but we now it is a singleton
// sequence, so we get its first element
val n = transform( src ).elements.next

val doctpe = DocType("html",PublicID("-//W3C//DTD XHTML 1.1//EN","../default.dtd"), Nil)

/** write document to console, with encoding latin1, xml declaration
* and doctype
*/
XML.write(ConsoleWriter, n, "iso-8859-1", true, doctpe)

}
else error("need one arg");
}
}

Talking about cumbersome! The real problem of this code might not even be in the notation. That is just a matter of getting used to Scala syntax. The problem is in its lack of maintainability. This kind of code assumes that XML documents always follow a fixed pattern. And this idea is confirmed by an earlier section of the manual.

The manual (surprisingly adequately) makes a separation between XML that can be considered text with markup and XML that models data structures in a hierarchical tree format. But (disappointingly quickly) after that, the text starts displaying the first signs of a lack of knowledge of XML technologies:

[...]
There are several points of view that can be taken:

1. XML is regarded as text. We ignore the tree structure completely. Some text/regular expression search is used to retrieve or manipulate information. This can get you quite far for small tasks. Go away, use perl :-)
[...]

What ever happened to XPath?

The Scala designers must have suffered from the Not Invented Here Syndrome and defined their own Scala equivalent of XPath, like this: (Examples taken from scala.xml)

package bib;

object bib {

import scala.xml.{Node,NodeSeq};
import scala.xml.PrettyPrinter;

val biblio =
<bib>
<book>
<author>Peter Buneman</author>
<author>Dan Suciu</author>
<title>Data on ze web</title>
</book>
<book>
<author>John Mitchell</author>
<title>Foundations of Programming Languages</title>
</book>
</bib> ;

val pp = new PrettyPrinter(80, 5);

def main(args:Array[String]):Unit = {
Console.println( pp.formatNodes( biblio \ "book" \ "title" ));

// prints
// <title>Data on ze web</title><title>Foundations ...</title>

Console.println( pp.formatNodes( biblio \\ "title" )); // prints the same

Console.println( pp.formatNodes( biblio \\ "_" )); // prints node and all descendant

Console.println( pp.formatNodes( biblio.descendant_or_self )); // prints the same

}
}

Judging from the list of references under the manual, it is more likely that this XPath-surrogate is modeled after XPath 1.0 than XPath 2.0. And judging from the opinions of the author, it seems unlikely that Scala will adopt XML standards in a proper way any time soon.

Probably the most fundamental critique on this approach is that Scala is mixing concerns that are usually separated from each other. Usually, the challenge of software development is to separate concerns that are otherwise mixed. So why would anyone start working the other way around, working in the wrong direction?

In my opinion, the answer to the current Java 7.0 controversy is not in ~~dynamic~~ languages like Scala. I think it is more likely that the answer is in proper standards like XPath 2.0, XQuery 1.0 and XSLT 2.0. Over the years since 1999, these technologies matured into powerful programming platforms themselves. As I am researching these specs more and more, it turns out they are even a lot more interesting than I thought before I started. And I thought I knew these technologies. So a lot more details on these technologies will follow in later posts!

People: Front-End vs Back-End

2008-09-11T23:54:00.002+02:00

A developer who never misses a deadline might be a good developer. He might reduce the quantity or quality of his work to the bare minimum to do so. Quite possibly, the management team values such choices in favor of the deadline more than an increased quality released after the deadline. On the long run, a pile of bare minimum solutions becomes unstable and completely ignores the wishes of the end user. This means that end users may be spending a significant part of their day being frustrated to work with an inefficient application.

In many cases, the management team makes it even worse by purchasing large software systems intended to improve business processes, resource planning and accounting. From management perspective, these products deliver results. On the other hand though, these kinds of products pay little attention to the end user in most cases.

But there may be good news on the horizon: the latest trend is an increasing attention for usability and rich user interfaces on the internet. Sometimes this is at the cost of standardization, but the real front-end developer knows how to handle such issues properly.

Despite this trend, there remains a major part of the development community who consciously choose to ignore the end user perspective. They expect the end user to adjust to the application instead with intensive training instead of meeting their demands by optimizing usability.

Both kinds of developers can cooperate if they divide the tasks: one works on the user interface and the other implements the back end. The question is whether a poor front end programmer can make a good back end programmer. I have some doubts there. The complexity of the object models to create GUIs is outstanding training material to improve your object orientation skills.

The problem of creating rich clients became worse in the past few years, because the current generation of developers was raised making classic web applications. The majority of young developers has absolutely no experience developing real client side applications providing rich user interfaces by default.

There is good hope that the RIA trend will resolve this issue!

Other voices: Unit testing XSLT and XQuery

2008-09-09T20:19:00.003+02:00

Just like Java developers use JUnit and .NET developers NUnit, it might be a good idea for XML developers to test their XSLTs and XQueries with a specialized XSLT Unit test suite. The idea is not new, but the implementations are improving. Instead of testing one whole XSLT, it is also possible to test parts of XSLTs. And this using the technology of preference: XSLT! (what else?)

A noteworthy implementation is written by Florent Georges.

Simple Outline XML

2008-09-05T18:04:00.005+02:00

For developers who don't like XML, it might be good to know that even developers who do like XML only use it when it is appropriate. For example, XML technologies like XQuery (Recommendation) and RELAX NG Compact Notation have a notation that looks more like script than like XML. But if you want it to be XML, there still is XQueryX (Working Draft) and RELAX NG XML Notation.

For some XML documents, especially tree structures, XML is not always the best human readable format. For these purposes, there are technologies like Simple Outline XML (SOX) which produce XML equivalent notations for documents that can easily be translated back to the original XML format.

The website on SOX provides this example:

stylesheet>
   xmlns=http://www.w3.org/1999/XSL/Transform
   version=1.0
   template>
   match=node()
   copy>
   apply-templates>
   select=node()>

Wish list: Automatic drawings

2008-09-02T21:38:00.017+02:00

Working with tools like Subversion and CVS in complex environments, it is easy to get a lot of parallel branches with separate developments and these become hard to maintain very quickly. In general, the best advice is to minimize the number of branches. But in some special cases, you just don't want to because you want to keep your options open. In such cases, you need to keep a proper administration of what happens where. A drawing usually helps, but making drawings takes a lot of time that can be spent on useful things....

Wouldn't it be nice if you could translate an XML file like this....

<?xml version="1.0" encoding="UTF-8"?>

</branch>

</branch>

</branch>

<label>Integration</label>

</branch>

<label>TRUNK</label>

</branch>

</branch>

</roadmap>

...to a picture like this....

...only using code like this?

<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:svg="http://www.w3.org/2000/svg" version="1.0">

<xsl:template match="//roadmap">

<xsl:variable name="main" select="@main"/>

<xsl:variable name="width" select="(branch[@id=$main]/@len + branch[@id=$main]/@t) * 3 + 2"/>

<xsl:variable name="height" select="count(branch) * 5 + 4"/>

<xsl:apply-templates select="branch">

<xsl:with-param name="merge" select="'true'"/>

</xsl:apply-templates>

<xsl:apply-templates select="branch">

<xsl:with-param name="merge" select="'false'"/>

</xsl:apply-templates>

</svg:svg>

</xsl:template>

<xsl:template match="branch">

<xsl:param name="merge"/>

<xsl:variable name="start" select="@t * 3 + 1"/>

<xsl:variable name="end" select="$start + @len * 3"/>

<xsl:variable name="pos" select="count(./preceding-sibling::*) + 1"/>

<xsl:variable name="y" select="$pos * 5"/>

<xsl:choose>

<xsl:when test="$merge='true'">

<xsl:apply-templates select="merge">

<xsl:with-param name="pos" select="$pos"/>

<xsl:with-param name="y" select="$y"/>

</xsl:apply-templates>

</xsl:when>

<xsl:otherwise>

<xsl:apply-templates select="label">

<xsl:with-param name="start" select="$start"/>

<xsl:with-param name="pos" select="$pos"/>

<xsl:with-param name="y" select="$y"/>

</xsl:apply-templates>

<xsl:apply-templates select="start">

<xsl:with-param name="start" select="$start"/>

<xsl:with-param name="pos" select="$pos"/>

<xsl:with-param name="y" select="$y"/>

</xsl:apply-templates>

<xsl:apply-templates select="text">

<xsl:with-param name="pos" select="$pos"/>

<xsl:with-param name="y" select="$y"/>

</xsl:apply-templates>

</xsl:otherwise>

</xsl:choose>

</xsl:template>

<xsl:template match="label">

<xsl:param name="start"/>

<xsl:param name="pos"/>

<xsl:param name="y"/>

<xsl:value-of select="."/>

</svg:text>

</xsl:template>

<xsl:template match="text">

<xsl:param name="pos"/>

<xsl:param name="y"/>

<xsl:variable name="start" select="(../@t + @t)* 3 + 1"/>

<xsl:value-of select="."/>

</svg:text>

</xsl:template>

<xsl:template match="start">

<xsl:param name="start"/>

<xsl:param name="pos"/>

<xsl:param name="y"/>

<xsl:variable name="level" select="@level"/>

<xsl:variable name="endpos" select="count(../../branch[@id=$level]/preceding-sibling::*) + 1"/>

<xsl:variable name="endy" select="$endpos * 5"/>

</xsl:template>

<xsl:template match="merge">

<xsl:param name="pos"/>

<xsl:param name="y"/>

<xsl:variable name="start" select="(../@t + @t)* 3 + 1"/>

<xsl:variable name="level" select="@level"/>

<xsl:variable name="endpos" select="count(../../branch[@id=$level]/preceding-sibling::*) + 1"/>

<xsl:variable name="endy" select="$endpos * 5"/>

</xsl:template>

</xsl:stylesheet>

Oh wait! You can! The only thing is that you need to convert the SVG file to a PNG file, but that can be done using several tools!

People: Process vs Information

2008-09-01T21:21:00.003+02:00

A developer may think that his favorite solution worked for 3, 5 or 12 clients in a row and therefore is the universal best solution to any problem. Such developers fail to see that these similarities probably resulted from the fact that he is hired as a specialist by the exact companies requiring his specific knowledge. It is very well possible that 98% of the other companies in the world have different requirements, but the specialist never learns about them, because these companies hire other specialists with different experience. And quite likely, these other specialists may also be under the illusion that their solution is the silver bullet to every problem.

For example: There are many discussions about the question whether RESTful web services are better than SOAP based services or the other way around. And in most cases, the answer to this question depends on the problem domain.

My idea is that both opinions can be explained by the different preferences of two kinds of developers: One of them likes to present large amounts of information from several data repositories and integrate them into an accessible web site. These people tend to work on projects creating public internet sites, requiring high quality templates and appropriate data formats storing and exchanging data.

His counterpart has spent most of his life creating rich web applications running on the local intranet within the walls of a single company. Typically such developer is much less interested in data formats because of the limited reach of the application. Far more important for him, is business process modeling, case management, workflow, security and data integrity.

The information specialist and the process expert may clash with each other for several reasons:

The information specialist spends a lot of time on details to get semantics, maintainability and formatting right. The process specialist may consider that a waste of time, because such things can be generated using his tools. The other way around, the information specialist considers generated code as poor quality. Quite likely, the process specialist is more interested in business modeling than information modeling, cares less about code quality and is better capable of meeting business requirements in a short period of time.

There are roughly two things you can do with these two developer types:

- Separate them and let them work on the assignments they are good at (or prefer)...

Or more interesting:

- Separate them and assign them to each other's work in order to learn to see a bigger picture than they are used to!

Other voices: Intellectual Cramps

2008-08-31T18:01:00.002+02:00

Intellectual Cramps is a weblog that discusses a lot of technologies related to this weblog. It is written by David Carver and contains a lot of posts about XML and Eclipse. Certainly one to add to your Google Reader!

Other voices: Versioned namespaces?

2008-08-29T19:18:00.004+02:00

This week, there was a short but interesting discussion on the Cocoon mailing list regarding the namespaces to be used voor Cocoon 3.0. The question was: should namespaces contain a version number?

In short, the outcome is:

Don't use versioned namespaces; use versioned XSDs instead!

The longer version is in the archives: this is one of the key posts...

XRX and XPath 2.0

2008-08-28T20:03:00.003+02:00

Here is the truth about XPath in the XRX architecture if you follow the specs literally:

Within the XForms spec, there is a small but important sentence though:

A future version of XForms is expected to use XPath 2.0, which includes support for XML Schema datatypes.

And that is just enough to allow XPath 2.0 usage within Orbeon Forms. It has been supporting XPath 2.0 for quite a long time already by the way.

Sneak preview: Cocoon 3.0 pre-alpha

2008-08-27T18:13:00.002+02:00

Just out of curiosity, I checked out the current pre-alpha code for Cocoon 3.0 (aka Corona). To my surprise, it took little effort to build and deploy. Like Cocoon 2.2, it is based on Spring and uses Maven in the best possible way.

Cocoon 3.0 is a full rewrite of Cocoon by the people with experience from the earlier versions. It is designed to meet the demands of modern web applications. This means:

Cocoon 3.0 aims to become the best platform for RESTful services
You can use Pipelines without being tied to the Servlet API
It is easier to integrate your own Java code into a pipeline
Superfluous packages are cut out of the platform

It will be up to the client to provide stateful Form frameworks. This makes the platform an ideal replacement for a pure XQuery backend in an XRX architecture. For example, you can think of an Orbeon Forms - REST - Cocoon 3.0 combination...

A more detailed post on the changes planned for Cocoon 3.0 is written by Reinhard Pötz, one of the driving forces behind the new Cocoon version.

Answering mail #9

2008-08-26T20:01:00.003+02:00

Brian Reindel writes:

Hi Adriaan,

I just read "Create a maintainable extensible XML format" on IBM's Web site, and I had a quick question. How come the move from validating against a DTD to validating against a schema? I am using PHP and SimpleXML, which only provides DTD validation tools, while the new PHP DOM provides schema validation. I really want to use SimpleXML instead of DOM because it is much more concise. However, I need to do validation because my XML schema will be public and must be followed strictly. Can you tell me at what point it is absolutely necessary to move from using a DTD to using an XSD?

Thanks for any advice you can offer.

Brian Reindel

Hi Brian,

I think you can continue using DTDs for as long as you like. Document Type Definitions are mature, time proven, and still used to describe even the newest versions of XHTML. Many people are using it to describe their XML or SGML format and not just to maintain the old ones.

XML Schemas are popular because they are supported by many tools and recommended by the World Wide Web consortium. That does not stop a large group of developers from preferring DTDs or another alternative like RELAX NG - which shares some characteristics with DTD. Personally, I think RELAX NG (Compact Notation) is a bit easier to read than a DTD, but that may be different if you are used to reading and writing DTDs.

The main advantage of a DTD is that you can use inline notation within an XML document to describe its element structure. However, XML Schema and RELAX NG (XML Notation) can be expressed as XML documents themselves which make them easier to manipulate / translate.

The main advantage of XML Schema is the possibility to put detailed restrictions on the content within an element. Using DTD or RELAX NG, you could put these restrictions in a separate Schematron validation with the added benefit that detailed element content validation and element grammar specification are not tightly coupled in one file.

If you consider my recent example translation to RELAX NG, you could also imagine doing the same translation to the DTD format below... (with my apologies for any errors: I do not know too much about DTDs)

The bottom line is: there is a lot of choice and it is fully up to you to decide. In this weblog, I'll try to cover more on RELAX NG and Schematron. For my own daily usage, I'd say "XML Schema, unless..." Your example is a legitimate unless, if you ask me! To stress this, here is a quote from Elliotte Rusty Harold's article "The Future of XML":

DOM isn't a least-common-denominator API: it's a worst-common-denominator API. You couldn't design a worse API for processing XML if you tried.

Kind regards,

Adriaan.

<?xml encoding="UTF-8"?>

<!ELEMENT car (brand,type,kind,(tr:tire|wnd:windscreen)+)>

<!ATTLIST car

xmlns CDATA #FIXED 'http://car.org/car'>

<!ELEMENT brand (#PCDATA)>

<!ATTLIST brand

xmlns CDATA #FIXED 'http://car.org/car'>

<!ELEMENT type (#PCDATA)>

<!ATTLIST type

xmlns CDATA #FIXED 'http://car.org/car'>

<!ELEMENT kind (#PCDATA)>

<!ATTLIST kind

xmlns CDATA #FIXED 'http://car.org/car'>

<!ELEMENT tr:tire (tr:brand,tr:type)>

<!ATTLIST tr:tire

xmlns:tr CDATA #FIXED 'http://car.org/tire'

tr:count CDATA #REQUIRED>

<!ELEMENT wnd:windscreen (wnd:brand)>

<!ATTLIST wnd:windscreen

xmlns:wnd CDATA #FIXED 'http://car.org/windscreen'

wnd:count CDATA #REQUIRED>

<!ELEMENT tr:brand (#PCDATA)>

<!ATTLIST tr:brand

xmlns:tr CDATA #FIXED 'http://car.org/tire'>

<!ELEMENT tr:type (#PCDATA)>

<!ATTLIST tr:type

xmlns:tr CDATA #FIXED 'http://car.org/tire'>

<!ELEMENT wnd:brand (#PCDATA)>

<!ATTLIST wnd:brand

xmlns:wnd CDATA #FIXED 'http://car.org/windscreen'>

Other voices: REST

2008-08-25T23:02:00.003+02:00

I tried to write something like this down myself before. But at least as interesting is to read it in someone else's words. How do you respond to this misconception?

Misconceptions About the REST Architectural Style
With that out of the way I can address the straw man argument presented in Damien's post. Damien states that building a RESTful Web Service is about using the HTTP PUT and DELETE methods instead of using HTTP POST when designing a Web API. In fact, he goes further to describe it as "the RESTful verb architecture" implying that choice of HTTP verbs that your service supports is the essence of REST.

This is incorrect.

In his post "Explaining REST to Damien Katz", Dare Obasanjo explains the advantages of REST in his words. A must read in my opinion!

People: Integration vs Standardization

2008-08-25T20:59:00.004+02:00

What do you prefer: A complete software solution that works right out of the box and is capable of solving all your problems as long as you adjust it to a model prescribed by the product? Or do you rather spend several weeks or months setting up the optimal set of building blocks you carefully selected to meet the requirements?

Considering the questions from a distant viewing point, this seems like a tough question. Many developers may pick a side with little hesitation though. And the argumentation of each side might be equally convincing.

The choice of programming languages may be a deciding factor. For example, .NET and Ruby on Rails specialists may argue in favor of fully integrated solutions: everything fits, it is meant for its purpose and it lives up to the promises in most cases. At least as long as you use the tool for the purpose it was meant for.

Experienced developers using Java or Python may be equally convincing with their explanation of the advantages of standardized components. Avoiding vendor lock-in, freedom to switch components and open data formats are some of their key points.

A standardization specialist can easily switch to an integration oriented platform. He is used to switch platforms and testing alternative components. In many cases, after making this switch, he will genuinely appreciate the ease of use for a while, and the feature richness and development speed that are typical for an integrated platform. Nevertheless he is unlikely to hesitate for a minute to give that up again in favor of independent components and standards and most importantly: freedom of choice!

The other way around might be a bit more difficult. Living in a relatively simple world of fully integrated platforms, there are many challenges in selecting your own set of standardized components that you might not know of. A common Ruby on Rails expert or .NET developer visiting the Java world is likely to spend a lot of time complaining about all the countless choices for modules and libraries. You might get a similar feeling when you try to select the best Linux distribution for your desktop computer. (Aaaaaarghl! ;-))

I hope it is clear this post only talks about differences in personal preferences. A discussion about the actual value of standardization or integration would be a bit longer. If standardization means that you are limited to using the greatest common denominator in features... And you are willing to pay the set up costs mentioned earlier... There must be some special value in standardization that is not mentioned in this article... Let's save that for later!

The more important question: is Integration vs Standardization the exact same thing as Application vs Infrastructure? Is there a significant correlation? Or are they completely separate of each other?

Answering mail #8

2008-08-23T10:45:00.003+02:00

In a comment on the post "Introducing Orbeon XForms in 2008", Erik Bruchez wrote:

Adrian, good post, and overall very postive about Orbeon :-)

Just one precision: Orbeon does not use Dojo, but YUI. But if I may ask, why do you say Dojo is the worst possible choice?

-Erik (Orbeon developer)

Hi Erik,

Thanks for pointing out that mistake. It is Chiba that uses DOJO, not Orbeon. I put that sentence between DEL-tags. It is nice to know that Orbeon developers are reading this blog by the way!

I must admit that it is not a nice thing to say that DOJO is the worst possible choice. That is an opinion and depends on selection criteria. Your mileage may vary.

For me, there are two reasons why I am not fond of DOJO.

1. Its size: The full DOJO toolkit is 4.1MB compressed, 19.2MB uncompressed - compared to 16KB compressed for jQuery and 97KB uncompressed! Even though you might not use the full DOJO tool set and even though they do offer a 23KB compressed core, the usage of large libraries is not my taste.

2. IFrames. Where XHR is unavailable, DOJO tries to fall back on IFrames. It is worth a separate post to point out the disadvantages of IFrames. The main point here is the possible security vulnerabilities.

Personally I prefer JQuery. Here is a set of slides that introduces jQuery and how it follows the principle of Progressive Enhancement rather than Graceful Degradation:

http://www.slideshare.net/simon/unobtrusive-javascript-with-jquery

It also mentions YUI and DOJO on slide 47.

Thanks again!

Adriaan.

Other voices: 97 Things

2008-08-23T08:47:00.003+02:00

My colleague, Bart Kooijman, wrote an internal e-mail with the subject "97 Things Every Software Architect Should Know":

Hi all,

When we talk about skills in software development we tend to think about the hard skills first. Java, JEE, Maven, UML, PHP, HTML, Object Oriented Design and so on. We think about them first because we think they are important. And, yes, to some extend they are. But there is more to it.

Once you start working with all these hard skills and apply them in your projects you learn the concepts behind them. And once you see the concepts you will see the concepts behind the concepts which are the general principles/guidelines in software development every developer and software architect should know and work with.

Richard Monson-Haefel has done an excellent job in writing down some of the most important concepts, principles and guidelines together with the community. It is something I have been looking for a long time. A great effort from him. Check it out on http://97-things.near-time.net/wiki.

One of the best ones I like from the site is this one http://97-things.near-time.net/wiki/show/challenge-assumptions-especially-your-own. I make this mistake still to often.

Enjoy it and hopefully learn from it(and challenge it if you will :-),

Bart.

Dividing the Java spectrum even further

2008-08-21T21:52:00.004+02:00

In two previous posts ("Reevaluating XML in the Java spectrum" and "People: Infrastructure vs Application"), I gave my opinion on how the Java community can be divided over roughly three clusters. Despite the common choice for Java, there is a noticeable difference in the tool selection in each cluster. In my opinion, this is a result of different preferences.

(Again, please note that my opinion is NOT based on official research and is NOT supported by the vendors mentioned in the picture!)

To explain the opinionated tool selection in this Weblog, I'd like to further elaborate my view on the clustered Java universe. I think each cluster has its own modeling approach, audience and future direction. Here is my opinion on these additional dimensions on top of the original picture:

Original spectrum:

Modeling approaches:

Audience:

Future direction:

What do you think? What are the additional dimensions you can think of? Send your feedback to adriaandejonge@gmail.com

Answering mail #7

2008-08-21T19:39:00.002+02:00

In reply to the post "People: Infrastructure vs Application" I exchanged some comments with Mark Berck; here is a summary - read the full conversation under the original post.

You can visit Mark Berck's blog in Dutch at http://markberck.nl/

mark berck said...
[...]
Reading your post I get the feeling that you really think that Infrastructure can't be mixed with Application, while I feel that there are situation where you don't want to separate these concerns.
[...]

adriaan de jonge said...
[...]
What are the situations where you feel you don't want to separate concerns? I am curious whether I can address these. If not, I would be happy to acknowledge I am wrong on this one!
[...]

mark berck said...
My opinion is, that for small businesses, stand allone applications, this is the perfect way to go.

But if your application is part of something bigger, the application uses shared data sources, publishes to other system, etc. I completly agree with your statements.

Mark

I see your point. I think it is wise to make a clean separation between

what is possible now
what is desirable later

Thinking about current possibilities, it is not always easy or possible to make a clean separation between Infrastructure and Application level. This is especially the case within smaller environments. The popularity of products like Joomla, Liferay and Access (your own example) are signs that moving away from technicalities maybe even more desirable in smaller environments than in large enterprises.

Products like Intalio may be too much for such environments. I am still curious whether a proper PHP / Visual Basic programmer could get effective results out of a product like Orbeon following the XRX pattern. I think such a product would be a nice first step also in smaller environments.

Introducing RELAX NG

2008-08-20T21:19:00.003+02:00

Previous posts mentioned RELAX NG as an alternative to XML Schema. There are both advantages and disadvantages to using that technology. Before mentioning all these, it might be interesting to see what your first impression is, just by giving an example.

This example is a translation from the XML Schema presented in the developerWorks post on creating maintainable and extensible XML formats. It is from the section "Design a more general format". This is RELAX NG Compact Notation:

default namespace = "http://car.org/car"

namespace tr = "http://car.org/tire"

namespace wnd = "http://car.org/windscreen"

start =

element car {

element brand { text },

element type { text },

element kind { text },

(element tr:tire {

attribute tr:count { xsd:integer },

element tr:brand { text },

element tr:type { text }

}

| element wnd:windscreen {

attribute wnd:count { xsd:integer },

element wnd:brand { text }

})+

}

People: Infrastructure vs Application

2008-08-18T20:04:00.008+02:00

If your development team is larger than 5 developers, it is likely that you have at least one developer with high quality standards. As your team size increases, there's a good chance, you have yet another, maybe even more demanding developer.

Considering you have two developers demanding high quality code, there is a good chance they have a very different opinion about what makes good code. Somehow, this does not necessarily cause conflicts however. In many cases, both colleagues appreciate each others demanding attitude and they are willing to learn from each other.

The question is whether they actually copy each other's habits though...

There are roughly two ways you can handle your code: you can consider code as a list of instructions intended to make the computer do what you want it to do. Or you can use code as modeling tools to describe the real world and make it come alive. The resulting computer program is considered an implicit result of the working model.

Instructing the computer is done on Infrastructure level. Building models on Application level requires a working infrastructure . In an ideal situation, your infrastructure logic and your application logic fit together well but are clearly separated.

If you have developers with different quality standard, the chances are that one is more interested in infrastructure level and the other is more interested in application level. This could be a perfect team.

More problematic is the tendency to mix infrastructure and application, not being aware of the potential problems of this approach. In my drawing a few posts ago, this tendency can be seen in developers with a preference for the Sun cluster:

Technology in the IBM cluster offers more options to make a clear separation between infrastructure and application. More about such technologies in upcoming posts.

Wish list: Two way XSLT

2008-08-17T22:11:00.003+02:00

You already knew that you can use an XSLT transformation to convert data from one XML format into another XML format. Something like this:

Last week I remembered that a few years ago, just before I discovered XForms, I was actually looking for an XSLT processor that works bidirectional. Meaning that after the transformation, if you change something in the output document, the equivalent values in the input document also change. Like this:

Back then, there was an alpha quality example project called Wobzilla that demonstrated a technique that works this way. XForms also works this way, but that is a bit less flexible than two way XSLT. The latter can be used in a pipeline and might even function under an XForm to simplify the input/output document used in the XForm.

How difficult is it to create technology like this that works in a Cocoon pipeline or an Orbeon pipeline?

This is a post from 2003 explaining the Wobzilla demonstrator.

Other voices: "Anything you can do, I can do meta"

2008-08-16T21:25:00.002+02:00

In January 2007, Technology Review published an interview with Charles Simonyi, the original creator of Microsoft Office and Hungarian notation. In this interview, Simonyi outlines a different way of thinking about software development that might become fundamental for the way we treat software in the next decade.

Simonyi shares much of the common dissatisfaction with software. "Software as we know it is the bottleneck on the digital horn of plenty," he says. "It takes up tremendous resources in talent and time. It's disappointing and hard to change. It blocks innovation in many organizations."

Simonyi's ambition is to unstop that software bottleneck--characteristically, by going meta. He's developed an approach he calls intentional programming (or, more recently, intentional software), which he hopes will overturn programming. If Simonyi has his way, programmers will stop trying to manage their clients' needs. Instead, for every problem they're asked to tackle--whether inventory tracking or missile guidance--they will create generic tools that the computer users themselves can modify to guide the software's future evolution.

There are many ways you can fill in this philosophy. In my opinion, pure XML platforms on top of Java foundations may become key technology to this way of thinking. Take a look at Intalio for example (which uses Orbeon XForms by the way).

Read the full article over here. Yes, it is long but it is worth reading!

XML is NOT about tree structures

2008-08-15T20:01:00.008+02:00

On first sight, XML appears to be all about trees. And on second sight, reality seems to confirm this misconception:

The internal structure of XML parsers is based on trees
Most custom XML formats follow rigid tree structures
XML critics introduce alternative notations to describe trees
Poor usage of XPath is restricted to basic tree traversal

Of course you can use XML to model trees. The title of this post does not intend to question those capabilities. XML is very well capable of capturing structured data.

The point is that most XML does not live up to its full potential. The real potential of XML is in semi-structured data. Modeling only structured data with XML means being stuck in second gear.

What is semi-structured data?

The difference between semi-structured data and structured data is in the degrees of freedom. You can mix your elements in countless variations of compositions. Instead of rearranging your data into a flawed tree format, you can choose an outline that fits the document being modeled.

This may sound abstract and theoretical or it may appear messy and unpredictable. But before you question semi-structured nature... Are you aware that you already know a semi-structured format that is a major standard available on almost any computer anywhere in the world?

The XHTML standard is the best possible example of a semi-structured format. Although browsers parse it into a DOM tree under the hood, its real world representation is a document. When writing XHTML you can think about markup freely. You're not wasting energy thinking how to force your data in an inadequate tree.

To stress the importance of its semi-structured nature, imagine what XHTML would look like if it was designed according to the tree based philosophy like most formats:

<html>

[...]

</html>

Answering mail #6

2008-08-14T17:53:00.006+02:00

An anonymous comment on the IBM developerWorks article "Create a maintainable extensible XML format" says:

The situation is simplified if one creates named types rather than adhering to the DTD convention of working with elements. The use of named types permits a one-to-one correspondence to JAVA and other object oriented languages, such as C# and Ada.

That is valuable feedback. Although I do not fully agree, it does make a new point and it is food for further discussion...

Within the article, named types are used in the section "A word about extensions". This demonstrates the advantage of named types for extensibility.

"Simplifying the situation" is not the first thing I'd think of though when talking about named types. Their purpose in XML-Object mapping is new to me, but I can imagine they have their added value there. Personally I prefer using API's like XStream for that purpose, because it just serializes XML regardless of its schema.

XStream is what I used in the developerWorks article "RESTful SOA using XML".

An additional advantage of that approach is that it also allows you to serialize XML formats that are described by a RELAX NG schema instead of an XML Schema. RELAX NG is more DTD style, like most XML Schema examples in the article.

There is another advantage of using named types that is not in the article and not in the comment, but still within the scope of the article because of maintainability. This is the fact that a named type can be reused under multiple different element names. Think for example about a <VisitorAddress>, <PostalAddress> and <BillingAddress> all containing the same <Street>, <Number>, <PostalCode>, <City>, <State> and <Country> elements.

XML Schema can do a lot more than what was shown in the post, and things that are not possible in RELAX NG. An interesting discussion is whether such detailed constraints actually belong in a schema or not. But that is a whole different post! :-)

Differences between XML 1.0 and 1.1

2008-08-13T18:27:00.003+02:00

There are days when you think you know XML. But every once in a while you meet someone who asks a very basic question that you cannot directly answer. Most of the time it is just a minor detail you can look up in the spec. Some details however, feel like very relevant details.

For example, a recent issue of the German magazine Java Spektrum discussed new XML technologies. Part of this is the difference between XML 1.0 and XML 1.1. And the question is: "is it important?"

This is a snippet from the Spec on W3C.org:

The overall philosophy of names has changed since XML 1.0. Whereas XML 1.0 provided a rigid definition of names, wherein everything that was not permitted was forbidden, XML 1.1 names are designed so that everything that is not forbidden (for a specific reason) is permitted. Since Unicode will continue to grow past version 4.0, further changes to XML can be avoided by allowing almost any character, including those not yet assigned, in names.

http://www.w3.org/TR/xml11/

This sounds fundamental. Basically it means that (generally speaking) a well-formed XML 1.0 document is also well-formed in XML 1.1. But XML 1.1 documents are not necessarily well-formed in XML 1.0.

The next question is: who is waiting for the changes in XML 1.1. Aren't many people still using XML 1.0 for most purposes?

I think this depends on the language of your format. If you are designing a format with English tags, XML 1.0 is all you need. For languages like Dutch and German, XML 1.0 is still a reasonable choice. For languages using many special characters, things may be different.

From an Agile XML perspective... Is it more agile to design a format with tags in your own language / characters than it is to translate everything to English?

developerWorks: Create a maintainable extensible XML format

2008-08-12T19:31:00.003+02:00

IBM developerWorks just released the article I wrote together with Sally Slack on creating your own XML format.

This is the abstract:

Create a maintainable extensible XML format
Reduce change when you design XML formats agile enough to incorporate new requirements

XML is a communication format for exchanging structured documents and data. Too often, an XML format is chosen arbitrarily and on the fly during development, without much planning or design. Design the right XML format up front, and you can meet the needs of everyone involved in the communication. If you don't, you're in for a long journey of format revisions. Learn how to design a format less likely to require change and agile enough to incorporate new requirements with the simple addition of new extensions instead of full changes.

The main points explained in the post:

Stop treating XML Schemas as generated technicalities
Use multiple namespaces and mix them
Mind the details
Think the XML way
Plan extensibility up front

I hope the article helps! Read more over here:

http://www.ibm.com/developerworks/library/x-extensxml.html

Other voices: XRX

2008-08-11T21:52:00.002+02:00

The newest buzzword supposed to beat LAMP over the next few years is XRX:

XForms + ReST + XQuery

At least when it comes to XML technology, this combination should be up for the job. A nice feature is the scalability of the architecture: it is easily extended with additional layers, for example in SOA environments.

In May there was a post on XML.com called

XRX: Simple, Elegant, Disruptive

This post still uses the Mozilla XForms plugin as a client. As other posts point out, the Orbeon XForms platforms qualifies for the title XRX just as well.

Read more in this post!

Answering mail #5

2008-08-10T15:53:00.002+02:00

Harvinder Singh wrote:

Hi Adriaan,

I have one doubt on Restful service. What is the defintion of Restful service?

When can I call my service a Restful service? Any web application which is not using SOAP seems to me as an Rest service.

Regards

Harvinder

Hello Harvinder,

Unfortunately that is what a lot of people think. Personally, I call most of such non-SOAP services by the name XML over HTTP.

Just a very few of those services are actually RESTful. Most important properties of these services are the fact that they are stateless (meaning no session ids or long living connections) and they use descriptive URLs to describe resources - which can be manipulated with just a very few simple actions.

They don't necessarily use XML, but that is the format of my preference.

I hope you can recognize the beauty of REST compared to XML over HTTP.

Kind regards,

Adriaan.

Answering mail #4

2008-08-10T15:46:00.002+02:00

Jeff Lu wrote:

Hi Adriann, I've just finished reading the first three of your four tutorial series. I want to drop you a note of thank you for a thorough and practical tutorial you have put together. This is exactly what I'm looking for to do dynamic and data driven forms. Please keep writing and share your experience and insight on Xforms. Just an FYI, tutorial-part-2.xhtml & tutorial-part-3-repeat.xhtml links are broken. I really would like to the sample code for tutorial-part-3-repeat.xhtml. Also if you have an example of an xform that binds several xml source documents and each xml document has its own xsd, would you post it up or email me the example source code? Thank you again Jeff

Hi Jeff,

Thanks for the positive feedback and sorry the examples are no longer there. I am working on a new series of posts rewriting the old tutorials with new and updated examples that do justice to the year 2008.

This is a first start...

Kind regards,

Adriaan.

Answering mail #3

2008-08-10T15:27:00.002+02:00

Tiago Giusti wrote:

Dear Adriaan,

My name is Tiago Giusti, I'm from Brazil, Santa Catarina Province, and a student of the highschool UNESC (www.unesc.net). I need to write a short article about SOA with XML or something else, wich will be it showed on a big event about technology here is this university in November, 2008.

I read your document: http://www.ibm.com/developerworks/library/x-restfulsoa/ and I got take something, but not everything.

Could you help me, sending to me some links, documents, texts, and more things that may help me to construct it?

I'm looking for, but how I found much documentation just in english, over myself 'cause my English, wich is so bad!!!

If you know about a documentaion a little simple about all it, with examples, images, I will be so thank you.

Take care,
Tiago Giusti

Hi Tiago,

Actually, that article was intended to be simple documentation. But the definition of "simple" may depend on your background knowledge. Some basic Java and XML knowledge is required.

The open internet may not be the best source for information. Sometimes the bookstore is a better and more friendly place for information. For example, I got a lot of interesting information recently from this book:

Web Service Contract Design and Versioning for SOA
By: Thomas Erl; Hugo Haas; Anish Karmarkar; Kevin Liu; David Orchard; Priscilla Walmsley; Umit Yalcinalp
Publisher: Prentice Hall
Pub Date: August 04, 2008
Pages: 550

This is available online on Safari.

Good luck with your technology event in November!

Adriaan.

Answering mail #2

2008-08-10T14:44:00.002+02:00

Melody Liu wrote:

Hi, Adriaan:

I have read your paper "RESTful SOA using XML" from http://www.ibm.com/developerworks/xml/library/x-restfulsoa/
I want to do a experiment about the content in order to understand more, but the code given in this paper only include servlet, no jsp and xml.
So, could you please send me the whole code project? I really want to take. Thanks in advance!

Best regards,
Melody

Hi Melody,

Thank you for your response and my apologies for the late reply.

The code example is just a very minimalistic basic sketch of what a solution would look like when you are following the solution patterns described in the article. If you're saying this is not a realistic example, I fully agree with you. If I'm not mistaken, that is already acknowledged in the text itself.

However, if you are asking for JSPs or XMLs... that is a whole different point. In the solution the way it is set up in the article, there aren't any. Here, the Java objects are representatives of the XML elements and the XStream API is used to map them from and into XML on the fly when handling GET and POST requests.

This assumes most of your business logic is written in Java and you are connecting to services the way you develop them on a platform like Spring.

Recently I started thinking about working more with native XML platforms and use Java only for the underlying infrastructure; not the application itself. Solutions as described in the article may should become a lot simpler (almost trivial) using such software.

I am still working to write that down in a series of comprehensible blog posts. I hope you are already doing well with your experiments.

Kind regards,

Adriaan.

Introducing Orbeon XForms in 2008

2008-08-09T08:49:00.002+02:00

My criticism of XForms written in 2006 is outdated. In my new series of posts, you can recognize my efforts to prove my old arguments wrong. Basically, there are three fundamental issues to be resolved:

1. Web browsers do not support XForms

2. There is no shrink-wrapped solution handling the server side of the equation

3. XForms are too complicated for the average developer

And this is the roadmap to address them:

1. The server side implementations of XForms are now mature enough to eliminate the need for XForms support in the browser. Although XForms plugins may be applied in closed environments, the open internet can be served with the current AJAX-based technology.

2. At least one of the server side implementations - Orbeon - has evolved into a full stack framework. In addition to XForms, it offers a native XML Pipeline mechanism to handle requests and integrates Exist Native XML Database (NXD) as storage back end right out of the box.

3. To get an average developer up and running with XForms, requires a complete and thorough explanation of key XML technologies. In order to create a powerful XForm, you are better off being an expert in XPath than just knowing XForms notation itself. Similarly, half of your success is in the design of the data format of the document being edited.

To be honest, the Orbeon implementation is far from perfect. For example, it heavily depends on JavaScript / AJAX libraries (~~in fact the worst possible choice: DOJO~~) and does not offer graceful degradation (yet). Nevertheless, it is the framework of choice during the first few parts of XML tutorials, because of its user friendliness and full-stack integration.

Summarizing: the upcoming posts prove the old criticism wrong by introducing a full stack XML framework running on the server side and explaining all XML technologies needed for the solution.

And again: stay tuned! Which means: install an RSS reader or use an online version! ;-)

Summary of Rebuttal in 2007

2008-08-09T08:21:00.004+02:00

In March 2007, Kurt Cagle wrote a rebuttal to the XForms vs Ruby on Rails post. There are some key points in this post that are fundamental for "Agile XML". Pay attention to the quote printed in bold font for example.

Here are some quotations from the post "xforms vs. ruby - a rebuttal (sort of)" written on March 30th 2007 by Kurt Cagle:

"Adriaan de Jonge's article Xforms vs. Ruby on Rails has created quite a stir in both the XML and Ruby communities, and for good reason. He asked a fairly important question - are XForms an also-ran technology that Ruby has managed to supplant?"

"My quibble with the argument is that to me Ruby is an apple, XForms an orange - or perhaps more accurately, Ruby is a framework, XForms is a systems component."

"[...] regardless of whether you are working with XForms or Ruby or ASP.NET, your data models are likely to be considerably more complex than a single stream of static XML"

"The XForms architecture pushes the level of abstraction up a level. In this case, the components that you interact with on the page are not actually involved in maintaining the data model - only in changing it. This is a subtle distinction, but an important one."

"This is actually a pretty profound expression of Model/View/Controller architecture."

"XForms is useless for a fair number of web development tasks."

"[...] XForms applications are most likely going to be utilized precisely in those areas where Ruby won't be."

"Where XForms comes into its own is when you're dealing with data - lots and lots of data."

"Most of the clients that I deal with when building XForms applications are not, in fact, looking at building another community site."

"I personally think that the days of shrink-wrapped software are history."

"Increasingly, software is becoming oriented towards helping people with lots and lots of data get at that data more effectively."

"It actually works better in conjunction with something else, which I am increasingly coming to believe is an web-enabled XML database working with either XQuery or XSLT (preferably both)."

"XML has become THE language of data transfer."

"[...] in an XQuery/XForms/XSLT solution, you are quite literally floating in an XML sea, where XML coming from a collection or external server is only a function call away."

"XForms has taken off in those places where it's most suited, and this even before there's a native version of XForms sitting in a browser."

"I'd far more like to see XForms and Ruby on Rails rather than XForms vs. Ruby‚ and I suspect so would your customers."

And this is one of the comments on this article:

"Thanks for these insights. Adriaan's posting was impressive, almost intimidating, so it's good to have some balance from another web tech / XML guru. Here I was wondering if I needed to abandon XForms for RoR. (Incidentally, you keep referring to "Ruby" but I think Adriaan's argument was really only about Ruby on Rails.)

[...]

So there is a real value to being able to pick one framework and stick with it. Part of the value of RoR over XForms is that RoR provides the whole stack (or at least more of it).

Lars | July 11, 2007 08:15 AM"

Summary XForms vs Rails in 2006

2008-08-09T08:13:00.002+02:00

In december 2006 I wrote down some criticism on XForms in comparison to Ruby on Rails. The current series of posts points out I no longer agree with my own old criticism. It is interesting to summarize the key point though, in order to get clear where we are right now!

Here are some quotes from the post XForms vs. Ruby on Rails written on december 24th, 2006:

"I lost my faith in XForms as a widely accepted standard supported by all browsers on all platforms."

"Rails is a so-called full stack framework."

"When I first saw source code for Rails, it didn't make much sense to me. "

"A large majority of software geeks make the world think that the main purpose of software is instructing the computer to perform series of actions. This is a crucial mistake. The main purpose of software is communication between people."

"Although very different, both XForms and RoR are solutions to the same flaws."

"Native XML databases are still an exception."

"Considering the additional effort required to handle the server side aspects of XForms, it comes nowhere near the RoR framework."

"XForms logic beats any other technique when used in XML documents with semi-structured nature."

"For someone with the knowledge and experience of a software architect, the simple tools in XForms can be the building blocks of a very advanced and intelligent application."

"A key design choice in Rails is to do every aspect of the app in Ruby and avoid writing even a single line of JavaScript or SQL."

"In my earlier XForms tutorial I learned that I can push the logical model way beyond anything I have seen in a browser so far."

"Subscribers of my weblog may have noticed that I'm not a big fan of AJAX but consider it a necessary evil for usability sometimes."

"Choosing open source software, it is easy to forget about the commercial aspects because it's free anyway."

"While I was waiting for XForms, the only thing I did with AJAX was to criticize it."

"Rails wins easily when it comes to innovation and buzzword compliance."

"XForms is more elegant than pragmatic."

"The XForms concept was born from a vision but it is being implemented like a mandatory school assignment. The key advantages of XForms have an academic nature. This is the exact reason why I like them myself. It is also the exact reason why it fails in the real world."

"Ruby on Rails is based on clear vision and goals and implemented accordingly."

"Ruby on Rails is a role model example for pragmatism."

"For me, right now, I prefer Ruby on Rails over XForms."

Answering mail #1

2008-08-07T05:21:00.003+02:00

Selmer wrote:

I liked your division figure. There is just one little thing. Intuitively I assume that technologies near the divisor are more commonly used in all clusters: Sun, IBM and Oracle. Judging from the position of XML, I suppose you think otherwise? Or do you have a different view on XML?

(roughly translated from Dutch)

That is an interesting point. I think you're right that XML is commonly used in Sun and Oracle technologies as well. The picture says more about preference than actual usage though. I think the Sun and Oracle developers only use XML as a necessary evil in order to be able to communicate with "exotic" platforms like dotNet. But if they would have the chance, they'd use RMI over IIOP any chance they get.

The Oracle guys are a whole separate chapter. I think they have a very thorough grasp on relational modeling. I usually question their object oriented modeling skills though. And I think they are not keen on semi structured documents (XML) because they're hard to store in an RDBMS.

Your last question hits the key point: most likely I have a different view on XML. Using a little bit of XML for configuration, WSDL, SOAP or BPEL is not really intriguing. Especially not if the platform puts a lot of effort in abstracting away from underlying XML technology. What you'll see on this blog over the next few weeks is the exact opposite: I'll be abstracting away from the underlying platform and focus on the XML itself!

New Theme: Agile XML

2008-08-06T20:47:00.002+02:00

The new series of articles have a common theme: Agile XML. This theme is also the new title of this weblog. At least it is more descriptive than my own name and it is a good prefix to have in the titles of the new posts.

But what does it mean?!? Stop documenting and release untested code directly to the production server???

Most certainly not! In my definition, Agile is all about involving people in development processes. This means:

Technical details should be hidden under the hood and just do their work properly.
Formats are modeled after their real world purpose and implications; not after technical boundaries.
Solutions should be open for change anytime; designs should never be fixed constants.
Implementations are written to be read by people: both self-documenting and well-documented.
Even XML technology can be combined with test driven development!
The need for non-XML code is reduced to nearly zero for solution implementations; all requirements are handled by the generic infrastructure code.
Full stack frameworks are preferred over custom integrations.

Tomorrow I'll introduce a first candidate for a full-stack framework that can compete with Rails...

Stay tuned!

Reevaluating XML in the Java spectrum

2008-08-06T05:46:00.006+02:00

From an outsider's perspective, Java might seem a strange world with many developers with many different opinions. It might even seem chaotic and hard to manage. For example, some Java developers dislike XML while others adore it.

I think it is possible to divide the Java world into roughly three clusters of related technologies, each of which has its own share of followers: (this is only my opinion...)

As you can see, I put XML in the IBM cluster. Apparently the developers who don't like XML are in the Sun cluster. Not coincidentally the same cluster where Ruby and EJB are. I think there is a reason for this.

The older versions of EJB extensively used XML inappropriately and these developers associate XML with this wrong usage. Ruby on Rails answered these feelings by minimizing the usage of XML. Within the Sun cluster, they had a point. But looking a bit further than that, it seems that the value of XML is currently underrated.

In my old post comparing Rails and XForms I argued that XForms was trying to find new solutions to old problems, where Rails is improving the old solutions to do the current tasks better. I think, Rails unintentionally inspired many Java developers to start doing questionable things with Java technology. (Like hardcoding things in Java because they are used to doing that in Ruby)

Now the Rails hype is over and started doing at least as much damage as EJB's XML abuse, it is time to clear up the mess and take a look at proper technologies. First of all, let's focus on web standards (most of which are XML based) and use technology that properly supports them...

The next few posts cover XML technology, but keep in mind that the underlying platforms better fit in the IBM cluster than in the Sun cluster!

The Sun, IBM and Oracle clusters are the opinion of the author. There is no official source claiming there is any truth in this division. Most probably, the companies themselves will disagree.

XForms tutorial part 5: The Restart

2008-08-05T21:37:00.003+02:00

Due to popular demand, it is time to reevaluate the value of XForms and make a new fresh start with the tutorials. I have to be honest about the first four parts... Not only are they a bit lengthy and unfocused... I also lost the source files and examples when my hard drive crashed and I did not make a backup.

I was a bit reluctant to rewrite the samples because many of them are outdated. But taking baby steps, I can reproduce the interesting parts in a way that fits the year 2008! This means the Mozilla XForms plugin is replaced by server side solutions like Chiba and Orbeon. It means shorter posts covering one clear subject. And last but not least: it means covering the right subject under the appropriate title.

As you might have read in the early parts, to learn XForms, you need to know more technologies than just XForms. So from this point on, you'll find short tutorials on:

XML and namespaces
XPath
XML Schema
XML Events
XSLT
XML Pipelines
XQuery
and maybe if there is some time left... XForms!

To start keeping promises, I won't make this post too long: please do subscribe to the Atom feed of this Weblog and keep an eye out for upcoming posts!

First article on IBM developerWorks

2007-11-25T19:21:00.000+01:00

Last week, IBM published my first article on developerWorks: HTML V5 and XHTML V2. This is the result of a proposal I wrote in April, just after the WHATWG proposed HTML5 as a W3C standard. I learned that the process of accepting proposals, writing, editing, accepting, editing again and publishing takes a bit longer than I had seen before, working for Radio Netherlands. This is the difference between a news site and a resource for developers: the quality is more important than news value.

Although my name is on top of the article, writing this is not completely a solo process. I am well aware of the fact that I am from The Netherlands and the English I write on this weblog is "International English" at best. Michelle Anderson edited the article into proper American English and proposed some improvements in the content. Also I like to thank Dawn Cyr, project manager at Studio B, for the nice communication in the last months. Because of this, more articles will follow!

The article covers the bigger picture behind the standards. The implementation details are well covered on other sites. For example, take a look at this article on XHTML.com: X/HTML 5 Versus XHTML 2.

But don't let that distract you from my article: http://www.ibm.com/developerworks/xml/library/x-html5xhtml2.html

;-)

JyRCP/RAP: The end of the classic AJAX webapp request/response model on the company intranet

2007-05-11T18:08:00.000+02:00

Abstract

This article introduces the JyRCP demo combining Eclipse's Rich Client Platform platform with the Jython scripting language. It anticipates on the release of the Rich AJAX Platform later this year and investigates the added value of Jython scripting to make RCP apps run on RAP with minimal effort. For internal company web applications running on an intranet, this post predicts the end of classic AJAX webapp technology and encourages the early adoption of its improved successor(s). The current client side RCP framework is tipped as a relevant training environment in order to start using RAP as soon as it is released. The combination of Jython scripting and the RCP/RAP programming model should blow away any classic webapp in terms of productivity, usability and maintainability.

A modest attempt to predict the near future

The power of Python and Ruby scripting languages for rapid improvements in the usability of GUIs is already acknowledged by a growing number of software professionals. With the current focus on both productivity and usability, we can predict that the next step in upcoming technologies addresses the exact same concerns.

The most fundamental flaw in current rich web applications is the HTML language describing interfaces to the browser. HTML can describe document models very accurately. Rich UIs follow a model very different from documents. Most of the current developer effort is wasted translating the GUI model into the inaccurate document description language. Most usability flaws are a result of limitations and lack of insight in the resulting forest of workarounds.

The next step improving both productivity and usability is the introduction of a language or programming model natively designed for rich GUI modeling. This doesn't mean the death of HTML and AJAX in general. Those are still great for document-based sites on the public internet. For web based company applications running on an intranet the new GUI technologies should be quite valuable.

Evolving intranet UIs into more usable, better maintainable applications should be possible with a minimal amount of development effort. These applications will be as rich as client side applications while they remain web based, connected and easily deployed like their classic ancestors.

Developers should be early adopters and always be two steps ahead of the current hypes. After years of struggling with request/response-interaction and inappropriate presentation languages, I recommend spending some time studying the good old client side GUI programming models again anticipating their upcoming availability in web environments in the near future.

The current Rich Client Platform (RCP) programming model allows you to create extremely rich interfaces in an absolute minimum of code and effort. The class models are designed by the world's top architects and were continuously improved over several years. The platform has proven to be capable of managing very large applications because of a flexible extension and plugin model.

The release of Rich AJAX Platform (RAP) introduces a new programming paradigm for the web. The most important question is whether experienced web developers can quit their bad habits and recognize the value of this different approach. RAP only changes the programming model, but keeps using web browser technology under the hood. Potential competitors for RAP are likely to use browser plugins. For classic web developers RAP might be more appealing than the plugin-dependent counterparts.

This article discusses an early preview release of the Eclipse Rich AJAX Platform which ports the Rich Client Platform to the browser. By familiarizing with the classic RCP framework right now, you will be ready to take advantage of the full RAP platform at the end of the year. RAP completely abstracts away from technical details like HTML, AJAX, Request/Response interaction and web server management. Most code running on client side RCP can be ported to RAP with minimal modification.

The only thing they forgot in the RCP and RAP platform, is the advantage of scripting languages like Python and Ruby. For the current RCP platform, this article explains how you can fix this by introducing Jython in your own projects. And more importantly, it explains how Jython helps with the migration from RCP to RAP once it's there!

My first open source mini-project on Google Code

This week's experiment was to create an Eclipse Rich Client Platform application in Python. The outcome of this experiment is a tiny RCP app written in Java with the sole purpose of launching Jython scripts. The Jython classes are responsible for the actual implementation of GUI logic for your application. I created a mini-project in Jython to demonstrate some basic possibilities.

To share these preliminary results with the rest of the world, I created a Google Code project containing a Subversion repository, a Win32 binary ZIP and some open issues. The project description says:

JyRCP: Create GUIs in Jython based on the Eclipse RCP platform

Minimal set of Java classes in a Rich Client Platform application launching a minimal set of Jython files (demo app) to set up a client side GUI screen.

This project intends to allow GUI developers to be able to specify an interface using limited developer tools. Notepad should be enough; a text editor with Python syntax highlighting is recommended

This does not pretend to offer a complete framework. It's just an example how you could connect two existing frameworks (RCP and Jython) together assuming their combination offers more value than they offer separately.

The advantages

In the last post on Plone, we concluded that in order to optimize graphical user interfaces for usability, a powerful scripting language is more agile than a compiled language. You write less code to implement the same functionality. This implies you need to adjust your programming philosophy to Python's philosophy and realize the priorities are a bit different. Considering you're working on a GUI and not a back-end, you'll quickly learn this is a Good Thing.

Both Java and Python are strongly typed languages. The difference is that Python is dynamically typed where Java is statically typed. This means you save time, effort and code in the declaration of variables in Python and at the same time you can always be sure that an instantiated variable never changes from one type to another by accident. 1 + 1 = 2 and never 11 like you may have seen in other scripting languages.

Proper back-ends require less agility and more quality assurance and consistence. The compiled Java language has proven its strength guarding stability and continuity with compile-time error checking, explicit static type definitions and a wide variety of supporting development tools.

A full application ideally consists of a front-end in a scripting language connecting to a back-end in a compiled language. The Jython platform compiles Python script to native Java bytecode. For the virtual machine, there is little difference between the bytecode from Java and the bytecode from Jython. The bytecode compatibility reduces interoperability issues to a minimum. The only remaining challenge is the fact that Jython scripts are typically translated to bytecode at runtime making it more difficult to get the classpath right.

The Eclipse RCP platform offers a proven programming model for large desktop applications with very rich GUI features while remaining extensible, flexible and maintainable with a relatively easy to learn class library. It is based on the combination of SWT and JFace. SWT feels more responsive than Swing (although research does not confirm this) and is better appreciated by end users because the appearance remains consistent with the native look and feel of the platform it is running on. RCP ensures a clear separation between a widget library (SWT) and a programming model (JFace). RCP doesn't force you to use JFace if you prefer to invent your own programming model. To create a usable Swing application, you are practically forced to adhere to Swing's MVC programming model, which is overly complex for small applications. If you create a small application on the RCP platform, you can create perfect results using only SWT without JFace. As your applications grow, there is an increasing number of advantages in following the JFace programming model. In this case, the additional complexity of JFace pays for itself in the form of maintainability and standardization of your code base. The JyRCP demo app uses JFace by default. Small pure SWT applications in Jython are already available on other web sites and presentations.

On the Google Code site, you find an early release of Win32 binaries offering an Eclipse RCP application loading the Python files in the /demo directory by default. If you use this as a starting point to create your own GUI only modifying the Python files in a text editor, you'll be surprised by the complete lack of overhead of continuous recompilation and exporting of the resulting application. No Eclipse installation is required as long as you can run Java code on your machine.

Disadvantages

It may be my fault, so please correct me if I'm wrong, but I had some difficulties with RCP's classloading mechanism to locate my own resource files; and the Python sources in particular. As a result, I could either hard code the Python source directory, make it a configuration property or use a path relative to the runtime work directory - which may be error prone.

Writing SWT and JFace code in Java you have the advantage of Eclipse's code assistance, auto-completion, syntax checking and refactoring capabilities. Even though you're writing more code, the support of the IDE increases your productivity enough to compensate for the advantages of concise Python code.

On the other hand, in Python, you depend on manual code documentation and API references. When you're doing something new, this may slow you down. The advantage of the concise language only pays off after gaining experience or while doing repetitive efforts.

The current RCP code documentation and code examples assume Java as the default language. Borrowing existing code takes more time because you need to translate the code first.

In general, Jython, Java and RCP all give very descriptive and useful error messages when something fails. In my demo application however, there were some cases where the application refused to start without returning even the smallest hint of an error. Probably this is also just a flaw in my demo code and can be fixed later on. Descriptive error messages are a fundamental requirement when you're writing Python code in a simple text editor.

The current version of the demo application runs on Windows XP on my company laptop. Last week, I did some successful experiments on my Ubuntu Linux machine at home. To me, it seems like the development on both platforms is exactly the same. I know there are some differences in the underlying libraries because of native widget bindings, but that shouldn't affect the code base too much. Nevertheless, I couldn't get the Win32 code base to run on Linux without modification. I hope I can solve this issue very soon.

Definitions of extension points in the plugin.xml assume implementations in compiled Java classes. The current demo app offers default Java implementations dynamically redirecting calls to Python implementations indicated by the IDs specified in the plugin.xml. A single Java class can be bound to multiple different Python scripts when the IDs differ from each other. In order to make this work without additional configuration, Java and Jython need to agree on a naming convention similar to the launching routine. In the demo app, the ID in plugin.xml can be translated to the package and class name in Jython.

Design choices and considerations

Early in this post I mentioned that the demo app is an example of how you could connect RCP and Jython. There are many other ways to accomplish similar results.

I use a few Java classes to get the application started. The Java classes use a PythonInterpreter object to launch the Jython scripts. There are some assumptions in the Java code regarding the naming of the Jython packages and classes. You might prefer some configuration over these conventions or hard-coded dependencies.

Another alternative is to use only Jython code without a single Java class. You could choose whether you precompile the classes and describe them in the plugin.xml or whether you find a way to setup the GUI dynamically without a plugin.xml file.

The Jython scripts contain import statements directly referring packages and classes in the SWT and JFace class libraries. As a result, the scripts feel less Pythonic and more like direct translations of Java code. The import statements are only necessary when you're creating new instances of classes. After instantiations, all reference typing is done implicitly. In early experiments, I introduced Factory helper objects written in Java and injected into Python in order to get rid of all the explicit SWT-imports in Python. Inspired on the "Dependency Injection" design pattern, a simplified solution provides new class instances to the scripts. The consequence is you need to develop a large set of factories representing every part of the RCP framework including even the tiniest parts. The dependencies on factories turn out to be very intrusive on the code and the creation of classes feels unnatural to experienced RCP developers.

The demo app introduces a relatively large number of packages, each containing very few files. This may seem like overkill. Each major Python class has its own .py file with a name similar to the class name. This implies a little bit of duplication in order to simplify the search for specific classes. Additional classes in a .py file can be considered similar to private classes and inner classes in Java and should preferably not be used outside the .py file.

I'm not sure what the Python convention is for package definitions, naming and code separation. I think the current packages are intuitive to work with and prepared for larger applications. The division between several concerns in the application allows future addition of a code generation mechanism similar to the Ruby on Rails script/generate scaffolding construction. Such a script might also simplify the maintenance of dependencies in the plugin.xml file. Feedback and suggestions regarding Python packages are more than welcome though.

The initial demo application is pure application code and provides no abstractions anticipating on future reuse. Some Python implementations in the demo could have been more concise by abstracting away some details to more generic base classes. At this point however, I managed to control myself and chose not to start creating a framework (yet). For the framework to be any good, it's better to wait for real world application implementations and extract only proven parts to an abstract level. Those abstractions may become either Python script or Java code, depending on specific requirements.

Future potential of the Rich AJAX Platform

My initial motivation to connect Jython and RCP was the upcoming Rich AJAX Platform (RAP). The RAP project intends to translate most of the client side RCP platform into a server side AJAX based browser interfacing framework. With the RAP platform, many existing RCP applications can easily be modified to run through a browser interface with minimal impact. Or at least in theory they should. This asks for experimentation.

I saw a challenge in creating a single piece of code that runs on both platforms without modification. At the time of writing, the early releases of RAP are impressive proofs of concept but absolutely too premature for production usage. The exact level of maturity is also part of the research question.

In the M2 release of the RAP preview, many of the classes are similar to RCP classes, but the package names were different. I considered several different solution strategies to work around the package naming problem. After some analysis, I chose Jython as the most promising option to start with. Using the factory helper code I mentioned earlier, I succeeded in getting a single Jython code base running on both RCP and RAP despite the difference in package names between the platforms. RAP turned out to be surprisingly stable, fast and responsive for as far as it is currently implemented, but there are too many key features missing to create something meaningful right now. Within these limitations it is so convincing that I'm certain the development will lead to a competitive innovative solution framework before the end of 2007 (current planning of 1.0 final indicates the end of September, this is quite an accomplishment for a project of this size that started mid-2006).

In the same week I created the initial demo, the M3 milestone of RAP was released. Apparently many developers requested to change the package names in RAP to be the same as their RCP equivalents. A wise decision, I think, even though it seemed to render my Jython strategy useless.

It didn't take long to realize there is still a lot of added value in using Jython together with RCP and RAP. The advantages mentioned earlier with the main focus on agility. But also it turns out that there will always be a few minor differences between RCP and RAP because of the simple fact that browser interfaces and client interfaces are not the same.

In the current demo code you may recognize some hints separating current differences between the platforms. I thought about releasing a version of the code that runs on the RAP platform. For now, I think the RAP platform is evolving too fast to keep up with it maintaining the examples.

The RAP version of the demo app will follow as soon as the platform reaches an acceptable level of maturity. I'll watch the progress and report anything worth mentioning.

Before I end this section, I should mention that my earlier posts on AJAX on this weblog do not agree with the JavaScript-dependent approach of the RAP platform. I would recommend against using RAP on public open websites, if not for accessibility reasons, then performance issues will stop you. However, for internal usage in companies, there are many valid reasons to consider RAP. Especially if you are using a client side RCP application installed on your company machines and you want to make it available to employees working outside the building at clients with strict system administration policies or internet cafes with limited freedom, RAP will offer a very smooth migration path from client side to server side.

Comparing with alternative AJAX frameworks and successors

When compared to other similar AJAX frameworks, RAP sets itself apart with the proven powerful and flexible programming model taken from RCP. Almost any web/ajax framework forces developers to think in terms of request/response interactions, JavaScript and HTML document element definitions. RAP takes a different approach, it completely hides all underlying web browser details and allows the developer to think in terms of model specifications, event listener registration, UI widgets as rich as client side equivalents, and reusing existing RCP implementations.

In no more than a few days, I created a very simplistic proof of concept for a potentially powerful combination of technologies. With my initial focus on the Rich AJAX Platform, I connected the Jython language to Eclipse's more classic Rich Client Platform and found out that even without RAP, the combination can be quite powerful.

These days, software development seems to be dominated by web based solutions. Without denying the advantages of web based solutions, sometimes I miss the richness of client side software development.

I have a lot of experience with Swing and I learned to appreciate its complex MVC modeling requirements. For professional purposes I haven't used Swing since the end of 2004. Since then, RCP, SWT and JFace were on my list of things to experiment with. But all I did was web applications, introducing MVC on the web, working around limitations in web browser interfaces and fix typical web app issues. If you read earlier posts on my weblog, you might notice my focus on web technology and I must admit I even started to like it.

Although I'm convinced of the future of rich web apps, there is always this little voice inside my head that says that web technology is meant for document presentation and rich user interfaces are better off with client side GUI technology. You can abuse document presentation elements to fake a rich user interface and you can combine a large set of GUI widgets to look like a document, but you'll always keep running into practical issues because the technology wasn't meant for that purpose. Even the best solution is basically no more than a very clever workaround. To be quite honest, the Rich AJAX Platform is no exception. But the fundamentally different class modeling paradigm and the high level of abstraction are impressive enough to make up for this. On the long run, I wouldn't be surprised if the RCP framework is also ported to a platform that does not yet exist today, fixing all flaws imposed by current browser technology.

Even Microsoft acknowledged this and invented XAML as an XML-based language natively designed to accurately describe rich user interfaces. If you consider that XAML is a key part of the Silverlight platform, you should see a great potential for this technology to replace a substantial number of AJAX interfaces. When used for internal company applications in particular, XAML has a value proposition quite similar to RAP's we described earlier.

Conclusion

This article anticipates on the end of the AJAX hype for creating rich GUI web applications available internally on company intranets. New technology will provide more accurate programming models for describing rich user interfaces while remaining web based like classic web applications. With many web developers locked in to classic web programming models, it might be a big step to migrate to new programming models. Developers are encouraged to get familiar with client side GUI development again, so it is easier to pick up the new web based GUI programming models when they become available. The article covers a combination of Jython and the Rich Client Platform platform as demonstrated in my mini-project on Google Code. After discussing advantages and disadvantages, it explains how RCP relates to the upcoming Rich AJAX Platform. Microsoft's Silverlight is mentioned as an alternative.

I hope this article makes you curious to find out more about Jython, RCP and RAP. It took me time, effort and perseverance to get it up and running. But you can save some time if you use my demo as a starting point.

Any feedback, suggestions and comments are more than welcome!

Tags: RAP | RCP | Jython | Python | Java | Eclipse | AJAX | SWT

Plone: The difference between a web framework and a CMS

2007-04-19T20:46:00.001+02:00

Abstract

This article introduces the open source CMS called Plone. There is a discussion of the relationship between Plone and other web frameworks as presented in an online video by Sean Kelly. This post tries to verify the claims made in the video about both the extreme ease of development in Plone and about the advantages of scripting languages for user interfaces in general. An alternative comparison is presented briefly discussing a number of other popular open source CMS frameworks. The article concludes with a general explanation how you can translate the considerations from this article into a completely different and possibly unrelated consideration of your own. Or, more accurately, it caricaturizes the wrong way of making decisions, with a wink to the video.

Discovering Plone

Plone first caught my attention when I was investigating alternatives for Ruby on Rails and I watched the Better Web App Development video by Sean Kelly. The video presents sort of a contest between the classic J2EE framework, a couple of Rails clones and Plone. The video presents Plone as one of the best performers in the contest but somehow this wasn't enough to convince me at that point. I dismissed Plone and preferred Rails and TurboGears as a new generation of agile web frameworks.

A few months later I was searching the internet for a simple content management solution to replace Joomla. Not a bad word about Joomla here. It's a great piece of software written in a not so great language. PHP is not my favorite. With most simple CMS software written in PHP, any viable alternative based on Python is likely to catch my full attention.

At the time I was searching, I had forgotten about Plone from the web app video. When I saw the name, I did not immediately recognize it, but it did ring a bell. The front page on plone.org impressed me, but still I didn't connect the name to the Better Web App video before I started watching the Plone screencasts, not coincidentally also by Sean Kelly.

On my second encounter with Plone, I did not dismiss the software at all. On the contrary, I thought it was the most brilliant CMS innovation I had seen in years. I liked the internationalization, the W3C standards compliance, the wide availability of plugins, the user-friendliness, the matureness, the development activity and most of all I liked the choice for Python as the implementation language.

While recognizing the strengths of Plone, I still did not consider Plone as a competitor for Ruby on Rails or TurboGears. Those are meant for completely different purposes. The things you can do in Rails with ease, require a lot of effort in Plone. At the same time, Plone comes with a rich set of CMS-features out of the box, far too specific for the more general-purpose Rails framework. There are CMSs available based on Rails, but they don't fall in the same category. If all you need is simplicity and a powerful platform, they might work for you. If you're looking for many ready-to-use features and a pluggable architecture, there is no reasonable Rails-based alternative to Plone right now. I have good hope this will change over time and there already are some developments with this potential.

Plone and the video

Over the past few weeks, I started thinking about the video. I asked myself whether Plone actually belongs in the better-web-app-video or whether it was accidentally mixed up in the wrong contest. And I quickly realized there is no easy answer to this question. Your answer to this question depends on your point of view when you're watching the video. Your point of view is likely to be different from mine and it is likely to change over time when you're working on different projects.

Right now, I can think of roughly three perspectives to watch the video:

1. Choosing a buzzword-compliant full-stack web development framework (a Rails clone)

2. Choosing a ready-to-use platform for content management minimizing the time spent on development

3. Learning about the advantages of powerful dynamic scripting languages when developing rich graphical user interfaces focusing on usability

Up until now, I only considered the first two perspectives. With just those two perspectives in mind, the choice for the combination of Plone and the other frameworks in the video might raise some questions. In this case, the only logical explanation is Sean Kelly borrowing the Rails-hype as a clever way to promote his favorite project called Plone.

Maybe he does, but it is more likely that Sean Kelly has a legitimate reason for presenting Plone alongside the Rails clones when considering the third perspective. I had to watch the video for a second and third time before I realized his focus on the advantages of scripting languages for developing friendly highly customized user interfaces evolving from customer feedback.

Depending on your opinion, you might say Plone is more mature than the other frameworks or you might think Plone is a bit old-fashioned in comparison to the latest hypes. The video claims extreme ease of development for Plone. Some sources on the internet mention a steep learning curve.

The truth is somewhere in the middle and depends on your background. You need some basic knowledge of the machinery before you can do something useful. Experience with other CMS software will help you understand how this machinery is a well-considered solution guiding you away from many practical issues and helping you create modular and pluggable solutions.

For more advanced usage of Plone, you need to learn about the underlying Zope framework. Zope offers a lot more power than you're actually asking for and may overwhelm you with things that aren't immediately relevant to you. This complexity is compensated for by the wide availability of ready-to-use and free plugins for any purpose you can think of. These plugins are non-intrusive, easily customizable and very powerful. The architecture allows more experienced developers to package any customization as a plugin by itself making your own software as flexible and reusable as the framework itself.

But no matter how pluggable and flexible, I keep having ambivalent feelings about some claims made in the video. The video says Plone requires zero lines of code for creating a time logging application and it claims that zero lines of code means you can't make development mistakes.

I found out that even without writing code, there is still room for errors. If not in code, then there must be an alternative way to specify a model like a point-and-click interface. Even if you do manage to point-and-click your way through an UML modeling tool without making errors, then there still are configuration details offering room for failure. It's an innovative approach with unique advantages over coding, but it just isn't as easy as the video claims.

Initially, this was not what scared me away though. I think the creation of the first Hello-World example gave the wrong impression. I still think it is a bit strange that "hello world" takes so much effort if a complete interactive application is possible without a single line of code.

I know there is a good explanation best summarized with the keyword "Zope". The video didn't clearly choose between plain vanilla Zope (for hello world) and full featured Plone (for the time logging app). As mentioned earlier, Zope is so powerful that it may seem scary if you haven't seen it before. If you search the web, you may encounter some criticism of Zope, being overly complicated, unpythonic, old-fashioned or complete overkill. But you'll also find counter-arguments explaining Zope has been ahead of time for years and it is catching up with the latest pythonic insights in Zope 3, which was built from scratch to combine new insights and lessons learned from current flaws with all the successful aspects of the existing framework. A more incremental approach is "Five" (Zope 2 + 3) backporting Zope 3 improvements to the existing software which includes the latest stable release of Plone (2.5).

You may still criticize Zope 3, but I'd rather say: Zope is an acquired taste as a web framework, and the Plone CMS is the reward you get for learning Zope.

Zope doesn't compete with Rails; just like in many sports the heavy-weights champion does not compete with the light-weight champion. For most home-grown applications, a light-weight framework is recommended. But the Plone CMS is the living proof that a heavy-weight solution may be the most robust foundation for ready-to-use solutions on a larger scale.

Real world usage

From my recent experience working on a Plone web site, I learned that Sean Kelly is right about the advantages of scripting languages for developing highly-customized user interfaces. The only times I needed to restart Zope, was after the installation of new plugins. During development, every other change was visible instantly. And in contrast to my experience with other environments, I was never afraid of breaking the system or making errors I couldn't quickly correct.

Custom changes are always completely separate from the original code. An undo-tab allows you to revert to older versions. The very limited costs of testing even the smallest changes encourages you to get constant feedback for every step in the development process. Basic errors in typing or formatting result in descriptive error messages often pointing out the exact line and position of the failure in the code.

The extremely modular approach for separating partial templates, style sheets and scripts is puzzling for anyone new to Plone regardless of technical skills. The locations of the snippets are not intuitive but do turn out to follow logical rules after getting familiar with the details. A few hours of playing around is enough to recognize how all these separated snippets relate to each other as parts of a very clever solution with unique potential to be both flexible and pluggable.

After my first steps in Plone, there are some things I should further look in to. If they turn out to be worth discussing, they may be covered in future posts:

Packaging the current customizations as a plugin product
Creating my own plugin with custom item types
Using the object oriented nature of the backend for very flexible and powerful behavior described by larger class diagrams

Back to the video

I agree with Sean Kelly that script languages are the preferred technique for creating and customizing graphical user interfaces for both client side applications and web based solutions. To be safe, I would like to add that some script languages are more appropriate and powerful than others.

Without having investigated every scripting language available, I think it is safe to say that both Python and Ruby in particular are outstanding for this purpose. As a result of the power of these languages, I'd say modern development frameworks based on Ruby or Python have a tendency to be even more outstanding as long as they are used for the exact purpose each framework is specifically designed for.

Comparing web frameworks shouldn't be about winning or losing in general. It should be about finding the most precise and clear definition of the problem at hand and investigating which framework is best suited to solve that problem. This explains why Plone is only appealing when you're searching for a CMS and why it seems a bit odd when you're solving any other problem. This is also why Sean Kelly starts with a specification of his requirements - which are likely to be different from yours and mine.

Comparing open source CMS software

When you intend to make a contest comparing Plone to its alternatives, it would make more sense to investigate open source CMS software instead of general purpose web frameworks. There are well known CMS solutions with a large audience and an active developing community. To mention a few, I think some reasonable alternatives to Plone are Joomla, Drupal and Alfresco.

Each of those has its own set of unique selling points in many regards, ranging from functionality to server limitations and from target audience to standards compliance. A full objective comparison is way beyond the scope of this article.

In order to make a point and without pretending to present proper research, what follows is an extremely subjective and narrow-minded comparison under the unrealistic assumption that the underlying technique is important enough to reduce every other aspect to a trivial side remark or less.

Being a software developer being naive and forgetting about the social aspects of software, I (falsely) assume the technical quality attributes make all the difference and with the right choice of technique my development efforts are stimulated enough to make up for lacking functionality.

From this technical perspective, I appreciate elegance in the design, aesthetics in the code, maintainability and flexibility of the framework, fault-tolerance, automated testing, standards compliance and a relatively high level of skills found in the community behind the product.

These are still way too many factors to consider in this brief discussion. For the sake of argument, let's assume there is a single discriminating factor implicitly covering for all these other factors.

Let's assume the implementation language is the single all-explaining discriminating factor when choosing a CMS. This helps us oversimplifying our comparison between Joomla, Drupal, Alfresco and Plone as a comparison between PHP, Java and Python.

Under all these assumptions, I can present some equally narrow-minded and prejudicial claims for each language. And please don't flame me for these, the undertone should be more than clear by now:

A poor and error prone scripting language with a large audience of hobby users (PHP; Joomla and Drupal)
A powerful but inflexible and time-consuming programming language whose main strength is found in larger enterprise application backends (Java; Alfresco)
A mature and powerful agile object oriented scripting language best appreciated by the most innovative happy few of developers on this planet, who were responsible for some of the most ground-breaking improvements over the past few years, for example: GMail, the GMailFS hack, Blogger, the first Podcasting software, BitTorrent and SpamBayes (Python; Plone)

We can further summarize these claims to the following generalizations; without even the smallest piece of supporting evidence or credible source other than a vivid imagination:

Me-too-software (PHP)
(Over)regulation-is-strength-software (Java)
Pragmatism-and-elegance-don't-necessarily-bite-each-other (Python)

The other contestants are no match for Python. It's sad there was no Ruby CMS offering some tough resistance. This means Plone is the winner on all my strict criteria and Alfresco is the runner-up that may prove to be valuable in large enterprise settings. Both Drupal and Joomla share a third place being the environment of choice for hobby usage or low-budget websites.

This last paragraph wasn't completely nonsensical, or was it?

As crappy as it is, this unfair comparison is no more and no less than a caricaturization of the unfair comparison in the video by Sean Kelly. And even though that comparison is unfair on the contestants, the video is still making some valid points in general.

Concluding

Rather than recommending Plone, I recommend thorough investigation of both problems and solution alternatives. Rather than selling my own preferences, I'd like to show an alternative view on things and let you decide for yourself based on your specific situation. There are numerous aspects way beyond this article that may outweigh any of the arguments made in this post.

Choosing Plone means you should value the specific power of Plone enough to be willing to accept the learning curve. I can imagine that a majority of people prefers quick gains on the short run over more abstract qualities on the long run.

In my opinion, Plone is an acquired taste.

And so are both Python and Zope in many ways. Like any other acquired taste, there is a small number of fanatic people who completely adore it and are willing to make major sacrifices in order to get the product of their preference. At the same time a majority of other people doesn't recognize the difference between an outstanding product and a cheap surrogate. In many cases, they might even prefer the surrogate because all the convenience stores are offering it off the shelve.

Having an acquired taste, it's easy to fool yourself into feeling superior and you might be too arrogant to see that your elite toy is not the ultimate answer to all your problems. There is no such thing as a silver bullet and there never will be. Put the focus on the problem before you focus on a potential solution and consider all alternatives, including the ones you disapprove of.

And if you happen to be in the fortunate position you're selecting an open source CMS, you have a proper technical background, enough time before the deadline and a dedicated web server of your own... Then take a look at Plone and decide for yourself whether it suits your needs.

N.B. If you haven't seen it yet, please take a look at the Better Web App Development video by Sean Kelly I mentioned in the first paragraph. Or watch it for a second time after reading this post.

Keep in mind some claims in the video are outdated. Agile frameworks evolve more quickly than anyone could update a video. For example, manual database creation in Rails has been replaced by automated pure-Ruby migration functionality for a long time now.

The general message in the video hasn't changed though. And I doubt it will any time soon.

Tags: Plone | Python | Zope

XForms vs. Ruby on Rails

2006-12-24T19:34:00.001+01:00

THIS POST IS SUPERSEDED BY A NEW SERIES OF ARTICLES ON XML TECHNOLOGIES AND XFORMS... READ THIS NEW POST AND ALL FOLLOWING POSTS...

Introduction

The title of this post may surprise you, but it's not as ridiculous as you think. I see value in both techniques within the same problem domain. The comparison in this article can be described as "doing things a different way" vs. "making the existing way a whole lot easier".

Any experienced web developer intuitively feels there's something wrong with traditional web development. Developers embrace new frameworks like Struts, JSF, Spring, Hibernate, AJAX, etc. because they recognize the improvements in those frameworks. Before seeing these solutions, they have little explicit knowledge about the problem they're solving though. The choice of techniques is a bit more random than your customers might have hoped.

From XForms to Rails in less than a year

In the exact same way I found out about XForms a few years ago. A funny detail is that I wasn't interested in web development that much. A lot of it's added value also applies to the world of client side applications.

Innovations in ICT are hard to predict. The value of technical solutions greatly depends on customer adoption, hypes and popularity. I saw potential in XForms and I can still make a good argument for their adoption. Nevertheless, I lost my faith in XForms as a widely accepted standard supported by all browsers on all platforms.

Although elegant, XForms are overly complicated. Although an XForms plug-in is being implemented in Mozilla, the development of this plug-in takes longer than the birth of a human baby. Although standardized by the W3C as part of XHTML 2.0, XForms are widely ignored as "the next big thing".

In the mean time, the world has been talking about AJAX, social networking and mash-ups. A few clever developers became well known for inventing simple mechanisms to apply the latest innovations in your own web sites. These simple mechanisms are available free of charge, they are not standardized but so widely adopted that their continuity is guaranteed.

One of the greatest surprises in this category is the Ruby on Rails framework. While everyone was locked in to Java, dotNet, PHP and Python, who would have thought of using yet another unpopular scripting language with a questionable reputation for performance? Who would have thought it was even remotely possible to convince experienced Java developers to switch to this unknown alternative?

Who would have thought of actually explicitly investigating the actual flaws in web development before figuring out what solution solves the flaws appropriately?

David Heinemeier Hansson did. He found out that the greatest evil in web development is caused duplication of knowledge through the application. By focusing only on duplication, he found out that the best solution is not in a framework. The elimination of duplication starts with a programming language designed for this purpose starting at the bottom.

The best development frameworks are not designed to be frameworks by themselves. Instead, adequate frameworks are extracted from successful software implementations. By continually improving software design and abstracting away from details, a project reaches a certain point where abstraction and implementation can be separated. The abstraction without specific problem implementation is the foundation for the new framework.

The majority of web applications are built by solving similar issues already solved thousands of times by other developers. The repetitive effort of solving common issues can be avoided by reusing the framework abstractions from earlier solutions.

This is exactly what Rails is about. Solving every-day challenges in the simplest possible way. Common frameworks are limited to providing reusable implementations. Rails does more than just that. It is designed to prevent any knowledge duplication in your web applications. And it does this amazingly well.

On top of that, Rails has some extra selling points:

Rails has built-in support for all the latest hypes, buzzwords and innovations. Using AJAX in Rails is simplified to be as trivial as any other standardized feature.
Rails is a so-called full stack framework. This means that it helps developers with every aspect of a common web application. Including database management, object-relational-mapping, modeling business logic, separating presentation from implementation (MVC) and many more details.
Specialized plug-ins can be injected into projects for less common requirements. These plug-ins are designed to be non-intrusive and loosely coupled.
Rails encourages its developers to further extend the platform following the philosophy of the existing components.

Obviously there has to be a down side to Rails...

At least there seems to be a down side to Rails at first sight. When I first saw source code for Rails, it didn't make much sense to me. Even though I already knew the advantages and supported its philosophy, I thoroughly disliked what I saw and tried to convince myself that it's just another framework like all others. Nothing that can't be done similarly in Java or any other language.

For a while I experimented with Python on TurboGears. Python already had my sympathy, TurboGears is a pythonic full-stack framework in the Rails philosophy and it seemed to me that the widely adopted Python would blow away Ruby.

My initial positive experiences with TurboGears opened my mind to really consider Rails as a serious alternative. My colleague, Jeroen Zwartepoorte, used the power of demonstration to convince me of the potential of Rails. In no more than two days he ported an existing Java web app to Rails adding improvements on the way. Seeing great and elegant results in so little time, I could no longer deny there must be something about Ruby on Rails to make it special and unique. I started studying the RoR framework for real.

I analyzed that my main objection to RoR was the fact that I couldn't intuitively read Ruby code. Because code readability is on top of my priorities, I started with the basics of Ruby. When I put my mind to it and translated my objections to open questions, it turned out that Ruby is just as easy as Python. In fact there are a lot of similarities.

These are some of my conclusions:

Python wins over Ruby for basic readability because I didn't need to learn Python to be able to understand the code.
Python wins over Ruby on performance aspects because Python classes are compiled to byte code on the fly.
Ruby wins over Python when avoiding duplication without introducing tightly coupled components because of advanced dynamic language features.
Ruby wins over Python for the declarative nature of its source code. The notation of method calls make them seem like sets of rule specifications.

The final step converting myself to Ruby on Rails follows logically from my fundamental development philosophies.

A large majority of software geeks make the world think that the main purpose of software is instructing the computer to perform series of actions. This is a crucial mistake.

The main purpose of software is communication between people. Instructing computers is just an important means of achieving this goal. The best software is a piece of art; poetic literature expressing the thoughts of the developer for future generations. To be maintainable, it has to be readable. To be usable, it should be maintainable to extend functionality. For performance, it must allow minor modifications without harming readability.

Comparing XForms and Rails

If I recall correctly, I started this article commenting on the title. At this point in the discussion, it should be more obvious why the comparison is more reasonable than it seemed at first. Although very different, both XForms and RoR are solutions to the same flaws. At this point we can list flaws and compare techniques.

Relational databases have a static nature and take a lot of effort and SQL knowledge to maintain. Software deployment requires manual action for database modifications. Transferring data between similar but slightly different database definitions cannot be done without affecting the quality of the data.

XForms treats data structures as semi-structured XML documents and assumes they can also be stored in a Native XML Database or on a file system. XML Schema definitions of data structures are easier to maintain and distribute. Also, they can easily be extended for specific problem domains without harming the quality of more general fields in the document. The maintenance problem is solved by doing things a completely different way. When applied to the appropriate problems, this works extremely well. Note that I'm talking about theoretical back-ends for XForms here. These back-ends are not defined by XForms in any way.

Ruby on Rails introduces the concept of versioned database migrations expressing metadata definitions in native Ruby instead of SQL. Changes in the database later on are automatically rolled out when updating the web application. To be safe and complete, migrations can also be rolled back if you want to revert to an old version of your application. Rails solves the maintenance problem by making existing techniques easier and standardized in the web app. It reduces knowledge requirements by providing native Ruby helpers.

Native XML databases are still an exception. Most companies are used to working with relational databases. To stick with the standards makes it easier for your clients to adopt your technology. For this reason I think that Rails is the winner here.

Native access to databases is unnatural in programming languages and mapping databases to object oriented code takes a lot of effort and duplication of logic

XForms use semi-structured document definitions instead of structured table definitions. Semi-structured data allows more degrees of freedom while preserving definition of all items in the document. Because XML can be stored in native format, there are no mapping issues. XForms solves the problem by doing things differently.

Rails expresses data models extending a technique called ActiveRecord. Duplication is avoided by dynamically reading column names from the database instead of specifying them in the model. In code, the column names can be accessed with the same notation as explicitly defined fields. Under the hood, Ruby uses a mechanism similar to Dynamic Proxies in Java. The key difference is that Java requires explicitly defined interfaces to implement its methods. Being a script language, Ruby just sends a message to a generic dynamic implementation. If you specify the nature of relationships with other entities in the model, the same dynamic implementation helps you to find related data for specific items.

The reasoning for the previous conclusion still applies here. I must admit that ActiveRecord avoids duplication. At the same time I prefer having my program logic in a single place and not scattered over both the database and the model. Even though I could read the column names in the migration code without accessing the database, it makes the code more error prone and harder for IDEs to offer auto-completion and auto-checking. The TurboGears platform has a different approach. At the cost of maintainability, there are no migrations. Column definitions are part of the model class. Databases can be generated from the models automatically. Because TurboGears is not part of this contest, I call this issue in favor of XForms. Not only because it works around the problem, but mostly because of the extensibility of the solution while preserving standardization of shared data definitions.

Proper software development separates presentation logic from business logic and use case implementations (MVC pattern). For new projects it takes a lot of repetitive efforts to set up a basic framework for MVC.

XForms is part of the XHTML 2.0 standard natively designed to separate MVC concerns on the client side. XForms is different from traditional HTML and client side forms because it keeps the model document in memory and processes changes in the model real time. In the XForm itself, there is some mixing of presentation with business rules. The presentation details are in CSS as usual. XForms try to eliminate the need for controlling code by abstracting them away to the form of business rules.

Rails is natively built up following the MVC pattern in a way that is almost obsessive. Rails encourages its users to separate MVC concerns by making it easy, logical, natural and intuitive. Up until now, I considered MVC to be difficult for inexperienced developers and only appreciated with substantial work experience. With the additional steps required to follow MVC in Java, beginners are easily lured into doing things the ugly way "temporarily, just for now, I'll fix it later on". Because Rails prefers convention over configuration, following MVC is easier than trying to mix things up. Some special properties in the Ruby language make method calls seem like declarations of logical business rules. Notation outweighs actual implementation here.

When considering the separation of models, views and controllers (MVC) of XForms and Ruby on Rails, I should take into account that XForms is just a client side helper and RoR is a full-stack framework. Considering the additional effort required to handle the server side aspects of XForms, it comes nowhere near the RoR framework. If I ignore this issue to make a fair comparison, I still think RoR wins over XForms. The declarative rules in XForms are nice for common implementation concerns. The limited size of the solution library in XForms are nowhere near the toolbox provided by Rails. Rails wins in the MVC department.

Traditional web applications impose a lot of duplication on developers. Duplication is avoided most easily in special purpose frameworks. These frameworks have something in common: they avoid duplication by choosing declarative problem specification models over executable programming instructions. Given the model, the implementation is taken care of under the hood.

Declarative specification of business logic targeted at XML structures is one of the key selling points of XForms. This is proven by the fact that OpenOffice borrowed the logic model from XForms without implementing the rest of the XForms standard. Specification and implementation of business logic is a very powerful feature in XForms, but the domain for its appliance is limited. XForms logic beats any other technique when used in XML documents with semi-structured nature. XForms logic is based on XML specifications and XPath queries. This notation requires a thorough understanding of XML, creativity with XPath, trial and error, and great talent for logical puzzles. For someone with the knowledge and experience of a software architect, the simple tools in XForms can be the building blocks of a very advanced and intelligent application.

As mentioned earlier, Rails is bordering obsessiveness when it comes to duplication. If you take a look at the source of a Rails generated web page, you might conclude otherwise though. A lot of JavaScript and XHTML structures seem copies with minor variations in the details. This is remarkable but it doesn't say anything about the web app on the server side. A key design choice in Rails is to do every aspect of the app in Ruby and avoid writing even a single line of JavaScript or SQL. The JavaScript code you see on the client side, is a result of a Ruby code generator. Calling the code generator requires a method call to a helper. Because of the syntax of Ruby, the call seems like a simple rule specification to the reader of the code. On the back-end side of the application, Rails uses tiny declarative-seeming method calls to mix in advanced functionality or modifications in behavior. The only downsides of these dynamic capabilities arise when debugging a complex model and it is hard to figure out the origin of its behavior. This debugging flaw is comparable to aspect oriented programming in Java.

Compared to other regular programming languages, Ruby's declarative features are outstanding. In the exceptional comparison to XForms it does a good job when it comes to simplicity. To draw a conclusion I have to acknowledge the fact that RoR rules only appear to be declarative while in fact they're just method calls. This is sufficient for calls in a limited scope. When looking at the bigger picture, business logic is best described in a rule base supported by a logical inference engine. Rules can recursively influence any other part of the model. Rules can be processed by making queries to the engine without worrying about it's implementation. In my earlier XForms tutorial I learned that I can push the logical model way beyond anything I have seen in a browser so far. Rails wins on avoiding duplication, XForms wins on processing pure declarative logic.

Many companies are replacing their client side applications with web based alternatives. IT departments love this, but end users are missing some of the richness and responsiveness of their old clients. The latest trend is to put usability on top of the priority list.

The second most important selling point of XForms (after declarative logic) is the built-in richness of the interface. The old limited form-widges from HTML are replaced by special-purpose XForms widgets. The widgets and control elements are designed to be bound to both the source document and to logical declarations. As a result, the widgets inherit dynamic behavior from the model and interactively modify the screen when the model rules require them to. Not a single line of JavaScript is involved. Fine tuning of the appearance is based on CSS with support for specialized XForms properties. The XForm can load multiple data instances at the same time. As long as all data is available client side, a complete and rich application can be offered in a single XForm without a single round-trip to the server when changing the model or its presentation. Dynamically fetching server data in an elegant way later on, is still an issue though.

Subscribers of my weblog may have noticed that I'm not a big fan of AJAX but consider it a necessary evil for usability sometimes. Part of my disliking AJAX is based on its cumbersome notation, its dependency on JavaScript and difficulties in creating a stable end result combining multiple server interactions. It turns out that Ruby on Rails is the first and only innovation that takes away my concerns. As part of its support for hypes and buzzwords, Rails simplifies every aspect of AJAX interactions out of the box. The dependencies on JavaScript are completely taken care of by Ruby helper classes. Server side handling of common calls like in-place-editing only require some rules in the controller. Advanced GUI features can be invoked by Ruby code generators in specialized AJAX templates.

XForms offers great value but only for a limited domain. Fast responding dynamic forms with advanced data structures are a lot easier in XForms than in Rails. Advanced forms are great for administrative business application running on an intranet. For the great majority of web sites forms are small and simple. XForms logic is complete overkill here. Because the simplified AJAX support in Rails can be used for an endless number of different purposes, Rails wins when it comes to adding dynamic behavior.

Choosing open source software, it is easy to forget about the commercial aspects because it's free anyway. All software contains bugs. You may encounter them or you may use the software in such a way the bug is never revealed. The first time you start thinking about commercial aspects is the moment of finding a bug. Who can I ask for help? Who's going to solve it and when will the solution be available? Is the producer of my commercial application willing to support the product when it depends on open source plug-ins? Can my plug-in be replaced by another standardized alternative?

XForms is an official W3C standard and part of XHTML 2, the successor of the widely adopted XHTML 1.1. As long as XHTML 2 is a draft and no attempts are made to offer implementation in Internet Explorer and Firefox, there is little value in this standardization. Agreed, you may transfer your XForm to a commercial implementation based on AJAX or Flash. But it just isn't the same. The Mozilla XForms plug-in project is taking too long to release a final version.

Ruby on Rails is not an official standard, but its de-facto-standardization value is a lot higher. There are regular releases of stable new versions, there is an open development process releasing ongoing work frequently. There is an enthusiastic community on the internet offering help provided you ask the right questions. RoR follows the latest W3C standards for producing web content. It uses other open source components for specific functionality. There is a growing number of independent companies offering support for Rails. The working with rails site lists specific consultants in your area. Given the successful adoption of RoR in production environments with millions of daily customers, assuming capable technical skills, Rails is a safe bet. Performance requires some fine tuning but that's no different in any other environment.

Personally I prefer de-facto-standardization over official standardization. Ruby on Rails wins easily with its fast paced adoption.

Software in 2007 should be innovative, rapidly evolving and buzzword compliant.

Initially, XForms was something new and fundamentally different. The open source community behind Mozilla decided against its implementation because it was too complex. A few large companies saw the value of XForms for their own applications and pushed the decision to implement an XForms plug-in in Mozilla Firefox. For being innovative, rapidly evolving and buzzword compliant, the companies should have invested in a large development team and delivered the implementation in less than 3 months. This way, they would have been earlier than the top of the AJAX hype and may have had a chance to compete. With the slowly evolving Mozilla plug-in, XForms lost momentum and became so-yesterday! I think it is a huge waste of potential.

While I was waiting for XForms, the only thing I did with AJAX was to criticize it. Judging my weblog statistics, this is a popular place for people searching constructive criticism on AJAX. History points out that nobody became famous for criticizing others, not even in the most constructive way. I'm glad that some other people took the criticism to improve the technique. As a result of this effort, we can now create web sites that work with AJAX when JavaScript is available and still work without AJAX when JavaScript is turned off. Combining usability with accessibility is a great strength of Rails. Now let's hope all developers take the effort to go all the way to accessibility and not just focus on usability. In the latest version of Rails 1.2, there is improved support for RESTful URLs and services. I already liked REST long before I found Rails.

The conclusion is almost trivial here. Rails wins easily when it comes to innovation and buzzword compliance. Rails deserves extra credit for having clear visions and goals in the adoption of new technology. Innovation makes sense in Rails, it's not just there because it happens to be new. Especially for an open source project, clear visions and goals are a strong asset. It is a sign of strong leadership and a no-nonsense pragmatic community.

The end conclusion: successful projects are based on pragmatism

XForms is more elegant than pragmatic. It is a solution designed on paper instead of extracting it out of the real world. It is designed to solve common real world problems in a limited scope and fails to further evolve from implementation experience. The XForms concept was born from a vision but it is being implemented like a mandatory school assignment. The key advantages of XForms have an academic nature. This is the exact reason why I like them myself. It is also the exact reason why it fails in the real world. History teaches us that all famous theories, models and ideas in any possible science, have one thing in common. Simplicity! The complexity of the XForms specification grew out of proportions. No amount of pragmatism is able to fix that once the harm is done.

My proposal is to split XForms into sub-components and improve each of those separately:

Declarative logic component
Separating data instance document from XForm doc
Support for semi-structured nature of data
Standardizing common client side code in declarative library
Submitting data in XML and other formats
Allowing more dynamic interface modifications without JavaScript
Native support for rich UI controls like date choosers and sliders
Support for fine-grained CSS, dynamically changing presentation details based on the state of specific elements
Simplification of XPath element query specification

Ruby on Rails is based on clear vision and goals and implemented accordingly. RoR finds it foundations in the real world and continually picks up on feedback from the community, new trends and innovations. As new versions are released, the maintainers of the project are not scared to remove superfluous deprecated functionality. They even decide to drop complete modules like SOAP-based web services when it no longer fits the overall philosophy. Old modules remain available as plug-ins for those who need them. Innovation does not stop the maintainers to keep improving existing work. Remaining duplication is removed, overly complicated features are simplified, minor feature requirements are added in the code base.

Ruby on Rails is a role model example for pragmatism. In Rails, pragmatism never gets out of proportions. There are high standards for code elegance, readability and standard-compliance. While rapidly developing to meet short-term goals, Rails never forgets about quality on the long run. It seems to acknowledge that long term quality wins over quick and dirty even in short term development.

Looking at the simplified support for AJAX and thinking about XForms dynamic visualization of ongoing changes I suspect Rails can push the AJAX framework even further to match the power of XForms.

I think this requires the following features in Rails:

Rule based inference engine influencing object models with a much wider scope
Simplified native support for presenting subforms with add/remove/up/down buttons
Improved automated support for updating multiple parts of a web page based on changes anywhere in the model
Support for Native XML Databases instead of Relational Databases
Binding model classes to XML documents instead of SQL tuples

Conclusion

The end choice between XForms and Ruby on Rails can be simplified by rephrasing the question. If you have to choose between never using XForms anymore or never using Ruby on Rails in your whole life, which one would you choose?

The only risk there is that XForms is a W3C recommendation and might become a widely adopted standard after all. Don't sweat it, this is just a hypothetical question based on current knowledge helping the decision.

For me, right now, I prefer Ruby on Rails over XForms. If not for other reasons, then mostly because it offers a lot more features in a larger problem domain. I'm convinced Ruby on Rails can be very helpful in my daily work and saves substantial development time.

Tags: XForms | Ruby | Rails

THIS POST IS SUPERSEDED BY A NEW SERIES OF ARTICLES ON XML TECHNOLOGIES AND XFORMS... READ THIS NEW POST AND ALL FOLLOWING POSTS...

Ubuntu Linux

2006-08-20T22:20:00.000+02:00

Last week when I was driving home, I decided I should switch from Windows XP to Linux, Ubuntu Linux to be precise. Don't get me wrong, I don't mean installing it on an old machine or using a dual boot installation together with Windows. I mean switching actually for real!
Why? My computer was ready for a complete cleanup and reinstallation anyway. Windows Explorer kept crashing, leftovers from deinstalled programs messed up the system. It didn't feel right like it used to. It occurred to me that I had no excuse to keep using Windows anymore. Since I left OS/2 nine years ago, I always had some special purpose Windows-only software to keep me from switching to Linux. But I realized that there is nothing left. And if I get in real trouble, there is always my company laptop to help me out.

Ten years ago, Linux had a bad reputation when it comes to usability. This reputation is changing. More and more people are saying Linux isn't that hard to learn. I tried Linux on old machines several times. I never found anything easy about it.
Also, I have always been a little bit annoyed by Linux evangelist's claims of Linux being a better system. For them perhaps it is a better system. They can't imagine how Windows is so popular and so many people still chose MSIE over Firefox. With the best intentions they are improving their software usability targeting a broader audience. But with their poor marketing skills I think they are unlikely to reach them.

The recent promotion of Ubuntu Linux feels a little bit different though. Their slogan “Linux for human beings” tells us that usability is their primary focus, not just an afterthought. Are they succeeding?

In some ways they are, in some ways they're not. I found that there are roughly two kinds of software components in Linux. One kind is the obvious components everybody needs. These are surprisingly easy to install and use as long as you stick with the main stream functionalities. The other kind is the special features and specific user preferences they forgot about. These are hard to find, need editing of difficult text files or require manual installation of additional components. For example, installing OpenOffice and Firefox is easy, but getting the Java plugin to run in Firefox or getting the RSS reader to launch on system startup requires a lot of searching and reading (or they hid the easy functionality really well).
And there is nothing in between. Either it is very easy or it is very hard.

For me as a technical guy, it all feels like a large puzzle where you can put the pieces in the right place and still decide what the resulting picture should look like. For me, Linux brings the fun back in using my computer at home. I never used my computer as much as last week.

For a broader audience, I'm not so sure. They're doing a good job but they have a long way to go.

The advantage of community software is that they welcome you to help improve the system yourself. I'll think about this and see what I can do. As for now, I still have a lot to learn about the system.

Mozilla XForms headache

2006-08-17T12:46:00.000+02:00

I upgraded the Mozilla XForms plugin to the latest version (0.6) and discovered it doesn't like my tutorial example forms anymore. A list of error messages is not the worst result. The example from part 4 crashes Firefox. In Linux the process is killed instantly. In Windows you get the chance to complain about this to Microsoft. Of course you should always send an error report to Microsoft, even if you know the software is not theirs in any way. Otherwise those guys would get bored and eventually they would lose their job cutting the costs of Windows and increasing Microsoft profits. Don't think Windows would get any cheaper. It won't.

I "fixed" part 4, meaning it doesn't crash or show error messages. Most of it it was my own fault. The ugly work arounds and strange behaviors were causing the errors. The translated help messages under the F1 keys don't work anymore for now. The original XForm can still be found in the ZIP file, which does run on the 0.5 version of the plugin. Be careful with that!

Again I think it's wise to wait for a more stable XForms plugin for a while. Error messages is one thing, but completely crashing the browser and all other open tabs, is something I cannot appreciate as a result of reading broken data.

XForms tutorial part 4: Declarative Binding Logic

2006-05-22T23:05:00.002+02:00

THIS POST IS SUPERSEDED BY A NEW SERIES OF ARTICLES ON XML TECHNOLOGIES AND XFORMS... READ MORE IN PART 5 OF THE XFORMS TUTORIAL...

Again, I found the limitations of using beta software for demonstrations in this tutorial. This time however, I decided not to wait for a more stable version of the Mozilla XForms plugin. Instead, I accept a little unstable behavior and a few bugs and promise to fix example code later when needed. The example in part 3 stopped working in the latest version of the XForms plugin. This is fixed when the final plugin is released. For now, I hope you'll understand the examples with a few shortcomings.

In part 3 you may have noticed this tutorial will never make it in the "XYZ for Dummies" series. I prefer to dive straight into the advanced concepts. Part 3 introduced grouping and repeating elements. By now, you should have a basic feeling of the XML oriented nature of XForms and the dynamic response of the user interface to changes in the source document.

In this part, part 4, dynamic behavior of XForms is pushed to the limits. This post demonstrates the use of the xf:bind element for declarative specification of business logic. We start using CSS style sheets to specify layout. Data sources are stored in separate XML files. Both the source document and the interface become bilingual. Resource bundles are used for translation, separating model and content and preventing duplication. Bindings are used for hiding irrelevant elements and constraining input values.

The more I work with XForms, the more I realize how complex they really are. Considering the fact that XForms are the replacement for classic HTML forms in XHTML 2, I have my doubts that XHTML 2 will ever replace HTML 4. I am curious whether companies like Macromedia will be able to offer easy and user friendly XForms editors in products like Dreamweaver. Nevertheless, I am convinced that software developers and professional web designers will appreciate the advantages of XHTML 2 and push XForms in major web sites just like AJAX is pushed right now. After all, AJAX is no easier than XForms but it is widely adopted to enhance rich user interfaces in many web sites.

The Eclipse Apogee XForms implementation

Another promising development is the adoption of XForms outside web browsers. I found a proposal for an XForms implementation in Eclipse, called Apogee. Eclipse is an IDE (integrated development environment) for Java developers. There is also a stripped version of Eclipse which serves as a platform for building Rich Client applications in Java using native GUI controls while remaining reasonably platform independent. The Eclipse GUI framework is called SWT. The Apogee project proposal intends to extend Eclipse RCP (rich client platform) with components like a calendar, rich text editors (Office, XHTML) and XML support (XSD, SOAP, XForms).

If this proposal is accepted by the Eclipse project, this means a great step forward in the development of client side business applications in Java. XForms provide major productivity improvements in the development of form based programs. An application providing both a rich client interface and a web based interface could use the same XForms document for both the web and the client. XML oriented interfaces are no longer limited to web based implementations. This could mean the end of one-on-one binding of GUI elements to database fields.

The current alpha implementation still lacks some key XForms concepts but looks promising. I hope the developers don't underestimate the effort required for a full and stable XForms implementation. But judging the quality of current official Eclipse subprojects, Apogee might become one of the best implementations available. If you're interested, take a look at the project page. For testing the first milestone, you need to connect to the Subversion code repository and compile the plugins yourself. Some Java experience is required to do this.

But enough with the introductions. Let's get to the real content...

Introduction to part 4

My employer wants me to keep my resume up to date in both Dutch and English. This means a lot of overhead, because every change should be processed in two documents. A large part of the documents are copies. The header texts are different. A few fields like function title and description are different. But many fields are the same in both languages. My name, birth date, company names and start and end dates are identical. Wouldn't it be nice if I could edit a single document to change my resume in both languages?

That's exactly what we're going to do in part 4 of the XForms tutorial. Header texts and list values are externalized to resource bundles. A few fields are duplicated so they can be filled out in both languages. Source documents are extracted to separate XML documents. Style and layout are specified in an CSS stylesheet. And there is a new button to switch between English and Dutch without using any server side code.

The example in part 3 had an annoying shortcoming. If you deleted all work experiences, the document was unable to create a new work experience. Part 4 demonstrates how XForms bindings can be used to work around this problem.

The example is available online under the following URL. If you want to experiment with the code and modify it, you can download a ZIP file containing all files required by the XForms document. In part 3, the complete example was contained in a single file. From now on, the examples are separated over a set of files. To run the examples, you require Mozilla 1.5 and the latest version of the Mozilla XForms plugin. The examples are not tested in other XForms implementations.

Specifying declarative binding logic

Classic programming paradigms require software developers to implement algorithms telling the computer exactly what to do. As applications grow, developers should find clever ways to support additional features without duplicating code. Code duplication harms maintainability of the software. The clever ways around duplication often harm readability of the code.

In modern software development, developers no longer specify *what* to do or *how* to do it. Instead, they specify a model of their problem and expect their software to interpret the model automatically and provide answers as they are queried. The *what* and *how* are preconfigured under the hood and may be replaced by an alternative implementation without affecting the model specification. An early effort in this direction was PROLOG. PROLOG allows users to input a set of logical constraints. The constraints are supposed to represent a simplified model of reality. The user can ask questions to PROLOG which are automatically answered given the constraints. Because PROLOG is designed to solve logical problems, it is easily capable of providing the answer to a logical quiz from a news paper. As the complexity of the logical model grows, there is a risk of run times for solving problems. After all, the user can only influence the model specification, not the solving algorithm.

After PROLOG, many other products were developed using declarative model specifications for both answering questions or describing the behavior of user interfaces. Most of these products are targeted at a very specific problem domain. Limiting the problem domain allows more adequate solutions for those specific problems.

XForms is an example of such a declarative solution. The domain is XML based Forms. It is very powerful as long as you are developing a form and as long as it is based on XML. As soon as you want to use the technique for any other purpose, it is nearly impossible to get a working result.

The declarative nature of XForms is recognized when you compare them to classical HTML forms. If you want to add any business logic to a classic HTML form, you are forced to implement tiny procedures in JavaScript. Where JavaScript code is error prone and differs between browsers, XForm models will be portable as XForms implementations mature.

The xf:bind tag

The xf:bind tag is one of the most versatile elements available in XForms. It is used for specifying almost any business logic in a form. A rough list of capabilities is:

item type specification
required element or attribute specification
disabled element or attribute specification
dynamic calculation of values
specification of relevance of an element or attribute
constraining values
binding nodes or nodesets to IDs

What makes the xf:bind tag even more powerful, is that each of these functionalities is not limited to a single node, but may involve a set of elements. These nodesets are not limited to sets of siblings but may consist of elements anywhere in the source document. XPath queries are used for referencing nodes or node sets. When your experience with XPath grows, you'll find it to be a very powerful way for accessing nodes even in the most complex XML structures.

As any other XForms element, xf:bind dynamically responds to changes in the source documents. In beta implementations of XForms you might find some quirks and bugs in this behavior, but I'm convinced these will be fixed in the final version.

Binding item types

When you're using xf:input elements referring to source document elements, you'll see text input fields in the resulting documents. For specific elements like a birth date, you would expect to use something like an xf:dateInput or an xf:input type="date". In XForms this is a little different. In your xf:model you specify an xf:bind ref="..." type="xs:date" and the input field automatically changes to a date chooser in the resulting document.

By binding elements to XSD item types, XForms decides which view is required for the input element. If the source document element is referenced by more than one input, the item type needs to be specified only once. This prevents duplication and therefore improves maintainability which is important in software development.

Talking about duplication, I was a little bit disappointed that XForms required me to specify the item type in an xf:bind tag. After all, I took the effort to provide a complete XSD XML Schema file for all source documents. XForms seems to offer additional tags to explicitly refer to such XSD files. In the current implementation I was unable to bind an item type without using the xf:bind tag though.

Required elements

In a similar way, XForms allows us to reference a set of elements and specify a required="true()" attribute. As a result, the form cannot be submitted to the server if no value is present for those fields. In the CSS style sheet, I can change the looks of required elements. In the example, these fields have a yellow background and a * postfix behind their labels.

More interesting is to replace the hardcoded "true()" value by a dynamic expression. In the example, the maidenname field is only required when the gender is set to female. Implementing such logic in JavaScript based HTML forms takes a lot more effort and is more error prone.

Disabled/read-only elements

Disallowing user input works the same way. In the example, a user specifying he is male, is unable to input a maidenname. This makes little sense, because the maidenname could have been hided just as well. This example is just a demonstration. A more practical purpose is a field providing the output of a calculation. No input is necessary, because the value results from earlier input.

Calculations

A powerful feature is the ability to calculate values or sets of values. Its purpose is evident in a tax form. But in any other form it is very useful behind the screens. In the example document, we introduce a helper source document containing the outcomes of calculations. The outcomes are used as shorthand notations for long repetitive queries elsewhere in the document.

The example also introduces a shorthand notation without using the helper document. Static texts like headers, labels and hints are provided in two languages in resource bundles. Referring to the text value in the current language involves specifying the current language in each query. Instead of copying these references, the actual values in the current language are copied to their parent elements and resources are referenced assuming they are in the current language.

Source data in multiple languages:

Relevance

Specifying whether an element is relevant or not allows you to hide elements and their respective input elements as needed. In the example, the end date of a work experience is hidden when the current employer is set to "yes".

More importantly, the relevance attribute solves our problem in part 3. The problem was that the form was unable to create a new work experience instance when no template instance was available. Going from at least one experience to two or more was possible. But from zero to one wasn't. In the updated example, there is always an empty template experience present. The relevance binding is used to hide the template instance on screen. When no other experience than the template is available, the screen is empty and shows a button to create the "first" experience, which actually is the second.

When we're going to submit data to the server in a next episode, it is good to keep in mind that XForms automatically strips the irrelevant nodes from the document sent to the server. This means that when you're reloading the form in the client, the template node should be restored server side, or the relevance property should temporarily be disabled during document submission.

Constraining values

In addition to typing and visibility rules, XForms also allows us to specify validity rules over a set of fields. In the demonstration form for example, in a work experience the end date cannot be earlier than the start date. How XForms enforces the constraining rules is explained in the sections on styling and error messages.

Binding nodes to IDs

In software engineering, it is good practice to extract arbitrary numbers and texts from your source code and put them in variables in a central place so they can easily be altered without affecting specific source code. This improves maintainability and prevents duplication. In many cases it takes little effort to make the texts available to users or system administrators in order to customize software without rebuilding the source. I like to refer to these as "resource bundles". When software is distributed in more than one language, you can't do without resource bundles. The example resume form in part 4 demonstrates how resource bundles can be used to dynamically switch from English to Dutch without server interaction.

A naive implementation of resource bundles in XForms would require an XPath query in any label referring to a specific resource. To improve readability however, in our example each label refers to a unique ID. In the model specification in the header of the document, each ID is bound to the corresponding resource element. It is assumed that the referred resource element has already been translated to the current language because of the earlier calculation. The order in which the XForms implementation processes the model is essential for this to work. It turns out that a forced rebuild action is required when the trigger for switching languages is invoked.

Most IDs refer to a singleton resource element. Some IDs refer to a list of elements filling a selection drop down box. A single list element resource can be part of one or more selection lists. In the example model, this depends on the availability of a listreference tag in the resource list. The values "yes" and "no" are shared by two selection boxes, while the value "unknown" is available in only one of them.

I do not believe there is a common standard for working with resources - both singleton and lists - in XForms. What you see here, is a result of custom modeling with an additional source document and clever XPath queries. Feel free to copy the suggested pattern in your own solutions. Any comments on improving the technique are welcome.

Part of the resource bundle source document; keep in mind that the value elements in the current language are copied to their parent resource elements by the earlier calculation binding:

The ID binding declarations in the model:

References to both singleton IDs and lists of resource bundles:

Styling XForms using CSS

In part 3, classic XHTML tables were used for positioning elements. In part 4, the table elements seem to have disappeared. The result on screen still looks like a table though. The trick here is that the style for each input type is declared in the CSS style sheet. The style sheet states that some elements should be represented as a table row and others as a table cell. Additional properties like cell width and background color are provided as well.

At this point, it took a lot of trial and error to get a decent looking form on the screen. I encourage you to take a look at the style sheet and experiment with your own forms. For now, I'll skip the explanation because I miss a complete understanding to write an authoritative explanation pointing out exactly what to do in order to achieve exactly what you want.

I ran into some strange behavior that seemed unpredictable and random to me. It might very well be intended and documented behavior but until the final version of the XForms plugin is released, I blame it on the immaturity. Some of the remaining XHTML paragraph tags you find in the example are quick and dirty workarounds for this behavior.

I'll get back on this as soon as I can. With the xf:bind tag being the main subject in this post, I think the demonstration still shows enough to release it as it is right now.

Error messages

With all the constraints and type definitions the next logical question is what the end user sees when his input violates the rules. If the example had a submit button for sending data to the server, violations should prevent submission with user friendly warning messages. This example does not have a submit button yet. Binding violations result in red colored labels in front of the input fields. For now, it is up to the end user to find out what's wrong by himself.

XForms has an xf:alert tag to provide user friendly messages on binding rules violations. An alert message pops up after changing an input value or before submitting the form to the server. Unfortunately, the alerts also popup in the example when switching languages from English to Dutch. Even worse, Firefox crashes after closing the message boxes. All I can do is send an error report to Microsoft. It's not like Microsoft can help this. But it is still good practice to submit the error reports anyway. This keeps a few of their employees busy and that's what they created the reporting feature for, isn't it?

Hints and help messages

For now, I have left the alert messages out of the example. I did include an xf:hint and an xf:help tag which work the exact same way. I added a small icon to the fields featuring hint- and help messages to indicate to the end user that there is more information there. Specifically, I used the hint and help messages on the fields bound to a bilingual element in the source document. The hint provides a tooltip telling the user to press F1 to see a quick preview of the translation of the targeted field. The help tag is bound to the value in the alternate language and is meant to improve productivity. When you're making your resume, you don't need to switch languages all the time to see what you typed in the other language. A more sophisticated solution is required when the example is extended to support more than two input languages.

Conclusion

With the new binding logic, the example is starting to look like a production quality solution to the challenge provided in part 1 of the tutorial. The main ingredients missing right now are the ability to save data, a lot more detail and a preview of the resulting resume document.

Part 5 of the tutorial will focus on server interaction. It becomes even more interesting when you can store your resume server side and use server side XSLTs to transform the XML document to an actual resume in (X)HTML, PDF or Office document.

If that isn't enough, we can add additional input fields bound to the resource bundles source documents and save these as well. With some extra effort, I think we should be possible to have the XSLTs read both the resume document and the resource bundles document as input for the generated output document. This way the end user can take complete control of both his input in the resume and the header- and label values.

The main open issue is how I am going to make a working live demonstration available. I think my home ADSL line without guaranteed uptime will have to do here. I am afraid this will be available for a limited period only.

One thing is certain though: some Java Servlet programming experience is required! As it turns out, XForms is advanced material. Not a single tutorial available online is going to change that, I've noticed.

Stay tuned!

The example is available online under the following URL. If you want to experiment with the code and modify it, you can download a ZIP file containing all files required by the XForms document.

Tags: XForms | XPath | XML | MVC

THIS POST IS SUPERSEDED BY A NEW SERIES OF ARTICLES ON XML TECHNOLOGIES AND XFORMS... READ MORE IN PART 5 OF THE XFORMS TUTORIAL...

XForms tutorial part 3: Repeating Elements

2006-04-16T23:00:00.003+02:00

THIS POST IS SUPERSEDED BY A NEW SERIES OF ARTICLES ON XML TECHNOLOGIES AND XFORMS... READ MORE IN PART 5 OF THE XFORMS TUTORIAL...

Introduction

After a long break, it's time to continue writing the XForms tutorial. The latest preview version of the Mozilla XForms plugin supports the features I planned for part 3. If you want to try the examples presented in the tutorial, please download Firefox 1.5 or later and the latest implementation of the XForms plugin. After installing the plugin, restart firefox and open the example. If you're getting serious with XForms you might also want to try the extension XForms Buddy.

Every episode of the tutorial introduces an XForms feature that clearly demonstrates how XForms sets itself apart from competitive technology. Most importantly it demonstrates the advantages over classic HTML forms. Each feature is an elegant declarative XML aware solution to a common problem in form implementations. Where competitive technologies depend on error-prone client side scripts and workarounds with server interaction, XForms provides a native language for dealing with semi-structured XML documents, multiple name spaces, XML Schema definitions, logical dependencies, advanced input elements and type checking and dynamic updates in the content model.

To stress the importance of the XForms technology, please keep in mind that in XHTML 2.0, classic HTML forms are replaced by XForms. However, this is no guarantee for the actual adoption of XForms in practice. This will depend on standards compliance of future browser versions. It also depends on the web development community. XForms has a steep learning curve. Not only does it require an understanding of the declarative syntax of XForms itself; it also requires a thorough understanding of XML standards. Despite the wide adoption of simple XML notations used for data transfer, relatively few people fully understand advanced XML concepts like namespaces, XML schemas, XSL transformations and the semi-structured nature of an XML document. I'll try to cover these topics as we go.

In this tutorial the code example presented each episode is based on the example from the preceding episode. Every episode is a step towards a full example solution to the problem presented in part 1. The complexity of the examples grows every episode. Complex examples are not suitable as reference material. This is a choice I made when I started the tutorial. Reference material for XForms is widely available on other web sites. Proper tutorials are harder to find. This tutorial expects the reader to start with part 1 and follow the linear step by step plan.

Summary of part 2

The last episode, part 2, was a Hello World introduction to XForms. The key feature demonstrated in the example code, is the dynamic nature of an XForms document. When the user changes input elements, all elements in the resulting view are automatically updated to represent the new data from the content document. The document source shows absolutely no script or action definition that invokes this process. The dynamic changes are a result of the XForms architecture. This architecture separates the data source document from declarative business logic specifications and a dynamic view of the data. The view is a result of the view definition, the logical constraints and change events on the source data document.

One-to-many relations

This part of the tutorial introduces the possibility to implement repeating subforms as an interface to parent-child-relations in XML documents. Data in classic HTML forms is one-dimensional. There is one unique identifier for each input field. A typical resume contains more than one work experience. Classic HTML forms would require manual duplication of input elements or server side interaction for each new work experience. Some classic database interfaces provide parent child views for similar data structures. The interface has a short list of child elements and one form to view or edit a specific child when it is selected. This interface is inspired on the Entity-Relationship-database structure with multiple tables. The main disadvantage of this approach is that it does not feel like you're editing a document. It feels like you're manipulating a database. A proper interface should hide the backend model and show an interface that represents the model the way it is the most familiar to the end user.

Reporting software usually does support repeating elements. The main limitation of reporting software is that it is output-only. There is no way to get changes in the output document back into the source document. Adobe XFA technology is a nice try. The number of repeating subforms is dynamic in the PDF form at render-time. However, once the form is visible on screen, the number of subforms is fixed and the user is limited to changing values within the subform. Adding or removing an instance of the subform is not possible because of the static nature of the PDF viewer. A common solution to this limitation is to provide a fixed and limited number of subform instances. The user can leave subforms empty when they're not needed. There is no way for the user to input more instances of the subform than the arbitrary number specified by the form developer. For an experienced employee writing a resume with a large number of work experiences, this limitation probably has too many undesirable consequences.

XForms allows dynamic instantiation and removal of repeating XML elements. The form developer can provide buttons to insert or delete additional elements or change the order of the elements available. The dynamic form document grows and shrinks in response to user input. Data eventually is sent to the server in XML format. Where the classic HTML form submission data structure is limited to one-dimensional URL-encoded key-value pairs, the XML format allows advanced semi-structured data structures with few limitations but strong expressive power.

Overview of part 3

Enough with the introduction texts. You want real content! You want examples! I have examples! But before I present them to you, please consider reading just a few more words explaining a general concept before we dive into the deep. I think it is important that you fully understand how to properly identify your own custom XML structures within a document that mixes them with other XML structures, both open standards and additional custom document types. To do this, I explain XML namespaces in my own words. After that I switch to actual XForms concepts. I introduce a way of grouping subsets of form elements within the document and abbreviating XPath identifiers within the group. Then I'll demonstrate similar functionality to create repeating groups as an interface to repeating data structures. Finally I will introduce the basics for triggering and handling events so the end user can manipulate the document structure.

Namespace declarations

Some of you are already familiar with XML namespaces. Many of you have seen them in several XML formats. You may have copied them into your own documents to get them to work. But I assume relatively few of you are experienced in inventing your own namespaces for custom document types and their instance documents. Creating custom namespaces and document types is essential in every part of this tutorial.

Please note that namespaces are only used to identify XML elements with a unique String. Namespaces do not provide descriptions of the format. Techniques describing document types are introduced in later parts of the tutorial. Document Type Definitions (DTD) and XML Schema Definitions (XSD) can be used for this purpose. This tutorial will focus on XSDs.

For simplicity, the example in part 2 omitted a namespace declaration for the instance data. The resume XML had no prefix and no namespace definition in the root element. I assumed you were familiar with basic XML notation and you would have no problem reading the namespaces used by mixing XHTML elements and XForms elements. But what about specifying namespaces for your own data?

In my opinion, namespaces are the actual "X" in "XML" (eXtensible markup language). XML documents are extensible because they allow us to mix one XML definition within another XML definition. The example in part 2 shows this concept by mixing XHTML (using the html: prefix) with XForms (the xf: prefix). The use of XForms is not limited to XHTML documents. There are also XForms implementations for SVG models (Scalable Vector Graphics) or Office documents (OpenOffice 2.0). XForms is designed to be modular. Otherwise XForms could have been integrated within XHTML and use the same prefix. Modularity improves the reusability of document type definitions. Reusability is key to set widely accepted standards and push future improvements for both the standard itself and related standards.

Separating XHTML and XForms elements involves more than only adding html: and xml: prefixes. In fact, these prefixes are short-hand-notations for a less common XML notation:

(please replace the square brackets [tag][/tag] by their proper XML equivalents...)

[{http://www.w3.org/1999/xhtml}body]
[{http://www.w3.org/1999/xhtml}p]
Paragraph text
[/{http://www.w3.org/1999/xhtml}p]
[/{http://www.w3.org/1999/xhtml}body]

This can be rewritten as:

[html:body xmlns:html="http://www.w3.org/1999/xhtml"]
[html:p]
Paragraph text
[/html:p]
[/html:body]

This notation improves readability, it prevents duplication and it is less error prone. The prefix text "html:" can be replaced by any other text as desired by the editor of the document. As long as the "xmlns:{prefixtext}" attribute specifies the same and right Uniform Resource Name (URN), the prefix may differ from document to document. For example, some XForm documents use xf:, others use xforms: or any other text. Nevertheless, I do recommend to use readable and descriptive prefixes. Any other randomly chosen prefix, like a:, b:, c:, would confuse human readers of the document. Human readability was one of the key design goals of the XML format. This design goal is best recognized in the redundancies in closing element tags. For computer algorithms something like [/] would be sufficient as a closing tag in a document with correct nesting. Using [/xmltag] and text indention is only meaningful to human readers. A little research on the internet teaches us that this redundancy is a subject of controversy. Some fanatics propose an alternative technique named S-Expressions (or Symbolic Expressions) to save disk space and band width.

Most XML documents declare namespaces in the root element. But Namespace declarations are allowed in any element in the document. The choice of the element used for the namespace declaration decides the scope of the prefix in the document. A declaration in the root element allows you to use the prefix anywhere in the document. Using any other element limits the scope of the prefix to descending elements enclosed in the declaring element. Please be careful this does not lead to duplications of the namespace declaration. A single declaration in a document is best for readability and maintainability. Nevertheless, there are some applications where it is very common to limit the scope of prefix declarations. For example the body of a SOAP message declares the prefix for the body content within the message, not in the SOAP element root.

When an XML structure is not using prefixes, it does not necessarily mean that no URN is specified. The example can also be rewritten as:

[body xmlns="http://www.w3.org/1999/xhtml"]
[p]
Paragraph text
[/p]
[/body]

This notation is recommended for documents using a single namespace. Overhead is avoided and readability improves when you don't need to distinguish namespaces within one document. However, in documents mixing multiple namespaces, I recommend against using the default namespace without prefix. Descriptive prefixes for every namespace prevent false assumptions implicitly made by human readers and editors. To assure the reader that he's reading the document correctly; tags, id-attributes and their references should always be as explicit and descriptive as possible. Choosing id-attributes is covered in a few moments when repeating elements are discussed.

At this point, you know how to handle the prefixes. But what about the URN text these prefixes are referring to? In theory, a Uniform Resource Name could be any text as long as the name is unique for your document type. In practice, some common patterns can be recognized in URNs available in public. For open standards it is common to use URL-notations. Using an URL ensures the uniqueness of the name it provides a descriptive definition of the standard. The host name indicates the source organization of the standard and the path specifies the name, date and or version. In some cases, the URL can be retrieved in a web browser to find additional information. For simple documents not trying to set worldwide standards a common URN notation is "urn:any-name". The examples in this tutorial use this notation for our own document types to separate them from real standards. Any urn:namespace document type is invented by the author. URL namespaces are W3C standards.

In our example document, the source data instance is rewritten as follows:

Grouping elements

In part 2 of the tutorial, references to specific elements in the source document use XPath queries specifying the full path from the data instance documents root to the referenced element. As the document size and depth grows, the duplication in the XForms document increases. For maintainability it is desirable to specify an XPath reference relative to a parent element instead of copying the full path to the root everywhere. An XForms xf:group element can specify a sub-element as the root within a limited scope. Within the group, additional groups can be recursively defined gradually covering every level in the document tree. This results in a more compact XForms document that can easily be adjusted to drastic changes in the document type definition.

In the next part of the tutorial, additional features of xf:group elements are demonstrated. When a group is bound to a specific element, the XForms implementation dynamically shows or hides the group depending on business logic regarding the element. Visibility of a group or an element depends on whether the element is defined as "relevant" or not. This relevance can be specified by declarative rules and conditions that should be matched by the source data document.

In this code example, the xf:group element works well for selecting the section containing personal details and referencing the input elements within the section. But for the work experiences, we would like to repeat a group element for each existing or new experience instance. For this purpose, XForms provides another more specific solution similar to xf:group but targeted at specific functional requirements for repeating element groups.

Repeating elements

To me, repeating elements is the key feature of XForms explained in this part of the tutorial. In earlier implementations of the Mozilla XForms plugin I couldn't get it to work and this was the reason for me to take a break writing the tutorial. Other tutorials on the internet tend to omit this feature or save it for the end as an advanced concept. I like to dive right into the advanced concepts because of the importance of the feature.

Repeating elements are less complicated than they seem. They allow very powerful and user friendly interfaces to be developed with little time and effort. There is no interaction with the server and no client side JavaScripts.

Only Microsoft InfoPath provides similar functionality but that is no W3C standard. InfoPath does not run within an internet browser. It is not part of the XHTML 2 standard which is planned to be the future of the world wide web. InfoPath depends on client side scripting and misses declarative logic handling common problems in form documents. But in defense of InfoPath, it does offer a far more user friendly interface to the form developer. Using wizards, forms can be developed with little technical knowledge in a very short time. The lack of web browser integration is compensated by MS Office integration.

The definition of the xf:repeat element is similar to the xf:group element. Instead of referring to a single element, a nodeset is specified. It may seem as if a single work experience is referenced with "cv:experience". But keep in mind that "cv:experience" XPath query referenced from the enclosing group. In XPath, when more than one cv:experience is available, the query "cv:experience" refers to the complete set of experiences. If you wanted to reference a single experience in the nodeset, then the expression would be something like "cv:experience[cv:company = 'First company']".

In addition to the nodeset, an id-attribute is specified. The id allows you to refer to the set of repeating elements from another part of the XForms document. In theory, the developer is free to choose any id text. Many other XForms references and tutorials on the internet use random id texts like "id0" or "experience" without thinking ahead. I recommend spending a little extra time and effort when you're defining ids. Experienced software developers understand the importance of using descriptive and readable id texts. In the example, the id is "repeat.experience". Earlier in the document, instance data defines its id as "instance.resume". The advantage is that XPath references elsewhere in the document explicitly describe what kind of element they are referring to. Accidental duplication of ids is avoided. You're less likely to refer to the wrong element. Assumptions about child elements are explicitly expressed in the XPath query.

At runtime, both XForms interface elements and XHTML structures within the repeat-tags are copied for each cv:experience in the source document. References to specific elements in each work experience are relative to the enclosing experience element. This is similar to the group element notation. In the example source data a single work experience is provided. To start the actual repeating of elements, an additional experience should be inserted.

Triggers and events

To add and remove experiences in the document, we need two things: a button the user can click to trigger an event and a piece of code that responds to the event and execute the desired action.

The button is defined as an xf:trigger with a label for the text on the screen. The trigger by itself is no different than any other XForms input element. It responds to the specified business logic and triggers events describing user interaction.

The actual code is contained in an xf:action element. The action element has an attribute "ev:event" to specify which kind of event it should respond to. In the example, the action is defined within the trigger element. The XForms specification allows you to define an xf:action outside the trigger element, for example as a child of the XForms root element. As you might expect, there are consequences. The position of the action defines its scope. The insert action is bound to DOMActivate events, but it only executes when the Insert-button is pressed. An action defined outside the trigger and bound to DOMActivate events, responds to both the Insert and the Delete button. For some applications and maybe other event types, this may be desirable logic.

The xf:action element container is defined in the XForms specification. The event itself however is not specific to XForms. XML Events are described and specified in the W3C XML Event specification. This is yet another example of the modular design of XML, XHTML2 and XForms. XForms automatically inherits all the features specified in the XML Events standard. For example, an interesting feature that comes with XML Events is "event bubbling". Event bubbling describes the way an event is dispatched in the Document Object Model (DOM). Dispatching starts with the element that triggered the event. For example, the insert button. In our example, an event is bound to this element, so the action is invoked. When the action is ready, event dispatching continues. The parent element receives a notification of the event. In our example, the parent is not bound to an event. As a result, the event "bubbles" up to the parent of this parent element. This continues recursively until another bound event is encountered or until the document root is reached. With this behavior in mind, it is easier to understand how the scope of an xf:action depends on its location in the XForms document.

For the implementation of the action a set of commands is defined in the XForms specification. No scripting is needed to implement common action handlers in an XForms document. However, the instruction set provided by XForms is extremely limited. It consists of "xf:insert", "xf: copy", "xf:delete" and "xf:setindex". When you're implementing actions, you should follow the KISS principle (Keeping It Stupidly Simple). When you feel the need for implementing more complex actions, you should reconsider your requirements. Most likely there are good reasons why you shouldn't want to implement such complex actions. In general, you can increase the complexity of your form implementation by being creative with XPath queries and XForm business logic. For developers this involves a paradigm shift from procedural progamming to declarative programming. Some experience with the PROLOG philosophy turns out to be helpful. The next parts of this tutorial start introducing XForms tools for business logic declarations.

Implementing the example code, I noticed a strange thing. When an xf:action is defined within an xf:group element, I would expect the ability to use XPath references relative to the xf:group root element. However, it turns out that XPath references within an action should provide the full reference from the instance root to the specified element. This increases the amount of code and harms maintainability. I'm not sure whether this is a quirk in Mozilla or intended behavior from the XForms standard.

Another odd thing I encountered, is the behavior of the insert command. This command does not actually insert a new set of elements, but rather copies an existing set and requires me to set empty values in all the elements. As a result, a minimum of one experience is required for the form to work correctly. When the user deletes the last experience, there is no existing experience to be copied as a new instance. I didn't yet find out whether an XSD specification would solve the problem. I did find another workaround on the internet, which is discussed in part 4 of the tutorial.

Conclusion

In part 3 of the XForms tutorial I explained how XML really is an Extensible Markup Language. I described how namespaces are essential for mixing modular XML document types using both open standards and home grown content definitions. I demonstrated element groups to improve maintainability, repeating elements for advanced data structures, and triggers and actions for user interaction.

In the following part I will demonstrate how declarative business logic can be used to solve some of the shortcomings in the current example form. A hidden empty experience serves as a template for every newly inserted visible experience.

With Firefox 1.5+ and the Mozilla XForms plugin, click this link to open the example.

Tags: XForms | Namespaces | XHTML 2 | MVC

THIS POST IS SUPERSEDED BY A NEW SERIES OF ARTICLES ON XML TECHNOLOGIES AND XFORMS... READ MORE IN PART 5 OF THE XFORMS TUTORIAL...

When NOT to use AJAX?

2005-07-17T23:04:00.000+02:00

The recent AJAX hype has lead to the belief that AJAX is a Good Thing. Without saying it is a Bad Thing, I do think in some cases it is being used for inappropriate purposes. It requires some explanation to tell when it's appropriate and when it's not. The main point being made here is that AJAX is an unreliable technique. I don't mean the word "unreliable" as negative as it may sound. It just means the technique should be treated with special care.

What makes AJAX unreliable?

Two things: the fact that it is JavaScript and the fact that it is Asynchronous.

Why is JavaScript unreliable?

First of all a rough 10% of web site visitors disable JavaScript. An increasing number of RSS readers like RSS Bandit tend to disable JavaScript in their browser component for security reasons by default. This means you cannot rely on the availability of JavaScript and you should ensure your web application is still accessible without JavaScript.

If JavaScript is enabled, it's still a common weak point in web applications. There are subtle differences between JavaScript implementations through several browsers and versions. It's a tough job to test through all browsers, versions and operating systems. It is recommended to stick with the standard JavaScript functionality that has been around for a long time and not to use browser specific features.

JavaScripts are error prone by their nature. JavaScript is dynamically typed and uses weak typing. Dynamic typing means variables are not explicitly declared with a type; instead all variables are a "var" and get a more specific type on initialization. Weak typing means that once a variable is initialized, it can still change types when needed. Strong typed languages (regardless of whether they are statically or dynamically typed) generate an error message when programmers forget to perform proper conversion of variable types.

The lack of error messages is a tricky part. An essential programming philosophy is the Fail Fast principle. The basic idea of this principle is that when something goes wrong, a program does not try to continue working. It should stop and display an error message. Contrinuing working after an error is very risky because it is unpredictible what the program will do after the error. To a web site visitor it is better to display an error message with an indication whether it makes sense to try again, come back tomorrow, install new software or go to a church and pray. In consultancy this is called "managing expectations". Telling the visitor it won't work is better than letting him continue working for 30 minutes and then find out the 30 minutes were a complete waste of time.

Implementing the Fail Fast principle in JavaScript is not easy if not impossible. The error handling mechanisms in JavaScript are very poor. Most of the time the best you can get is "something went wrong". In more sophisticated languages like Python, Java and C# you can get specific exceptions with stack traces leading to the exact point of failure. This is one of the reasons why I'm very glad with Microsoft's transition from ASP Classic to ASP.Net. Compiled dotNet code is much more stable, predictible and easier to debug than VBScript based ASP Classic.

What's wrong with Asynchronous code?

Nothing really, as long as you don't forget it is asynchronous and asynchronous is unreliable by definition. The difference between synchronous code and asynchronous code is the predictibility of the server response. A synchronous request blocks until a response is received. If the response times out, you get a clear error message. As long as you don't introduce multi-threading yourself, your code is completely predictible.

In contrast, asynchronous code continues working without knowing when the reponse will be received. If your code depends on the response, you need to build in a system that checks the status of the response before it starts executing. The risk is that while developing and testing your server responds so fast that you forget the possibility of not receiving a response. Building a system that checks the status of a response makes your code complex and hard to maintain. Building such a system goes against the nature of asynchronous requests. It means you're using functionality for a purpose it wasn't meant for. Denying the nature of a technique is a Code Smell. This means you should consider doing things in a fundamentally different way.

How can I still use AJAX?

You should have a clear understanding of why you need AJAX. AJAX is not a goal, it is a method to achieve a goal. What is your goal and why is AJAX the appropriate solution? Are there alternatives? Is there a backup method to fall back on when it doesn't work?

AJAX is a client side technology. As with all client side technology, your server should be developed in a defensive way. All client side checking of form input should be rechecked server side just to be sure. Also, in the absence of client side support for AJAX technology, the server should provide functionality to fall back on so the web application keeps working.

The most popular reason for AJAX is performance. AJAX reduces the load of traffic between client and server and reduces the need for page reloading in the browser. This is good as long as the page reloading method still works. Google suggest is a good example. Google still works when the suggest-feature is not available. It is very user friendly though that the feature is there when the browser is able to process the scripts and the server responds in time.

AJAX can be very helpful when providing early feedback on a form filled out by a web visitor. For example a registration form can warn that the username chosen by the visitor already exists before the form is submitted. This saves time. As long as the server checks the username again on form submission, this is proper use of AJAX.

Inappropriate use of AJAX would be if a search engine decided all search results are loaded through AJAX and classic form submission would no longer be possible because it would be slow anyway. Inappripriate usage would be if a drop down menu in a form depended on content provided by AJAX depending on earlier input in the form. Even though this might work in 98% of the cases, it is no proper way to provide your services to your visitors.

Conclusion

More general, what this article says about AJAX, can be translated to any client side technique. You should also be careful depending on client side technology when it's Flash, a Media Player, browser versions or devices. Making assumptions about client side features has been one of the main causes for irritation by visitors of many web sites. Of course there are differences between personal sites, entertainment sites and company or government sites. Especially in the last case, special care is required.

When NOT to use AJAX? Do not use AJAX when you depend on it. Use AJAX as an additional feature to spice up your web application and serve your customer.

AJAX | JavaScript

XHTML 2 and the MVC Design Pattern

2005-07-11T01:00:00.000+02:00

The World Wide Web Consortium (W3C) has been working on a slow paradigm shift to purify HTML since the introduction of HTML 4. Before that, HTML was mainly a layout language suitable for WYSIWYG editing of web pages. In the early days of HTML, the layout options were limited. HTML was intended as a document description language, not a layout model. Users wanted better looking pages, so browser vendors added layout features to HTML. HTML 4 Strict was intended to end this practice and return HTML to the early days without layout options.

HTML Strict forces users into using CSS style sheets for layout. Because most HTML content was written with layout within the HTML it is impossible to switch to HTML Strict overnight. That's why the W3C introduced (X)HTML Transitional version to allow layout within HTML to help the transitioning process. The result of these two versions: (X)HTML Strict made it just a little bit too hard to do the actual switch and (X)HTML Transitional made it just a little bit too easy not to switch. Only recently web developers start showing interest in the Strict version of HTML and XHTML because they see the advantages when supporting mobile web sites, RSS feeds, print views, text-to-speech software and options to allow visitors to choose their own style sheet. (X)HTML Strict has a few flaws. XHTML 2 will help to improve the standard to overcome these flaws.

MVC

The idea behind separating content and layout is based on the Model-View-Controller (MVC) design pattern. A design pattern is a commonly agreed upon best practice for software developers. The concept of design patterns was initially introduced by the four authors (gang of four) of the book Design Patterns. The MVC pattern has its origins in the SmallTalk programming language. SmallTalk is one of the earliest fully object oriented programming languages and many concepts in Java and C# are inspired by SmallTalk.

The MVC pattern suggests that in software design the Model (data) should be separated from its View (layout) and Controller (actions). Separating these aspects allows software to dynamically switch one of these modules without affecting the others. For example you could switch from a web browser view to an automated voice reading the message to visually disabled people, without affecting the data or the actions that can be performed on the data.

MVC allows designers and software engineers to work together without interfering with each other more then necessary. They only need to agree on a common definition of the data - the Model - and then the designers can work on the View and the developers on the Controller. Integration is only a matter of testing. It is an art to define proper models allowing reuse of existing code, easy maintenance and extensions for future additions. It requires both creativity and communicative skills, a good understanding of the problem area, experience with the modeling techniques and vision for future requirements. No single person should do this on his own, defining proper models is a team effort.

Drawbacks

Many developers love MVC, some hate it. The downside of MVC is that you need to know what you're doing. If you are used to WYSIWYG clicketyclicking until it works without thinking about why it works, MVC might turn out to be a disaster. For MVC you need a clear understanding of the underlying concepts of the platform you're developing for. If you are unable to define clever concepts or abuse existing concepts, the resulting application will punnish you with bugs, bad performance and a crappy interface.

An example is the overusage of MVC in the Java Swing platform. Swing is an API for creating graphical user interfaces in Java. It is nearly impossible to develop a proper user interface in an MVC environment without thorough understanding of the MVC concept. Many people think Swing is no good because they have only seen poorly written Swing applications. It takes at least a year experience for a developer to use Swing properly. There are several reasons for the popularity of Eclipse's SWT alternative. Performance and platform integration are not the only reasons. Nevertheless I love Swing and its overdesigned object model. If you know how to use it, it's a model that comes alive and starts acting on its own.

For the current version of Visual Studio, Microsoft delibarately chose not to use the MVC pattern but follow the clicketyclick-until-it-works-without-knowing-why-it-works-approach. The result is that it takes very little time and knowledge to implement an application in Visual Basic compared to implementing the same application in Java and Swing. However, when you want to reuse existing code to implement another application, it takes little time in Java and a lot of time and copy/pasting in Visual Basic. For larger applications reuse of code is fundamental for developing maintainable programs. Still, comanies with limited resources cannot afford to chose proper modeling over rapid development.

(X)HTML Strict

A common misconception is that transitioning from old HTML to HTML Strict is about replacing TABLEs with DIVs. A more accurate description is replacing meaningless tags and layout tags with meaningful and descriptive tags and defining the layout separately in CSS files. Tables provide powerful ways for positioning content. For this purpose tables are used with content that is not actually a table. In IE4 and Netscape 4 DIVs were refered to as "layers" and also meant to position content. IE4 introduced the idea of replacing tables with layers. In recent browsers almost all HTML tags have positioning capabilities similar to DIVs. DIV tags provide little meaning to content other than to separate unrelated parts of an HTML document. Therefore they shouldn't be used for other purposes than separating content.

Instead of using DIV tags, web developers should find tags that describe the nature of the content. For example many menu structures implemented as DIVs and TABLEs are actually unordered lists (UL) with list elements (LI). In combination with CSS and a little JavaScript, menus can very well be implemented in UL- and LI-tags resulting in menus providing the same functionality with less JavaScript code. The main advantage is that when you view the page in a text-oriented browser (Lynx) the page is still understandable and provides the complete functionality. You can see this by disabling the CSS file. It doesn't look pretty but if you want to, you can still find your way.

Layout oriented tags like B (bold) and FONT should be replaced with meaningful tags like STRONG and EM (emphasis) to separate the meaning of the text from its layout. TABLEs can be used for defining data that actually is meant as a table. Headers are used to structure a page and possibly allow the automatic generation of a table of content. Keep thinking how a text-to-speech engine would read the page to visually disabled people. Keep in mind it is a form of discrimination when you do not try the best way you can to enable your page for disabled people. They have the right to read your articles, buy your products and need to access government services from time to time.

Similarities

The idea of separating document structure and layout is not new. Experienced users of MS Word use headings and style profiles to markup their documents. The automatic generation of tables of content in Word completely depends on this idea. A better example is LATEX. This is text based document structuring language used for academic publishing for many many years.

XHTML 2

Web developers who are separating content from presentation are forced to use a few dirty tricks to accomplish what they want to do in the current versions of (X)HTML. For example to create a good looking menu, a designer may want to use pictures instead of text. The picture actually contains text and when viewing in a text based browser, you want to see the text. There are CSS tricks that allow the designer to provide the text in the HTML and replace the text with a picture using a style sheet.

XHTML 2 has new features to prevent the need for these tricks. In the current versions of (X)HTML an image can be provided by using an IMG tag or by using a CSS property. In XHTML 2 the SRC property of the IMG element can be applied to any XHTML element. For example it is possible to define a [li scr="image.png"]Descriptive text[/li] without the IMG tag. If the browser is unable to view the image, the descriptive text is viewed instead. Otherwise the image replaces the descriptive text. There is no need for ALT attributes anymore, a cleaner view is given when the image is unavailable, there are no unnecessary ID and CLASS attributes and there is no pollution of the CSS file. The CSS provides metadata that should be the same for every page; the menuimage in many cases actually is data that may change from page to page. XHTML 2 solves this issue and the trick is no longer needed.

Similarly it is not necessary anymore to use an anchor (A) to provide a link. The HREF-attribute can be applied to any element. For example [li href="/newpage.html"]Descriptive text [/li] generates a linked list item. The advantage is that links are no longer limited to text links - or images using the IMG tag. With the generalization of the image SRC attribute, it is a logical choice to do the same with the HREF attribute. If any element can be a link, the number of options increases tremendously. Emulations of this behavior using an ONCLICK attribute to trigger a JavaScript event are no longer needed.

By the way, the W3C chose not to use the XLink spec. I think this is a good idea because HTML editors are too used to the HREF attribute to change that. The XLink spec never became popular and received a lot of criticism. This design choice in XHTML 2 might be fatal to the XLink standard.

Modularity

In HTML 2 forms are defined using XForms. In other posts in this weblog I am writing about XForms. Right now it is hard for me to imagine that XForms is going to be the default implementation in a major standard like XHTML. In the new philosophy it is logical to separate a form from a document definition because the nature of a form is fundamentally different from the nature of a document structure. The same way frames are defined in a separate module XFrames. A frame has nothing to do with defining a documents. It is just a convenience to the user for example when viewing JavaDoc API documents.

Adoption

The main question for the success of XHTML 2 is whether it will be used by web developers or not. There are several threats that may stand in the way of successful adoption of XHTML 2. One of the main hurdles is the learning curve for XHTML Strict and CSS. An intelligent web designer will see the advantages and learn XHTML 2. However, most of the internet is filled with content written by average users, not web specialists. The average web editor uses a WYSIWYG tool to create web pages. WYSIWYG tools give the author control for what the web page will look like. Shifting to the new paradigm where layout is defined in CSS files, the content author loses control of the layout. Many users will not easily accept a decline in features and usability of their application. Solving this issue requires more than training. Perhaps brain-washing is a more accurate description.

At this moment, most HTML editors in content management systems are layout based and use B and I tags, font types and colors etc, etc. I haven't found a proper editor for document structuring yet. An editor that integrates with any possible CSS file and still provide a WYSIWYG view while defining structure instead of layout is not easy to develop. For this issue, we'll have to wait and find out. Tips from readers are welcome by the way. Please leave a comment if you know of any effort or have experience in solving this issue.

Another risk is the required skills for developing a proper web site where content and layout are separated. Document modeling is not an easy task. Large organizations can afford to hire qualified developers. But the average web site is developed by amateurs lacking the skills and experience to define an accurate document structure presented by style sheets. What we need is proper and clever tools. I'm not a fan of Dreamweaver, but I do think it is up to Macromedia to provide the tools to use XHTML 2.

Last but not least, don't forget the adoption of XHTML 2 by browser vendors. I have faith in the Mozilla organization, they'll create new versions of Firefox imlementing XHTML 2. But it is questionable whether Microsoft's Internet Explorer will support XHTML 2 in the near future. IE has a history of implementing what the users want. There's a good chance that time is standing still and we're stuck with (X)HTML Transitional. I really hope my pessimism is misplaced and I'm wrong here. Possibly the recent community-awareness of Microsoft will convince them to follow the standards, who knows?

For now, at the time of writing, the proposal for XHTML 2 is not even final yet, so let's be patient.

Your feedback?

Even though some of the above statements may sound like I'm presenting facts, everything is my opinion. I didn't do my homework and I didn't check every detail I've written. If you disagree with my opinion or especially if you can prove me wrong, please leave a comment. I'm always interested in opposing views and I know many web developers see things differently. Most interesting is not what you think, but why you think it. Also feel free to post links to your own weblog if you have content related to this subject. I don't mind shameless self-promotion as long as it leads to interesting stuff.

Thank you for reading my text! My next article will explain when not to use AJAX. Watch the RSS feed (Atom actually). If you do not use RSS news readers yet, try the open source RSS Bandit for a while. It's dotNet, looks like Office 2003 and does what it is supposed to do.

Tags: Semantic Web | XHTML | XHTML 2 | MVC

XForms part 2.5: OpenOffice

2005-07-02T22:17:00.001+02:00

THIS POST IS SUPERSEDED BY A NEW SERIES OF ARTICLES ON XML TECHNOLOGIES AND XFORMS... READ MORE IN PART 5 OF THE XFORMS TUTORIAL...

Please hang on, I did not stop writing the rest of the XForms tutorial. I'm just waiting for the XForms plugin in Mozilla to mature. In the latest stable releases I do not get the repeat-elements to work. Might be my mistake, but for now I blame the plugin. I tried the nightly build, but it does not load XForms at all. As soon as I have a working version, I'll continue.

In the mean time I did try the XForms part of OpenOffice 2.0. Maybe you read about it somewhere else, but you couldn't find the functionality in OpenOffice. The feature is hidden well. A simple File -> New -> XML Form does the trick though. OpenOffice might be one of the first major applications to offer a visual interface to XForms. Like InfoPath, the average Access user should be able to use the functionality; it's not meant for your grandmother though.

My most important observation: OpenOffice does not implement the XForms user interface; only the XForms backend. Instead it uses existing forms elements from OpenOffice and connects them to the XForms backend. At first I was a little bit disappointed about this. Especially the fact that I still miss some sort of repeat-element. However, when I think about it a bit longer, I can still be enthousiastic about the OpenOffice implementation. In the past, the XForms backend has been criticized for being too heavy for client side systems - especially browsers. The OpenOffice team chose to use the XForms backend even without the rest of the spec. to support their own forms. That says something.

Main conclusion: XForms is not about a cool new forms interface for the browser that happens to carry a backend with it... XForms is a forms modelling platform providing means to implement business logic in a declarative way and adds a cool new interface as an extra.

I can hear you thinking: shouldn't business logic be implemented server side? For XForms the same rule applies as for AJAX: it is meant for convenience to the user (ease of use and responsiveness); its working and the generated input to the server should be considered unreliable at all times. Use these techniques to offer options to the user but never depend on the availability of the options. Make sure the interface still works when the option fails.

By the way, OpenOffice 2 is worth looking at anyway. The killer feature for version 1 was the "Export to PDF" option. I loved it. Instead of buying an expensive Adobe Acrobat, using some freeware tool loaded with spyware or going through hell with "Print to PostScript" and a ps2pdf converter, I can do this with a single click for free. I switched from MS Office to OpenOffice for typing by C.V. I prefer sending this as a PDF to ensure the formatting doesn't change and to prevent red lines under words like "XForms".

Another killer feature (for geeks, not your grandmother) besides XForms is "Base" providing an Access-like interface to any database accessible through JDBC. JDBC is Java's standard for connecting to databases and exists for many years. The standard is mature and is used in major enterprise applications with large numbers of users. A product that offers this functionality can exist on its own. But with the integration with other OpenOffice applications it really comes alive. I will test some more and maybe report later.

Tags: XForms | OpenOffice

THIS POST IS SUPERSEDED BY A NEW SERIES OF ARTICLES ON XML TECHNOLOGIES AND XFORMS... READ MORE IN PART 5 OF THE XFORMS TUTORIAL...

Creating hypes

2005-06-25T11:31:00.000+02:00

Unlike W3C, some companies are pretty good at creating bubbles. The art is to draw a lot of attention to something that seems to be new and innovating but really is not! I haven't really figured out how to do this, otherwise I could have been marketing manager. But here are some ingredients: using curiosity of the public, using weblog communities to spread the word and promising lots of new cool features.

Two recent examples: "AJAX" and "RSS functionality in Longhorn".

AJAX technology has been available for years, but only recently got a lot of attention since Google shows off with cool features (auto-completion in web pages). AJAX won't be used much near Rotterdam because it shares its name with an evil soccer club. They would have made a better chance if they found an acronym for Feyenoord.

In the past week Microsoft has created a lot of hype around the new RSS capabilities of Internet Explorer 7. The IE team has renamed itself into the Browsing and RSS-team. Why? You know the little orange button in the right part of the Mozilla Firefox status bar? IE7 will have something similar. Only where Firefox supports more than one RSS feed for one web page, IE7 is limited to the first feed found on the page "because this is easier for users". They're focussing too much on the user not knowing the difference between RSS and Atom and they're forgetting some pages have multiple RSS feeds that differ in content and update frequency.

How do they manage to create hype around such things? They keep secrets, give away a few hints, announcing the date and time of the next announcement with more hints and still not giving away all details. In addition they're mixing up names. They know they shouldn't be too proud of IE7 because IE7 should have shipped at least one year ago. So they are talking about the "RSS capabilities in Longhorn". RSS capabilities in Longhorn?!? Haven't they learnt anything over the past years? For example you shouldn't mix functionality at operating system level too much with functionality at application level? If it's not a technical mistake, it could get them into new lawsuits, I think.

But never mind that, they're doing great creating a hype around RSS. I read somewhere that new versions of Media Player may be supporting podcasts in future. This could be a good thing for the client I'm working for. So let's see what happens.

Tags: Microsoft | AJAX | RSS | IE7 | Hype

The problem with W3C standards

2005-06-21T14:53:00.000+02:00

W3C standards suffer from the same problem as the proposal for the European constitution. In earlier posts I pointed out I'm in favor of the constitution, yet the French and the Dutch voted against it and there's little hope the constitution is going to make it. The problem summarizes to "Nobody reads the text".

Can we blame people for not reading the texts?
- Not really.

I can blame W3C for poor dissemination of their results though. W3C produces white papers that seem to be boring at first glance. At second glance, their introduction texts turn out to be interesting texts. But most of the time these white papers don't even get a first glance.

I say the W3C should work on their marketing skills. First of all, they should benefit from the current Weblog hype to spread their message and support the strategy they're setting out for standardizing the web. Secondly, I think many new standards are interesting enough to be reported in major news papers with a little background information and a summary of new developments.

Lacking an audience, let me ask myself a few critical questions:

Where did I get the idea nobody is reading the W3C standards?

Open 5 or 10 websites and view their HTML source. Many of them are full of nested tables, font tags, b- and i-tags. Few of them adhere to standards or even contain remote clues that someone is trying to follow a standard. Ask an average web developer about the difference between XHTML Strict and XHTML Transitional. There's a good chance he'll tell you that XHTML Strict is meant for nerds and it's practically impossible to adhere to this standard. XHTML Transitional is more reasonable and should be used for designing a web page. If you ask him what "transition" the word transitional refers to, you probably won't get an answer.

The answer is really easy though, if you're familiar with the introduction texts of the W3C white papers. The transition is about moving away from mixing document structure and document presentation in (X)HTML. And separating presentation to CSS files. Reading through weblogs I noticed some web designers think that this transition is about "replacing all tables with divs". This is wrong on so may levels I should spend a separate post explaining it.

I work on a content management system with a large number of editors and journalists. Should we require them to read through W3C standards?

The latest developments on (X)HTML are meant to make it easier. The it's-too-hard-assumption is a poor excuse not to adhere to standards. If the basic page setup is right and the tools provided by the CMS make it easy for the content editor to produce valid XHTML Strict without worrying about CSS, there should be no problem. If anyone can point out such tools, please let me know!

Web developers are not the real issue, the problem is with IE forcing web developers not to use standards.

At least Microsoft did read the text and chose not to adhere to the standards. Their excuse is backward compatibility. Microsoft is responding to popular demand. As long as the majority of web developers is not demanding standards compliance, Microsoft is not going to offer any. I'm worried about the implementation of upcoming standards - XHTML 2 and all related modules like XForms and XML Events. But it's too easy to point at Microsoft. Not long ago, in the FireFox community some were arguing against implementation of XForms. Because of the difference in goals and stakeholders, they're working on it now. But the fact that both browser manufacturers are/were reluctant to start with their implementation tells me more about the poor job W3C is doing conveying their ideas than about Microsoft - a company we know for quite a few years now.

Aren't you forgetting to mention something?

Thanks for reminding me! Another thing that isn't helping the adoption of new standards is the lack of cool new features. While I (being a software developer) am convinced of the advantages of an improved XHTML structure, to the large public there's no added value in any of this. They're playing around with the latest hypes like AJAX and new Flash versions. I think both of them are overrated, abused for the wrong purposes and poorly understood by their user community, but that's not going to stop them.

How do you convey to the large public that modularization of XHTML will increase the options for adding in new features in future without needing to replace your complete browser suite?

With a little luck, time will heal the wounds. XHTML standards are meant to last over time and it's likely we're still using them in 10 years. For now I'm a little bit sceptical. We'll find out...

Tags: W3C | standards | XHTML | CSS

InfoPath: OPML editor / RSS reader

2005-06-04T11:39:00.000+02:00

In addition to my somewhat premature posts about XForms, I think it would be good to discuss a more mature product that has been there for some time now and is related to XForms in many ways but also has fundamental differences. InfoPath is an Office product for editing XML documents using typical Office interface elements. To explore its capabilities, I'm building a form for editing OPML files and viewing the referenced RSS feeds. The end result is alpha quality, not suitable for every day use, but a nice example of the current status of InfoPath.

Introduction

As a software developer, I'm interested in InfoPath as a development tool, helping me create advanced forms interfaces in little time. However, InfoPath is not the latest Visual Studio component. It is part of Microsoft Office, so it should be easy enough for an experienced Office Manager to create fully functional forms using InfoPath.

XML was intended to be easy to read by humans. XSLT was intended to be easy enough to be used by others than software developers. As I see it, XML and XSLT are mainly the territory of software developers and more experienced web designers. Many XML concepts like wellformedness, validity, namespaces and XML schema's are too comprehensive to be grasped without a background and experience in informatics.

InfoPath took the challenge of making XML files accessible to the large public in a user friendly way. In most cases user-friendlyness goes at the cost of developer capabilities. In the rest of this post I'd like to explore both the user-friendlyness and the developer capabilities.

An OPML editor in less than 5 minutes

If you have used InfoPath before, you might be able to do this in 10 seconds I think. This is also the quickest way to convince someone who hasn't seen InfoPath before of its power. Take the following steps (my apologies for bad naming, I use the Dutch version of InfoPath, so I cannot provide the correct steps here):

Save an OPML document to My Documents - make sure it has nested outline elements, like mine.
Open InfoPath and create a new document based on an XML file and select the OPML file
From the Data View, drag an "outline" element to the form.
Done!

If your not too demanding, this could function as a basic OPML editor. It creates valid OPML files, allows you to add elements and levels and it views the full content. It requires a lot of tweaking to make it work like you would want to, but that's mostly minor details.

Finding the limitations

The quickest way to find the limitations of a software product is to use it for something it was not intended for. Without going overboard, I think I'm being reasonable if I want to be able to make a nice looking view for my OPML editor that also dynamically shows the most recent content of the RSS feeds it's refering to. I'm sure someone with inside knowledge of InfoPath could do better than I did. I'm also sure that I could have done better if I took a full week to develop this. But as a lazy person, it's much more interesting what could be done with little effort and in a few hours. Being lazy, I got to a point where I wanted to switch to server side programming to solve issues because this would be easier. This was my clue to stop trying and write this post. This is what I did:

I created a second view for the InfoPath document, so I have one "OPML editing mode" and one "OPML/RSS viewing mode". In the editing mode I tweaked some form options to prevent loads of empty lines appearing when I added new OPML elements. I made some decorative adjustments and a "Switch View" button. I'm quite pleased with this simple and straightforward editing mode. If I ever need an XML editor and I don't have time, this will be my first choice.

RSS viewing is a different story though. Again, I may very well be wrong about this, but this is what I found in a few hours with little effort. In InfoPath I can define secondary data sources. There is a single instance for each data source. The options to load more than one RSS view at the same time are limited. I can attach data elements to a secondary data source. So viewing the RSS feed basically is very easy.

This is where it went wrong: InfoPath expects data sources to follow the same XML schema and producing predictable content structures. XML feeds referenced by OPML files may be very different from each other. Some RSS feeds are simple and straightforward; others are mixed with RDF structures. These cannot be read from one secondary data source. In addition, many XML feeds are not RSS feeds but Atom feeds. Somehow my InfoPath crashed when I tried to load an Atom feed. Also many RSS feeds (those from MSN Spaces for example) contain HTML markup. This messes up the text in InfoPath.

Some preprocessing using XSLT would help a lot here. The secondary data source can only be assigned using a FileURL attribute, not by directly providing a reference to a DOM source which could be the outcome of an XSLT translation. This was the point where I was considering programming server side code. This would work well to get the application up and running, but it would not be an honest review of InfoPath anymore.

Links:

Screen shot 1: view OPML/RSS
Screen shot 2: edit OPML
The InfoPath document (use "File->Merge with XML" to load OPML data)

A few more words

In the example above I used JavaScript code to implement actions for the buttons in the form. Microsoft provides an InfoPath SDK for developers. With this SDK you can write action handlers in managed C# code and be able to use the full capabilities of the .Net platform. This increases the capabilities of InfoPath tremendously.

I have played with the SDK and I liked it. But something I was missing - again, this might be my mistake - is the ability to use InfoPath as an embedable component in my .Net applications the same way I can embed Internet Explorer in my own windows. If I were able to do this, I could develop very powerful and good looking business applications. Maybe I should write a letter to Microsoft. The getting-hardcore-Java-developers-to-switch-to-dotNet-team might be interested in my case...

In the developers vs. experienced users and Visual Studio vs. MS Office design choice, I think InfoPath is comparable to MS Access. You require some background knowledge to use it, but with proper training, you don't need to be a computer expert to get results. Access is a nice application to do complicated things very quickly. For professional use, its capabilities are limited. InfoPath fits in the exact same category. In this category, Access only recently got competition by the Open Office product Base. For InfoPath this might take a while. If you're interested, go play with InfoPath.

If you manage to do better than I did - with not too much effort - please let me know!