VTD-XML

VTD-XML
Developer(s) XimpleWare
Stable release
2.12 / Nov 19, 2015
Operating system Portable
Platform Java, C#, C and C++
Type XML parser/indexer/slicer/editor library
License GPL and Proprietary License
Website vtd-xml.sourceforge.net VTD-XML blog

Virtual Token Descriptor for eXtensible Markup Language (VTD-XML) refers to a collection of cross-platform XML processing technologies centered on a non-extractive[1][2] XML, "document-centric" parsing technique called Virtual Token Descriptor (VTD). Depending on the perspective, VTD-XML can be viewed as one of the following:

VTD-XML is developed by XimpleWare and dual-licensed under GPL and proprietary license. It is originally written in Java, but is now available in C,[14] C++ and C#.

Basic Concept

Non-Extractive, Document-Centric Parsing

Traditionally, a lexical analyzer represents tokens (the small units of indivisible character values) as discrete string objects. This approach is designated extractive parsing. In contrast, non-extractive tokenization mandates that one keeps the source text intact, and uses offsets and lengths to describe those tokens.

Virtual Token Descriptor

Virtual Token Descriptor (VTD) applies the concept of non-extractive, document-centric parsing to XML processing. A VTD record uses a 64-bit integer to encode the offset, length, token type and nesting depth of a token in an XML document. Because all VTD records are 64 bits in length, they can be stored efficiently and managed as an array.[15]

Location Cache

Location Caches (LC) build on VTD records to provide efficient random access. Organized as tables, with one table per nesting depth level, LCs contain entries modeling an XML document's element hierarchy. An LC entry is a 64-bit integer encoding a pair of 32-bit values. The upper 32 bits identify the VTD record for the corresponding element. The lower 32 bits identify that element's first child in the LC at the next lower nesting level.

Benefits

Overview

Virtually all the core benefits of VTD-XML are inherent to non-extractive, document-centric parsing which provides these characteristics:

Combining those characteristics permits thinking of XML purely as syntax (bits, bytes, offsets, lengths, fragments, namespace-compensated fragments, and document composition) instead of the serialization/deserialization of objects. This is a powerful way to think about XML/SOA applications.

Conformance

VTD-XML conforms strictly to XML 1.0 (Except the DTD part) and XML Namespace 1.0. It essentially conforms to XPath 1.0 spec (with some subtle differences in terms of underlying data model) with extension to XPath 2.0 built-in functions.

Simplicity

As Parser

When used in parsing mode, VTD-XML is a general purpose, extremely high performance[17] XML parser which compares favorably with others:

As Indexer

Because of the inherent persistence of VTD-XML, developers can write the internal representation of a parsed XML document to disk and later reload it to avoid repetitive parsing. To this end, XimpleWare has introduced VTD+XML as a binary packaging format combining VTD, LC and the XML text. It can typically be viewed in one of the following two ways:

XML Content Modifier

Because VTD-XML keeps the XML text intact without decoding, when an application intends to modify the content of XML it only needs to modify the portions most relevant to the changes. This is in stark contrast with DOM, SAX, or StAx parsing, which incur the cost of parsing and re-serialization no matter how small the changes are.

Since VTDs refer to document elements by their offsets, changes to the length of elements occurring earlier in a document require adjustments to VTDs referring to all later elements. However, those adjustments are integer additions, albeit to many integers in multiple tables, so they are quick.

XML Slicer/Splitter/Assembler

An application based on VTD-XML can also use offsets and lengths to address tokens, or element fragments. This allows XML documents to be manipulated like arrays of bytes.

XML Editor/Eraser

Used as an editor/eraser, VTD-XML can directly edit/erase the underlying byte content of the XML text, provided that the token length is wider than the intended new content. An immediate benefit of this approach is that the application can immediately reuse the original VTD and LC. In contrast, when using VTD-XML to incrementally update an XML document, an application needs to reparse the updated document before the application can process it.

An editor can be made smart enough to track the location of each token, permitting new, longer tokens to replace existing, shorter tokens by merely addressing the new token in separate memory outside that used to store the original document. Likewise, when reordering the document, element text does not need to be copied; only the LCs need to be updated. When a complete, contiguous XML document is needed, such as when saving it, the disparate parts can be reassembled into a new, contiguous document.

Other Benefits

VTD-XML also pioneers the non-blocking, stateless XPath evaluation approach.

Weaknesses

VTD-XML also exhibits a few noticeable shortcomings:

Areas of Applications

General-purpose Replacement for DOM or SAX

Because of VTD-XML's performance and memory advantages, it covers a larger portion of XML use cases than either DOM or SAX.[18]

XPath over Huge XML documents

The extended edition of VTD-XML combining with 64-bit JVM makes possible XPath-based XML processing over huge XML documents (up to 256 GB) in size.

For SOA/WS/XML Security

The combination of VTD-XML's high performance and incremental-update capability makes it essential[19][20][21] to achieve the desired level of Quality of Service for SOA/WS/XML security applications.

For SOA/WS/XML Intermediary

VTD-XML is well suited for SOA intermediary applications such as XML routers/switches/gateways, Enterprise Service Buses, and services aggregation points. All those applications perform the basic "store and forward" operations for which retaining the original XML is critical for minimizing latency. VTD-XML's incremental update capability also contributes significantly to the forwarding performance.

VTD-XML's random-access capability lends itself well to XPath-based XML routing/switching/filtering common in AJAX and SOA deployment.

Intelligent SOA/WS/XML Load-balancing and Offloading

When an XML document travels through several middle-tier SOA components, the first message stop, after finishing the inspection of the XML document, can choose to send the VTD+XML file format to the downstream components to avoid repetitive parsing, thus improving throughput.

By the same token, an intelligent SOA load balancer can choose to generate VTD+XML for incoming/outgoing SOAP messages to offload XML parsing from the application servers that receive those messages.

XML Persistence Data Store

When viewed from the perspective of native XML persistence, VTD-XML can be used as a human-readable, easy to use, general-purpose XML index. XML documents stored this way can be loaded into memory to be queried, updated, or edited without the overhead of parsing/re-serialization.

Schemaless XML Data Binding

VTD-XML's combination of high performance, low memory usage, and efficient XPath evaluation makes possible a new XML data binding approach based entirely on XPath. This approach's biggest benefit is it no longer requires XML schema, avoids needless object creation, and takes advantage of XML's inherent loose encoding.[22]

It is worth noting that data binding discussed in the article mentioned above needs to be implemented by the application: VTD-XML itself only offers accessors. In this regard VTD-XML is not a data binding solution itself (unlike JiBX, JAXB, XMLBeans), although it offers extraction functionality for data binding packages, much like other XML parsers (DOM, SAX, StAX).

Essential Classes

As of Version 2.11, the Java and C# versions of VTD-XML consist of the following classes:

The extended VTD-XML consists of the following classes:

Code Sample 

/* In this java program, we demonstrate how to use XMLModifier to incrementally
* update a simple XML purchase order.
* a particular name space. We also are going 
* to use VTDGen's parseFile to simplify programming.
*/

import com.ximpleware.*;

public class Update {
      public static void main(String argv[]) throws NavException, ModifyException, IOException{
            // open a file and read the content into a byte array
            VTDGen vg = new VTDGen();
            if (vg.parseFile("oldpo.xml", true)){
                VTDNav vn = vg.getNav();
                AutoPilot ap = new AutoPilot(vn);
                XMLModifier xm = new XMLModifier(vn);
                ap.selectXPath("/purchaseOrder/items/item[@partNum='872-AA']");

                int i = -1;
                while((i=ap.evalXPath())!=-1){
                    xm.remove();
                    xm.insertBeforeElement("<something/>\n"); 
                }
                ap.selectXPath("/purchaseOrder/items/item/USPrice[.<40]/text()");
                while((i=ap.evalXPath())!=-1){
                    xm.updateToken(i,"200");
                }
                xm.output("newpo.xml");
            }
      }
}

References

  1. ↑ Non-extractive Parsing for XML
  2. ↑ XML Processing for the Future
  3. ↑ Manipulate XML Content the Ximple Way Introduce the concept of document centric XML Processing and why it is where the future lies
  4. ↑ VTD-XML: XML Processing for the Future (Part II)
  5. ↑ Simplify XML Processing with VTD-XML
  6. ↑ Better Faster XML Processing with VTD-XML
  7. ↑ VTD-XML: XML Processing for the future (Part I)
  8. ↑ Index XML documents with VTD-XML
  9. ↑ Cut, Paste, Split and Assemble XML documents with VTD-XML
  10. ↑ XML on a chip?
  11. ↑ XML on a Chip
  12. ↑ XimpleWare's W3C binary XML workshop Position Paper
  13. ↑ Improve XPath efficiency with VTD-XML
  14. ↑ VTD-XML: a New Vision of XML by courtesy of Victor R. Volkman
  15. ↑ Virtual Token Descriptor introduction at SourceForge
  16. ↑ i-Technology Viewpoint: The Performance Woe of Binary XML @ SOA WORLD MAGAZINE
  17. ↑ VTD-XML Parsing/Navigation Performance Report
  18. ↑ http://www.devx.com/xml/Article/30484 Process Large XML documents with VTD-XML
  19. ↑ Accelerate WSS Applications with VTD-XML
  20. ↑ W3C workshop presentation on XML security
  21. ↑ http://www.w3.org/2007/xmlsec/ws/papers/06-zhang-ximpleware/
  22. ↑ Schemaless Java-XML data binding with VTD-XML
This article is issued from Wikipedia - version of the 5/15/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.