Citation
Metadata Storage for File Management Systems: Data Storage and Representation Techniques for a File Management System

Material Information

Title:
Metadata Storage for File Management Systems: Data Storage and Representation Techniques for a File Management System
Copyright Date:
2008

Subjects

Subjects / Keywords:
Database design ( jstor )
Database management systems ( jstor )
Databases ( jstor )
HTML ( jstor )
Java ( jstor )
Metadata ( jstor )
SQL ( jstor )
Web browsers ( jstor )
Web servers ( jstor )
XML ( jstor )

Record Information

Source Institution:
University of Florida
Holding Location:
University of Florida
Rights Management:
Copyright the author. Permission granted to the University of Florida to digitize, archive and distribute this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.
Embargo Date:
8/8/2002
Resource Identifier:
52188880 ( OCLC )

Downloads

This item is only available as the following downloads:


Full Text

PAGE 1

METADATA STORAGE FOR FILE MANAGEMENT SYSTEMS: DATA STORAGE AND REPRESENTATION TECHNIQUES FOR A FILE MANAGEMENT SYSTEM By BRENNAN THOMAS FOLMER A THESIS PRESENTED TO THE GR A DUATE SCHO O L OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ENGINEERING UNIVERSITY OF FLORIDA 2002

PAGE 2

Copyright 2002 by Brennan Thomas Folmer

PAGE 3

iii ACKNOWLEDGMENTS First, I would like to thank my advisor, Dr. Doug Dankel II, for his guidance throughout my thesis research. I truly appreciate the time and energy he spent guiding me through the thesis process. I would also like to thank the other members of my committee, Dr. Joachim Hammer and Dr. Abdelsalam Helal. To my wife, Lumari Folmer, I owe a special thank you. She has always been my biggest supporter and source of inspiration, and this thesis would not have been possible without her. Finally, I would like to thank the rest of my caring and supportive family for always encouraging me to strive for greater achievement, both in the classroom and outside of it.

PAGE 4

iv TABLE OF CONTENTS page ACKNO W L E D G MEN T S ................................................................................................... iii L I ST O F F I GURE S ........................................................................................................... . v i A B STRAC T .................................................................................................................... . v ii CHAPTERS 1 PROBLEM DESCRIPTION...........................................................................................1 2 L I T ERATURE SURVEY AND TECH N I C A L OVER V I E W ........................................ 6 Technol o g ies .................................................................................................................. . 6 J a v a .......................................................................................................................... . 6 Servlets .................................................................................................................... . 7 Tomcat web s e rver .................................................................................................. . 8 J a va F ilter s ............................................................................................................. . 9 X M L ...................................................................................................................... . 1 0 X S L T ..................................................................................................................... . 1 1 Other Ref e re n ce s .......................................................................................................... . 1 2 3 DATABASE DE S I G N AND USAG E .......................................................................... 1 4 Database M a n a gement S y s t em ( D B M S) O v ervi e w ..................................................... . 1 4 Database Sc h e m a .......................................................................................................... . 1 6 Database Us a g e Patt e r n ................................................................................................ . 18 Maintaining Da t a base I ntegri t y .................................................................................... . 2 0 4 DOCUMENT SU B M IS S I O N ....................................................................................... 2 1 User’s Pers p ectiv e ........................................................................................................ . 2 1 Document Processi n g ................................................................................................... . 2 3 Document Transp o r tatio n ...................................................................................... . 2 3 Data Stream Parsi n g .............................................................................................. . 2 5 Directo r y Monitori n g ............................................................................................ . 2 6 Conversion to X M L ............................................................................................... . 2 6 Metadata Stor a g e ................................................................................................... . 27 Document Man a gement S y stem Availabili t y ........................................................ . 28

PAGE 5

v 5 V I E W S OF DMS CONTENTS..................................................................................... 29 Securi t y Mo d e l ............................................................................................................. . 29 Description of Vi e w s .................................................................................................... . 3 0 B a sic V i ew ............................................................................................................ . 3 0 Detailed Vie w ........................................................................................................ . 3 2 Administrator Vie w ............................................................................................... . 3 4 Creation of View s ......................................................................................................... . 3 5 B a sic V i ew ............................................................................................................ . 3 5 Detailed Vie w ........................................................................................................ . 39 Administrator Vie w ............................................................................................... . 39 Actions ....................................................................................................................... . . . 4 2 View ...................................................................................................................... . 4 2 Edit ........................................................................................................................ . 4 2 Delet e ..................................................................................................................... . 4 4 E x tending Views .......................................................................................................... . 4 5 6 DOCUMENT V I E W IN G .............................................................................................. 48 Document View C r eati o n ............................................................................................. . 48 Modi f y i n g the Displ a y of a Documen t ......................................................................... . 5 1 7 CON C L U S I O N S A N D EXTEN S I ONS ....................................................................... 54 E x tensions .................................................................................................................... . 54 More View option s ................................................................................................ . 54 More Metada t a ...................................................................................................... . 54 Document man a gement tools ................................................................................ . 55 Diffe r e nt views of s y st e m contents ....................................................................... . 55 I m plement int e rf a ce to a bstract data b a se a ccess ................................................... . 55 Thesis Conclusion s ....................................................................................................... . 56 Metadata a c c e ssibili t y f o r the DMS ...................................................................... . 56 F l e x ible with 3r d-par t y m e tadata sc h e me s ............................................................. . 57 Handling cha n g e s to the DMS s y stem .................................................................. . 58 L I ST O F RE F ERENCES .................................................................................................. 6 0 B I OGRAP H I C A L SKE T CH ............................................................................................ 6 1

PAGE 6

vi LIST OF FIGURES Figure Page 2.1 Sample X M L D o cu m e nt. ................................................................................................ 1 1 3.1. SpoolContents Table Creation Scrip t ............................................................................. 18 3.2 Database Us a g e Patt e r n ................................................................................................... 19 4.1 Top-level Nav i gation Menu s ........................................................................................... 2 1 4.2 Upload Scr e en ............................................................................................................. . ... 2 2 4.3 Upload Confirmation Scree n ........................................................................................... 2 3 4.4 Upload F orm HT M L Definition s .................................................................................... 2 4 5.1 B a sic DMS Vi e w ............................................................................................................. 3 1 5.2 Detailed DMS Vie w ........................................................................................................ 3 3 5.3 Administrator Vie w ......................................................................................................... 3 4 5.4 View Menu Options ........................................................................................................ 36 5.5 F ilter Confi g u r ation Settin g s for the B a sic a n d Detailed View s ..................................... 37 5.6 “Ac c ess denie d ” Me s s age Scr e en for Non-ad m i nistrator s .............................................. 4 0 5.7 F ilter Confi g u r ation Settin g s for the Administrator View .............................................. 4 1 5.8 F orm Used to Edit a Document’s Meta d a t a .................................................................... 4 3 5.9 Delete Confirmation Scree n ............................................................................................ 4 5 6.1 Sample Document V i ew. ................................................................................................ 49 6.2 F ilter Confi g u r ation Settin g s for Vi e w ing a D ocumen t .................................................. 5 0

PAGE 7

vii Abstract of Thesis Presented to the Graduate School Of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Engineering METADATA STORAGE FOR FILE MANAGEMENT SYSTEMS: DATA STORAGE AND REPRESENTATION TECHNIQUES FOR A FILE MANAGEMENT SYSTEM By Brennan Thomas Folmer August 2002 Chairman: Dr. Douglas D. Dankel II Major Department: Computer and Information Science and Engineering Print data managed by a spooling system typically include some form of metadata. The Barr Enterprise Print Server (BEPS), an enterprise print management application from Barr Systems, Inc., stores this metadata in the form of a serialized flat file along with a spool file. During any file access or routing activity that requires metadata access, this information must be de-serialized and stored in memory in a complex tree structure. This data structure is used throughout BEPS with many parts of its design and implementation becoming very dependent on it. This has proven to be a hindrance when new features or modules are added to the product, because any changes to this data structure invariably cause unintended and unwanted side effects. This thesis addresses the problem of metadata storage and introduces mechanisms for distributed file submission, viewing, and maintenance of the document system.

PAGE 8

viii The new design employs a relational database as the vehicle for metadata storage. A database system provides standardized access to the metadata and eliminates the need for a complex data structure. Standardized access to the metadata also frees the design of new BEPS modules from the reliance on a proprietary data structure and opens the door for third-party module development. In general, moving toward a commonly used and well-known data-access mechanism removes several obstacles of progress for this product. Web components, implemented as Java Servlets and Filters, provide access to the contents of the document system, as well as provide a way for new files to be submitted to the system. These web components contain the business logic of the system and enforce the security model. Several views of the spool contents are available and are presented according to the userÂ’s desires and security permissions. To provide flexibility for the design of these views, the underlying file and spool contents are stored as XML data. XSLT is then used to transform the XML data into HTML before providing a view to a remote client. This allows the specific layout of a particular view to be altered without having to change the underlying data. Finally, authorized users of the system are allowed to modify the metadata of a particular file in the system or delete the file altogether. The business logic and the database management system manage concurrency issues involved with these operations.

PAGE 9

1 CHAPTER 1 PROBLEM DESCRIPTION Documents managed by a document management system (DMS) typically include some form of metadata. This metadata can represent any number of properties related to the document, including the data type, document owner, printing destination, and creation time. The specific type and quantity of information stored with a document is truly a function of the document type and the application in which it resides, but the mechanism for the storage and maintenance of this metadata is a crucial component of the system design. The central problem explored by this thesis is: “What is an effective way to maintain metadata in a DMS so that this metadata is readily accessible to the DMS, flexible enough to work with 3rd-party metadata schemas, and robust enough to withstand changes to the DMS system without causing unintended side-effects?” The motivation for this thesis comes from a situation encountered in a document management system called the Barr Enterprise Print Server (BEPS), a product from Barr Systems, Inc., of Gainesville, Florida. This PC-based product is primarily used to gather documents from various input sources (legacy applications, mainframe reports received via channel extenders, TCP/IP input, FTP, text documents, etc.) and route them to various output devices (printers, disk, COLD storage, LPR, etc.). As documents move into the system, they are converted into a common data format and are stored on disk in a central spool directory. During this process, metadata fields, called Network Job Entry (NJE) Headers, are associated with each document and stored as a flat file on disk. NJE

PAGE 10

2 Headers contain dozens of pieces of information about the associated document, and this information is used extensively by BEPS to process and route the document. However, several problems exist with this design. Internally, NJE Headers are represented as a complex tree-like data structure. Whenever metadata from these headers are needed by BEPS, the flat file must be deserialized before that information is available. If changes are made to any part of the information, the data must be serialized back to the file on disk. Since there are many points during the life cycle of a document in the system that this information might be used, this conversion from flat file to data structure and back again potentially happens numerous times. Clearly, significant processing overhead is added to the system, even for the simplest metadata access. Another problem is the incompatibility of this proprietary metadata scheme with other 3rd-party systems that may want to exchange documents with BEPS. NJE Headers are a concept used by mainframes employing the Network Job Entry protocol. BEPS is generally compatible in that situation, but the majority of opportunities for product collaboration come from other PC-based document systems. Integrating BEPS with complex data transformation software suites has proven to be very challenging, and the end result is often a loss of metadata. A more flexible metadata storage design would allow for easier integration with 3rd-party products and increase the business opportunities for Barr Systems, Inc. Additionally, this proprietary metadata scheme has proven to be quite intolerant of design changes. Adding new metadata fields to the NJE Headers or changing the type of any existing field results in incompatibilities with older versions of documents. Since

PAGE 11

3 many BEPS customers archive their documents and need the ability to retrieve them on demand, this is unacceptable. Changes to any part of the serialization or de-serialization process invariably caused unintended side-effects elsewhere in the product. Additionally, the sheer complexity of this data structure introduces a major learning curve to any person new to the project. While some of these problems could undoubtedly be eliminated by better documentation, it would be even more beneficial to have a metadata schema that is self-documenting. And, a schema that is more tolerant of change is crucial to improving the stability and maintainability of the BEPS product. These are the main problems that are addressed in this thesis. The metadata storage system shall meet the following criteria: Efficiently accessible to the DMS Flexible enough that the metadata can be integrated with other DMS systems Tolerant to changes and additions to the metadata itself To solve these problems, this thesis moves the storage and maintenance of the document metadata to a database management system (DBMS). A (simple) metadata collection is used to demonstrate the storage, access, and maintenance concepts, and a web-based framework is included to simulate a DMS. This project will be referred to as DocMan (Document Manager) throughout the rest of this thesis. Chapter 2 of this thesis is an overview of the technical documentation related to the technologies used. These include documentation for the database system, the Tomcat web server, various Java technologies, and XML concepts. This thesis has several distinct parts. First, a simple DBMS is included to store the metadata. Using a DBMS solved the problem of proprietary data structure

PAGE 12

4 incompatibilities, as any system can easily access data from the well-defined database interface. A DBMS offers very efficient access to the metadata, and the DMS has complete control over the amount of metadata it requests at any given time. This approach is extremely scalable and enables a DMS to access the data remotely if that is desirable. In addition, a DBMS offers many other benefits such as the maintenance of transactional integrity, concurrency controls, and record locking not available (without significant additional work) when using a flat file. Chapter 3 describes the design and implementation details of the DBMS. Second, a web-based file submission mechanism is included. This allows files to be imported into the DMS and the specification of metadata values. For this thesis, only standard ASCII text files are handled, but a more extensive system could be adapted from this design to handle any type of input file. As a file is imported, the metadata is entered into the DBMS and the data itself are transformed into a XML document and deposited into the central spool directory of the DMS. A Java Servlet running on the Tomcat web server is responsible for all of these transformations. Users employ this mechanism to import data from a remote location to the DMS. The technical details and usage pattern for this component are detailed in Chapter 4. Third, the contents of the DMS are available through several different views. These views provide varying levels of detail about the contents of the DMS and allow users with sufficient privileges to edit, display, or delete individual documents in the system. The three views offered by this project are the following: Basic DMS view: This provides a display of the documentÂ’s name, size, edit/display/delete operations, and owner summary, provided that the current user has sufficient security clearance.

PAGE 13

5 Detailed DMS view: This provides the same information as the Basic DMS view as well as a brief overview of the documentÂ’s contents. Administrator DMS view: If the user is an administrator, this view provides all metadata and all operations as well as an overview of the document. The DMS views are created by another Java Servlet querying the DBMS for the systemÂ’s metadata and then converting that metadata to XML. The XML is in turn processed by one of the Java Filters running on the Tomcat server. The resulting HTML is returned to the browser as the display. Security enforcement happens on the server side. All of the details and sample views are discussed in Chapter 5. The last major component allows the display of the document contents itself. Briefly, the XML document stored in the DMS is retrieved from the central storage directory and processed by another Java Filter. Once again, the web server performs user authentication and authorization before the document is returned to the requesting browser. Chapter 6 explains this process in detail. Finally, Chapter 7 summarizes the findings and discusses potential additions that could be made to this system.

PAGE 14

6 CHAPTER 2 LITERATURE SURVEY AND TECHNICAL OVERVIEW Implementing a useful prototype system required the use of several different technologies. This chapter provides an overview of each technology and the corresponding reference material. Technologies Java The bulk of DocMan is written in the Java programming language. This includes the Java Servlets, Java Filters, and all utility classes. Version 1.3.1 of the Java Development Kit (JDK) provides the basis of development, including a compiler, run-time environment, and Java Virtual Machine (JVM). The JDK is available free of charge from Sun Microsystems, Inc. Java is a programming language designed and released by Sun Microsystems in the mid-1990s. Its original promise of, “Write once, run anywhere” was to be achieved through the use of a platform-independent intermediate data format, generally called bytecode, that is the product of compilation. During execution, this bytecode is interpreted by a JVM that is implemented for the target platform. The JVM takes each bytecode instruction and converts it into binary data that can be executed by the target platform. In this way, a collection of bytecode, grouped into Java classes, can be moved from one platform to another and executed without re-compilation as long as each platform has a functional JVM available.

PAGE 15

7 While Sun may not have fully achieved the original promise of complete platformindependence, Java is widely supported and is used in a wide variety of application environments. JVMs are available for a wide variety of operating systems, including versions of Windows, Linux, UNIX, IBM mainframes, and even micro-devices such as Personal Digital Assistants (PDAs) and other embedded systems. Desktop applications, web servers, real-time systems, and database managements systems have all been implemented in Java. The scope of potential applications continues to grow. Along with the JDK, complete documentation of all standard APIs and packages are also available from SunÂ’s website ( http://java.sun.com ). This collection of Java Documentations (JavaDocs) proved to be invaluable during the development process. The information in these documents includes class names and descriptions, derived classes, member variables and functions, and general usage guidelines. Servlets Java Servlets are a technology that allows programmers to extend the functionality of web servers. They offer the full capabilities of the standard Java APIs and can be used for a wide variety of purposes. The primary task of early web servers was to react to requests for static HTML pages and return those pages to the requesting client. As the popularity of the internet grew through the 1990s, more and more uses for web-based applications became apparent. Instead of returning static HTML pages, it is now desirable to be able to customize a display for a particular user or provide a more advanced level of interaction. Instead of building this capability into the web server itself, many servers are built with the ability to be extended. Servlets are one example of this type of web server extension.

PAGE 16

8 The term servlet refers to a very general type of web server extension. There is a generic Servlet class that provides basic request processing capabilities. However, it is much more common for a programmer to extend the subclass HttpServlet , which encapsulates many of the common functions necessary for processing a standard HTTP request from a web browser. This is the approach used for DocMan. The HttpServlet class provides entry points for the various types of HTTP requests. In addition, the standard input and output streams used to receive information from the client and send responses back to the client are available in convenient wrapper classes. The capabilities of a servlet are not limited any more than any other Java application, meaning that servlets present a powerful platform on which to build web-based applications. The book Java Servlet Programming by Jason Hunter and William Crawford provided most of the information about servlets used in this thesis. The 2nd edition covers version 2.3 of the Servlet Specifications and provides multiple code samples for examination [HUN01] . Sections on the configuration of web servers and web applications also proved helpful during the implementation. Tomcat web server The Tomcat web server is a product provided by the Jakarta Project, an organization that provides open-source applications based on the Java platform. Tomcat is a servlet container web server, which means that it can host servlets. DocMan was developed using version 4.0.1 of Tomcat, which supports the final released version of the Servlet 2.3 specifications. Tomcat itself is built using the Java platform and offers a myriad of configuration options with which to customize the web server behaviors as well as the hosted servlets.

PAGE 17

9 Among other features, Tomcat is able to host multiple web applications at the same time, support Secure Socket Layer (SSL) encrypted communication streams, and enforce user authentication and authorization. Since it runs as a Java application within a JVM, this application can easily be ported to multiple platforms. Even though DocMan was developed on Windows 2000 and XP systems, it could be deployed on a Linux or UNIX systems without major changes. Complete documentation of the Tomcat web server is available from the Jakarta Project web site ( http://jakarta.apache.org/tomcat/ ). Java Filters Java Filters are a concept introduced in version 2.3 of the Servlet specifications. They allow for the transformation of requests and responses routed to servlets. Filters are set as an administrative function, so their use and manipulation does not affect existing code in the servlets themselves. Filters are a very flexible mechanism to manipulate data streams as they pass into and out of servlets. Multiple filters can be created for any given web resource, and they can be placed in any order. One or more filters can be applied to a web resource, forming a “filter chain.” DocMan uses filters to perform data conversions between XML and HTML formats, as described in Chapters 5 and 6. Since they are separate and distinct from the servlets, they can be easily manipulated without changing the behavior of the servlets. Information about Java Filters can be found in the Servlet 2.3 specifications, which is available on the Sun Microsystems Servlet web site ( http://java.sun.com/products/servlet/download.html ).

PAGE 18

10 XML The Extensible Markup Language (XML) data storage format is simply a way to organize and identify data. A particular document type is described by a Document Type Definition ( DTD ) or schema , which is a collection of XML tags and an organizational hierarchy describing what a well-formed document looks like. The main benefit of storing data in an XML schema is that it provides a platform-independent representation of the data, which facilitates the transportation between and manipulation by completely disjoint applications. Instead of being locked into a proprietary data format that can be used by only a single application, data can be wrapped in XML in such a way that any application that is aware of the schema can manipulate the data. Consider a document produced by a word processing application. If that document is stored in a file in a proprietary format, other word processing application would be unable to open or edit the document. Now suppose a standard word processing schema exists to which the word processor adheres. When the document is saved in a file as XML, it would be possible to take that file and manipulate it with another word processor (or any other application) that supports the same XML schema. The second application knows what to expect when it looks in the file, and it is able to process it independently of the application that created the document. Essentially, the schema provides a contract or interface through which various applications can communicate. Figure 2.1 shows a very simple example of an XML document. The very first line provides some reference information about the version of XML and the encoding used for the document. The following information represents the data itself. Each element enclosed in a pair of <> brackets is called a tag. These tags are used to delimit and describe the data stored in the document. In this example, the entire document is

PAGE 19

11 enclosed in a pair. There is a header section followed by a series of pages. Parameters can be a part of any tag, as demonstrated by the number parameter used in the page tags. These parameters provide more information about the data stored within the given tag.
Headerrecord1 Headerrecord2
Firstlineofpage1 Secondlineofpage1 Thirdlineofpage1 Firstlineofpage2 Secondlineofpage2 Thirdlineofpage2
Figure 2.1 Sample XML Document. A schema designer has complete flexibility when defining a schema for an application. They can create as many different tags as they want and name them appropriately for the application. Also, the hierarchy of the tags can be created in any way the designer desires as long as there are always starting and ending tags for each data entry. Such flexibility is a large reason XML is an attractive intermediate data format through which to exchange data between applications. There are numerous books and web resources dedicated to the topic of XML. For this project, the book Professional XML by Didier Martin, et al. provided general information about the topic [MAR00] . XSLT XSLT stands for Extensible Stylesheet Language Transformations. As mentioned above, XML provides a way to organize and describe data, but it does not provide any

PAGE 20

12 way to actually do anything with the data. Knowing the specifics of an XML schema is a necessary step when trying to use an XML document, but it is impractical to expect every new application that wants to get at the information to have its own parser compatible with that schema. There was a need for a standard way to manipulate XML data, and XSLT fills that need. In much the same way that the SQL language provided a declarative way to query or manipulate data in a relational database, XSLT provides the same functions for XML data. Using XSLT turns an XML document into an information source which can be accessed in a standard way by different application. XSLT by itself is just a language that allows a programmer to process XML data. To apply XSL templates, there must be an XSLT processor available that takes the template and applies it to a given XML document, thereby producing the desired output. XSLT is flexible enough to allow many different kinds of conversions. XML data can be converted to HTML (as in the case of this project), plain text, PDF, postscript, and other XML schemas, just to name a few. The XSLT processor used for this thesis is provided as part of the javax.xml.transform and javax.xml.transform.stream Java packages. Specifically, the TransformerFactory and Transformer classes are provided which perform the application of the XSL template. The details of this XSLT conversion are discussed in Chapters 5 and 6. The main source of information about XSLT used for this project is the book XSLT: ProgrammerÂ’s Reference written by Michael Kay [KAY00] . Other References In addition to the technologies listed above, several other sources were consulted during the design and implementation of this thesis.

PAGE 21

13 Sun Microsystems maintains a developer resource called Java BluePrints that provides a wide variety of guidelines, design patterns, and code samples applicable to various Java development platforms. Included in this information source are articles related to webapplication development and database access techniques that were helpful during the initial design phase. Surveying the published design patterns also provided many ideas that were used in DocMan. Also from Sun Microsystems, a comprehensive tutorial covering the Java 2 Platform Enterprise Edition (J2EE) provided a working example of a complex web application employing a web interface, servlets, and a database backend. Studying this tutorial illuminated many of the design topics touched on in the Java BluePrints articles and enabled the design of DocMan to be more complete. Chapter 3 describes the database management system, Cloudscape, which is employed by DocMan. IBM provides a comprehensive data sheet and technical overview describing the capabilities and best-use techniques for this database. During the database system setup, the information in these two documents provided enough technical information about the database design to quickly get the DMS up and running. The following chapters detail the application of these technologies in DocMan.

PAGE 22

14 CHAPTER 3 DATABASE DESIGN AND USAGE Database Management System (DBMS) Overview This thesis employs a DBMS called Cloudscape™ 3.6, which is a product of IBM. Cloudscape is provided free of charge with the Java 2 Enterprise Edition 1.3 (J2EE) distribution by Sun Microsystems. The following features made Cloudscape a good fit for this research: Pure Java Application – This DBMS is implemented as a Java application, which runs within the same Java Virtual Machine (JVM) housing the Tomcat Web Server. No platform-dependency issues have to be addressed because of this feature. [WYA01] Java Database Connectivity (JDBC) driver availability – A compatible JDBC driver is readily available for communications between the Java Servlets that process the business logic and the DBMS. [WYA01] Easy setup and maintenance – Cloudscape requires no administrative maintenance after installation. Once the table schema is prepared and the driver settings are in place, the DBMS can be started and left alone. [WYA01] Scalable – Cloudscape can be scaled from a small embedded application to a multi-server environment, which is a benefit for the document management system. Various customers will have different capacity requirements, and a single database that can handle a wide range of situation is beneficial. [WYA01] Remotable – Remote applications can access this DBMS in a variety of ways (network connection, wireless, etc.). While this research does not take advantage of this capability, extensions very well could. Standard Programming API – The use of the industry standard JDBC API allows the application developer to write and test code that is compliant with the Cloudscape DBMS and later switch to another JDBC database at a later point without having to redo the implementation.

PAGE 23

15 Since this project is intended to be a proof-of-concept for the idea of storing document metadata in a database, maximum flexibility is desired from the DBMS platform. Cloudscape is sufficient for this task, readily available, and well documented. The RmiJdbc JDBC driver used to access Cloudscape is a Type 4 driver [WYA01] . It is pure Java on the client and connects directly to the database engine without any intermediate translation. RMI stands for Remote Method Invocation, a communication protocol available to Java applications. This driver is sufficient because Cloudscape executes in the RmiJdbc server that is provided with the Cloudscape distribution. When establishing a database connection to Cloudscape, the Universal Resource Locator (URL) supplied to the connection object for this project is jdbc:cloudscape:rmi:thesis . All URLs for Cloudscape must begin with the prefix jdbc:cloudscape: . For this implementation, the next part of the URL is rmi because the communication will happen over RMI. Finally, the name of the specific database holding the document metadata is named thesis . This URL completely identifies the destination database. The actual driver that is used by the Servlets to communicate with the DBMS is COM.cloudscape.core.RmiJdbcDriver , which is supplied with the J2EE distribution. Before any objects in the java.sql package can access the JDBC driver, it must be loaded using the following command: Class.forName(“COM.cloudscape.core.RmiJdbcDriver”).newInstance().This command loads the driver only once per application instance. After the driver has been loaded, a connection can be established in the following way: Connectioncon= DriverManager.getConnection("jdbc.cloudscape.rmi.thesis”).

PAGE 24

16 The con object can then be used to create Statement s and ResultSet s with which the database manipulation can be accomplished. Database Schema Metadata associated with documents in a DMS can take many forms. They can be as simple as a few bits of information about the document such as a name and author or involve hundreds of pieces of information describing a multitude of properties about the document. The scope of the metadata is truly a function of the document type and the application(s) creating and manipulating such documents. For example, the full NJE Headers that are associated with documents in the BEPS system may contain more than 300 individual fields. These fields hold data such as the job number, destination device, originating node, number of copies, and delivery room number, among many others. Most of these fields contain information that originates from the creating applications, which happen to run on various mainframe platforms (P390, POWER, JES2, etc.). While this research would have to be capable of handling that amount of metadata in a production or real-world environment, the concept of storing such metadata in a database can be demonstrated with much less actual data. There is no need to complicate matters with an enormous number of fields when those fields are just translated directly to fields in a database table. To simplify this process, a limited set of metadata are used and stored in the database. The following fields comprise that information: File name – The original name of the document when it is imported into the system. Description – Optional field supplied by the user when importing a document.

PAGE 25

17 Overview – The first 500 characters of a document. This information is used in some of the DMS web views to give the user an idea of the document contents without having to view the entire file. Owner name – Name of the document’s owner. Owner ID – User ID for the document’s owner. Group ID – Group name that the user supplies when importing a document. Version – Optional version information about the document. File size – Size of the document when it is imported. Owner view permission – Specifies if the document owner can view the document. Owner edit permission – Specifies if the document owner can edit the metadata of the document. Owner delete permission – Specifies if the document owner can delete the document from the DMS. Group view permission – Specifies if members of the document’s group can view the document. Group edit permission – Specifies if members of the document’s group can edit the document’s metadata. Group delete permission – Specifies if members of the document’s group can delete the document from the DMS. Others view permission – Specifies if any other user can view the document. Others edit permission – Specifies if any other user can edit the document’s metadata. Others delete permissions – Specifies if other users can delete the document from the DMS.

PAGE 26

18 These 17 pieces of information are sufficient to demonstrate the concepts of this thesis and provide enough information to illustrate reasonable content views of the DMS via a web browser as well as a security policy. Relationship designation between all of these attributes and the main database entity “document” is quite simple. Each document entity has one and only one instance of each property listed above. Each document must also be a unique entry in the database, so an integer primary key was included. Since Cloudscape is a SQL-based DBMS, the spoolcontents table in the thesis database was created with the SQL script in Figure 3.1. createtablespoolcontents (idintegerconstraintpk_spoolcontentsprimarykey, file_namevarchar(255), descriptionvarchar(256), overviewvarchar(500), owner_namevarchar(24), owner_idvarchar(24), group_idvarchar(24), versionvarchar(24), file_sizeinteger, owner_viewinteger, owner_editinteger, owner_deleteinteger, group_viewinteger, group_editinteger, group_deleteinteger, others_viewinteger, others_editinteger, others_deleteinteger);Figure 3.1. SpoolContents Table Creation Script The single spoolcontents table with these fields satisfies the needs of this thesis. Clearly, more complicated metadata schemas could be created and employed, with these complex schemas containing more than a single table. Database Usage Pattern All six Servlets used access the DBMS at some point. The DataImport Servlet is responsible for importing documents into the DMS, and part of this activity is to add an appropriate entry to the spoolcontents table for the new document. The TotalSpoolView

PAGE 27

19 and AdminView Servlets gather information from the DBMS in response to a user request from a web browser. Full document contents can be requested by accessing the DocView Servlet, which involves a query to the database to determine the correct location of the document contents on disk. Finally, the Edit and Delete Servlets handle metadata manipulation and document removal, respectively. Each Servlet follows a certain pattern when communicating with the DBMS. In addition to loading the correct JDBC driver and creating a connection (as discussed at the beginning of this chapter), valid SQL statements must be constructed, executed, and processed. The general usage pattern is shown in Figure 3.2. //CreateDBaccessobjects Connectioncon=null; Statementstmt=null; ResultSetresult=null; try { //Loadthedriver Class.forName(“COM.cloudscape.core.RmiJdbcDriver”). newInstance(); //Gettheconnection con=DriverManager.getConnection (“jdbc:cloudscape:rmi:thesis”); //UsetheconnectiontoQuery,Modify,Delete... //Closetheresultset,ifthereisone try{result.close();} catch(Exceptione){} } finally { //Closetheconnection try {if(con!=null) con.close(); } catch(SQLExceptione){} //Closethestatementobject,ifthereisone try{stmt.close();} catch(Exceptione){} } Figure 3.2 Database Usage Pattern

PAGE 28

20 To summarize, the standard process involves loading the driver, establishing the connection, querying or otherwise modifying the database, and then cleaning up any opened connections or objects. Maintaining Database Integrity Transactional integrity is maintained with the default settings of the Java objects. There are no situations in this implementation where multiple queries happen as part of a single transaction, so there is no need to implement an access scheme any more complex than that provided. The DataImport Servlet adds new tuples to the database table and the Delete Servlet removes tuples. All of the other Servlets either perform simple data queries (gathering information to provide as part of the view) or update parts of the metadata.

PAGE 29

21 CHAPTER 4 DOCUMENT SUBMISSION To facilitate the simulation of a real DMS, there must be a mechanism by which documents can be imported into the system. Submission via the internet is a logical choice since the rest of the project is managed through web screens. This chapter discusses the aspects and details of using the internet for submissions. User’s Perspective The DMS has a menu structure on the top left side of all screens, as shown in Figure 4.1. One of the top-level menu options is Upload Files . By selecting this menu item, a user is taken to a screen that gathers the relevant information to import a document. This screen is shown in Figure 4.2. Figure 4.1 Top-level Navigation Menus Several pieces of information are requested on the Upload Files screen. First, the user must identify which file is to be uploaded by clicking on the “ Browse…” button and navigating to the file. As mentioned previously, for this thesis the target file must be a standard ASCII data file, but there is no reason why this application could not be expanded to handle any number of data formats. The following sections discuss the data processing steps and indicate where changes would be made.

PAGE 30

22 Figure 4.2 Upload Screen After the file is identified, the user is prompted for the document owner’s name, a document version, a document description, and a group ID. These fields are optional but provide the user with a way of identifying their document. The only required field is the file name. Further, default permission settings for this document are provided, which the user can change as they see fit. The various persons who might access documents in this DMS are divided into three categories: owners, group members, and everyone else. This grouping mirrors the user grouping concept used on UNIX operating systems. A document has one and only one owner, and it is part of a group (even if that group is just the default group). The three permissions that can be set or restricted for any given document are the following: View – Can a particular user view the contents of this document?

PAGE 31

23 Edit – Can the metadata associated with this document be changed? Delete – Can this document be deleted from the DMS? The View permission not only applies to the contents of the document, but also to the metadata other than the document name and the owner’s user ID. The distinctions are discussed in detail in Chapter 5. Figure 4.3 Upload Confirmation Screen Once all the pertinent information is entered into the form, the “Upload” button is pressed to send the document and metadata to the DMS. A confirmation screen is displayed, as shown in Figure 4.3. After a brief pause at this confirmation screen, the user is redirected back to the main DMS contents view. Details of the document processing after this point are the subject of the next section. Document Processing Document Transportation When a file is uploaded to a web server through a form on a web page, the web browser is responsible for properly formatting the request. This involves gathering the data from the local disk, adding it to the HTTP POST header and body, marking the content type correctly, and transmitting the resulting buffer back to the web server.

PAGE 32

24 Whichfiledoyouwanttoupload?
Whatistheowner'sname?
Whatistheversion?
Whatisthedescription?
WhatisthegroupID?
Figure 4.4 Upload Form HTML Definitions Modern HTML forms support the tag type, which allows users to identify files destined for upload. DocMan uses an HTML form declaration similar to that shown in Figure 4.4. Note that the form is submitted with a HTTP POST method call. This allows an unlimited quantity of data to be sent as part of the HTTP request body, which is desirable when uploading documents whose size could easily be in the megabyte range. Users complete this form by adding information such as the group ID and version, which becomes the documentÂ’s metadata, and identifying the fileÂ’s name and location. Identification can be done either by explicitly typing the full path and filename or using the Browse button to navigate the file system for the file. Once the Upload button is pressed, the web browser builds the HTTP request headers and body. Normally, data streams containing HTML form data are labeled with a content type of application/x-www-form-urlencoded , but this is not sufficient when uploading data files. This standard content type creates a series of name/value pairs and sends them in the HTTP request. However, file contents must be segregated properly in the body of the HTTP request, so a content type multipart/form-data is employed [NEB95] . For this application, the metadata fields are written into the request header as encrypted parameters and the documentÂ’s contents are written to the body. This entire buffer is then transmitted to the web server over the standard socket connection where it is received and parsed by the Servlet.

PAGE 33

25 It should be noted that DocMan transmits the request buffer over a socket connection using the Secure Socket Layer (SSL) protocol, which encrypts the data stream before sending it across the network. DocMan establishes a secure connection between the web browser and the web server before the Upload page is displayed. Encryption protects the privacy of user data that might travel over unsecured channels on its way to the web server, which is often a critical requirement for commercial document management systems. Data Stream Parsing All parts of the HTTP request header and body are accessible to the web server and its applications. This allows application agents, such as Java Servlets, to gather user data and act on it in a meaningful way. In the case of servlets, the HTTP request header and body contents can be read through the standard input stream. Wrapper classes such as BufferedReader and ServletInputStream can be used to facilitate this data extraction, or the entire HTTP request can be encapsulated in a utility class. This second option is the approach used in this thesis. The class named MultipartRequest takes as initialization parameters the encapsulated HTTP request object (HttpServletRequest) and a destination directory on the web server for the parsed file contents. It then extracts all parameters from the user fields and all file data from the request body, making that data available to the servlet. Since the file requires further processing, the servlet instructs the MultipartRequest object to write the file to a staging directory and then sends a message to a separate processing thread that performs the final conversion. The details of this process are given in the following sections. The MultipartRequest object also has access methods allowing the servlet to easily query for the userÂ’s metadata. This feature is described below.

PAGE 34

26 Directory Monitoring The staging directory allows the servlet processing HTTP requests to dump files and metadata information into a holding location so that responsiveness for web clients is maximized. A separate worker thread is used to perform the data conversion and metadata entry for the DMS. Since this processing can be costly in terms of CPU processing time, it is better to allow the thread servicing the initial request to return a response to the client quickly and delegate the processing to a dedicated worker thread. This thread is started the first time the DataImport servlet is accessed (which corresponds to the first time a file is uploaded), and it monitors the staging directory every few seconds for new entries. In addition to the file itself, the associated metadata must be entered properly into the DMS. A queue is available for the initial thread to submit the metadata information, and the worker thread grabs this metadata when it is about to do the final processing for a particular file. The file is converted to XML and moved to the standard storage directory, and the metadata is entered into the database and associated with the new file. Conversion to XML To convert a new file to XML, the LineNumberReader and FileWriter classes are used. LineNumberReader takes a filename as an initialization parameter and allows the servlet to read a single line of ASCII text data from the file at a time. Since DocMan only supports ASCII text data, this process is quite simple. However, a more elaborate data processing scheme could be devised to process any number of different input file types. The XML format could be expanded to support as many data file properties as necessary for a particular application.

PAGE 35

27 Suppose an application has a need to import data generated by a proprietary, third-part product into this DMS. Currently, the DataImport servlet performs all data conversion, but this could easily be changed so one or more separate classes are used to convert various data formats to the single XML schema. This conversion process is not tied into the insertion of metadata information into the database, so it can be separated easily. The design is flexible enough to accommodate this potential scenario. A destination file is opened in the SPOOL directory using the FileWriter class, and the XML header information is written to the file first. Then, each line of the source file is read and marked with XML tags. Any invalid characters such as “&,” “<,” or “>”are exchanged for their coded equivalents at this point to prevent problems with the XML processing happening in other parts of the system. Each marked-up line is written to the destination file. When the entire file has been moved, the source and destination files are closed and the source file is deleted. At this point, the document is completely in the DMS, but it does not yet have its associated metadata in the database. Until the database is updated, the document will not show up in any of the content views. Metadata Storage As mentioned above, a queue is used by the request processing thread to pass metadata information extracted from the HTTP request to the worker thread for insertion into the database. As the worker thread monitors the staging directory, it is also looking in this queue for new metadata entries. When it receives one, the database insertion process is triggered. The object extracted from the queue is a simple class containing fields for each type of metadata supported in DocMan. The worker thread grabs this information and creates a

PAGE 36

28 SQL statement to update the database. It follows the database access pattern described in Chapter 3. Once the worker thread organizes the metadata information into a valid SQL statement, it calls the database adding a new entry to the spoolcontents table. Once this entry is in the database and the converted document is available in the SPOOL directory, the document is fully accessible to users of the system. Document Management System Availability Once a document successfully navigates the steps described above, it is part of the DMS system. It is then included in any of the three views available to users of the system, and its contents are available for viewing by users with proper permissions. The document stays in the system until a user either deletes it or it is removed by another application. .

PAGE 37

29 CHAPTER 5 VIEWS OF DMS CONTENTS A central feature of a document management system is the ability to view the contents of the system and manipulate the documents it contains. Users need the ability to get this information displayed in such a way that is meaningful and concise, and the system itself should allow views to be changed or added with minimal effort. This chapter describes the views available to users of the system and shows approaches for adding new views. Security Model Before covering the details of each view, the security model used for DocMan should be discussed. Each document has its own set of security permissions maintained in the database along with the other metadata. These permissions consist of the following three categories, called actions : View, Edit, and Delete, which are discussed later in this chapter. Every user of the system is also categorized into one of three groups for each document: owner, group member, or everyone else. As an illustration, consider the following scenario: Joe from Group1 submits a document to the system named SampleDoc . When Joe views the document in the system, as far as the document is concerned Joe is in the owner category, giving him all the permissions granted to SampleDoc Â’s owner. Now suppose Sara, who is also part of Group1, views the system. From the perspective of SampleDoc , Sara is not the owner but she is part of the same group, giving her all the permissions granted to group members of SampleDoc . A third

PAGE 38

30 user, Eugene in Group2, views the document and is classified in the “everyone else” category. Eugene is neither the owner nor a group member and is, therefore, placed into the most general category. This example demonstrates the fact that these categories are of decreasing exclusivity. Joe, the owner of the document, is also clearly in the group where the document originated, but since the owner category is more exclusive, Joe is granted owner permissions. Likewise with Sara; she is not the owner, but she is in the group, giving her that primary categorization. Eugene only qualifies for the “everyone else” category. One other interesting aspect of this setup should be mentioned. Suppose SampleDoc ’s permissions grant the right to delete the document to users in the group category, but not the owner or the “everyone else” categories. Because Joe is not only the owner but also a group member, he would still be able to delete the document, even though the owner delete setting is not granted. Sara would also be able to delete the document, but Eugene would not. This means that the system will authorize a user for a given action based on their category membership. First membership in the owner category is checked, then group and finally everyone else. So what is the relevance of this classification scheme? As the next few sections show, the information displayed and the actions available to a given user are determined by their classification from the perspective of each document in the system. Description of Views Basic View The first view offered by DocMan is the Basic DMS View. Figure 5.1 shows a sample of this view.

PAGE 39

31 Figure 5.1 Basic DMS View The main purpose of this view is to show every document in the system with a small amount of identifying information. This is intended to give users a quick overview of the contents of the system Each row displayed in the view represents a separate document with the actions available to the accessing user (which is bfolmer in Figure 5.1) are represented by the icons to the left of each entry. The three actions, view, edit, and delete, are represented by the eye-glasses, pencil and trash can, respectively. If one of the icons is not displayed next to a particular document, the accessing user does not have sufficient permission to perform that action. For example, the first document in Figure 5.1 is missing the trash can icon, indicating that the user, bfolmer , does not have permission to delete this document from the system.

PAGE 40

32 The only information that is always displayed for every document in the Basic DMS View is the documentÂ’s name and owner. If the accessing user does not have permission for the view action, this is the only information displayed. An example of this is demonstrated by the second document in Figure 5.1. The document name, debug.txt , and the owner, bfolmer , are shown but nothing else. Note that the view icon is also missing, indicating the lack of view permissions. Since the accessing user, bfolmer , is also the owner of the document, it must be the case that none of the three categories (owner, group, and everyone else) was granted permission to view the document. Every other document in Figure 5.1 includes more information, a situation reinforced by the presence of the view icon in their left columns. The extra information provided by this standard view includes the documentÂ’s description, group, version, and size. This information is identified by the column labels included with each document entry. Also, every action allowed by the permission set is made available with the inclusion of the appropriate icon. Detailed View The Detailed DMS View is similar to the Basic DMS View, but additionally includes an overview of the documentÂ’s contents. See Figure 5.2.

PAGE 41

33 Figure 5.2 Detailed DMS View The Detailed DMS View allows a user to quickly scan through the documents in the system looking at the first few lines of the document contents. This is useful when searching for a document of unknown name, but whose content is recognizable. Offering different displays of the DMS contents allows users to select what is most appropriate for their situation. Comparing Figures 5.1 and 5.2 shows that the amount of information displayed for each individual document is the same except for the addition of the overview information. Notice that the second document in the list is identical for both views. Since the accessing user does not have view permission, the extended metadata and the document overview are not available. For every other document, the same metadata information is displayed at the top of the entry with the overview added below.

PAGE 42

34 The same actions are available for each document in each of these two views. If a particular action, for example edit , is available in the Basic DMS View, it is available in the Detailed DMS View as well. Administrator View The final view of the DMS contents provided by DocMan is the Administrator View. Much more information is displayed in this view, as shown in Figure 5.3. Figure 5.3 Administrator View The Administrator View is provided so that an administrator of the DMS can see all relevant information about each document as well as perform any of the actions on these documents. As such, only users that are part of the administrator group, specified in the Tomcat configuration files, have access to this view. The file tomcat-users.xml contains the list of all system users and their group memberships, including the DMS

PAGE 43

35 administrators. Of course, as with any system, the group of users with administrative rights should be strictly limited to persons completely familiar with the workings of the system who also have some reason to access these features. Comparing the Administrator View in Figure 5.3 to the other two views shows that more information is presented to administrators of the system. In addition to all the information presented in the Detailed DMS View, the following items are included for each document: the document ID, the physical location on the serverÂ’s file system, and the permission settings for each category of user. Also, all actions are available to the administrator for each document, regardless of the documentÂ’s permission settings. A documentÂ’s ID is the unique identifier for a document in the metadata database. This information may be useful for a system administrator when there is a need to manipulate the database by hand. Likewise, the location of the document (in XML form) on the serverÂ’s file system may be useful for debugging purposes. Both of these pieces of information are required very infrequently, if at all, and are, therefore, only displayed to DMS administrators. The permission settings for each category of users are indicated in the grid. A green check mark means that the user category is granted permission for the associated action; a red mark means the category is denied. These settings are changed as part of the edit action, as discussed in the following sections. Creation of Views Basic View The process of providing a particular view to the user starts with a userÂ’s selection from the View menu, as shown in Figure 5.4. The three options from the View menu are Basic, Detailed, and Administrator.

PAGE 44

36 Figure 5.4 View Menu Options Once a selection is made, a request is sent to the web server and is passed to the TotalSpoolView servlet (in the case of the Basic and Detailed DMS Views). This servlet gathers the necessary information from the database, wraps it in XML tags, and sends the result along the response channel. Before the response buffer is returned to the web browser, it is intercepted by the XSLTFilter, a Java Filter, and the XML is transformed to properly formatted HTML using the XSL Transform language. The resulting HTML is displayed by the web browser. First, the TotalSpoolView servlet determines the type of request. This servlet handles the processing for both the Basic and Detailed DMS Views, so it must know what information it is expected to return. A parameter is passed with the HTTP request identifying the view, and the servlet extracts this information. An appropriate SQL statement is created to query the database (following the pattern described in Chapter 3) and the request is made. The main difference between the SQL statement created for the

PAGE 45

37 Basic and Detailed Views is the inclusion of the document overview field from the database. If the view type is Basic, it would be wasteful to extract that information from the database when it would just be discarded. Once the database information is obtained, the response buffer begins to take shape. The permission settings for each document are examined with regard to the requesting user. Available as part of the HttpServletRequest object is a method called getUserPrincipal to determine the identity (username) of the requesting user. A series of comparisons determine the appropriate category membership for the user, and three document actions are either allowed or restricted based on this categorization. XML tags are created for each piece of information, identifying it for modules later in the processing sequence. For example, if the version of a particular document is 2.0 , that information would be expressed in XML as 2.0 . Each field for every document is marked, and the entire XML buffer is sent out via the response channel. XSLTViewFilter filters.XSLTFilter xslt /WEB-INF/classes/xslt/view.xsl XSLTViewFilter /TotalSpoolView Figure 5.5 Filter Configuration Settings for the Basic and Detailed Views Tomcat 4.0 supports the ability to configure Java filters for input and/or output streams. Each web application on Tomcat has its own web.xml file, which store these settings. Filters are used to intercept data and examine or change them as necessary.

PAGE 46

38 Figure 5.5 shows a snippet of the configuration file used to create the filter processing for DocMan. Several important aspects of filter configuration are illustrated in this example. The first grouping identifies a filter to the system. A filterÂ’s name, Java class, and initialization parameters are necessary to define the filter. In this case, the XSLTFilter class is specified, and the xslt parameter is set to /WEB-INF/classes/xslt/view.xsl . This identifies a file (relative to the root directory of the web application) used as the XSL template. Such a template can be used when a data stream is converted from XML to HTML, as it is in this situation. The second grouping creates the filterÂ’s mapping. The filter name and URL pattern is specified. In this case, /TotalSpoolView is used as the URL pattern, meaning that any responses coming from the TotalSpoolView servlet will be processed. Filters can affect the data stream before and after it gets to the servlet, if so desired. However, this particular filter does not change the data on the way into the web server. Filters are quite powerful in part because they can be used in this manner, and also because they can be chained together in any desirable configuration. When the (XML) data stream reaches the XSLTFilter filter, objects from the XML transformation package are employed to apply the XSL template. Specifically, instances of the Transformer and TransformerFactory classes are created and initialized, and the XML data are given to them for conversion. Once the conversion is complete, the resulting data (HTML) are returned across the network to the web browser and the interaction is complete.

PAGE 47

39 Detailed View Creating the Detailed View is very similar to creating the Basic View. The request is sent to the web server and passed to the TotalSpoolView servlet. A database query is made to gather all the information necessary to create the view and the resulting data are wrapped in XML. The XML buffer is passed to the XSLT processor and converted to HTML, which is displayed to the user. As noted above, the main difference in the SQL queries created for the Basic and Detailed Views is the inclusion of the document overview field. This field is included for the Detailed View and it is wrapped in a pair of ... tags when the XML buffer is created. Every other field is placed in the same type of XML tag as is used when creating the Basic View. The same filter mapping is used for this view as the Basic View, as shown by the filter mapping in Figure 5.5. Since this mapping applies to the entire TotalSpoolView servlet, the same XSLT processing takes place. This is the primary reason the XML tags must be the same. If different tag names were used, a separate XSL template would have to be applied to the resulting buffer, which would create unnecessary complexity. Since the layout of both views is very similar, it makes more sense to use one XSL template that can handle the inclusion or exclusion of the tag. The rest of the processing is identical to that for the Basic View. The XSLTFilter applies the XSL template found in the /WEB-INF/classes/xslt/view.xsl and sends the HTML back to the web browser for display. Administrator View The Administrator View is more complex than the other two views, with its creation handled by the AdminView servlet. This view includes more information about each

PAGE 48

40 document than the other two views and gives the administrator full access to all actions for each document. Figure 5.6 “Access denied” Message Screen for Non-administrators Each request for an Administrator View is directed to the AdminView servlet. Since there is only one type of view processed by this servlet, there is no need for any parameters. First, this servlet authenticates the requesting user. Since this is a view intended solely for administrators, the requesting user must be authenticated as such. The HttpServletRequest object made available to the servlet framework provides a facility to determine group membership for any requesting user. A call to the isUserInRole(String) method of the HttpServletRequest object with the parameter “administrator” returns true if the requesting user is an administrator. If the user is not an administrator, a response is

PAGE 49

41 returned to the web browser immediately stating that access to the Administrator View is denied, as shown in Figure 5.6. Once the user is authenticated and authorized, the servlet proceeds to request all of the information used to create the view. In addition to the fields requested for the Basic and Detailed Views, including other fields such as the document ID, location on disk and permissions. The other views use the permission setting to determine which actions should be available to the requesting user, but this view includes that information right in the XML buffer itself. Since an administrator is granted permission to perform any action on any document, all of the actions are automatically included by the AdminView servlet. But, since the permission settings for each document are part of the display, they must be packed into the XML buffer along with the other information. Each piece of information is wrapped in XML tags and the resulting buffer is sent for XSLT processing. XSLTAdministratorFilter filters.XSLTFilter xslt /WEB-INF/classes/xslt/adminview.xsl XSLTAdministratorFilter /AdminView Figure 5.7 Filter Configuration Settings for the Administrator View Figure 5.7 shows the filter definition and mapping entry used for the Administrator View. This is similar to the settings used for the other views with a few important differences. The name of the filter (XSLT Administrator Filter) is different but the filter class XSLTFilter is identical. Also, the XSL template file adminview.xsl is different.

PAGE 50

42 Since the layout of the Administrator View is significantly more complex than the other views, a separate XSL template is used to model the conversion from XML to HTML. The filter mapping is the other main difference. The URL pattern /AdminView is used to indicate that any requests sent to the AdminView servlet should be passed through this filter. Just like before, the filter passes inbound data to the servlet without any changes, but intercepts the outbound XML buffer and performs the XSLT conversion. As always, an HTML buffer is the result that is returned to the web browser for display to the user. Actions Each of the three actions available for a document in the DocMan system is handled by a separate servlet. This section discusses the details of each. View If a user wants to see the full contents of a document, they select the View action. As shown in the figures at the beginning of this chapter, this action is represented by a pair of eyeglasses displayed to the left of a documentÂ’s name. This action might be desirable if DocMan is used as a repository and a user needs to retrieve some information from the contents of a document. Selecting this action sends a request to the DocView servlet on the Tomcat web server. The servlet gathers the contents of the document from the XML file stored on disk and sends it to a XSLT filter for conversion to HTML. The resulting HTML is displayed to the user. This process is described in Chapter 6. Edit Editing a document involves making changes to the documentÂ’s metadata. This action is represented by a pencil icon next to the documentÂ’s name. When this action is requested, an HTTP request is sent to the web server and directed to the Edit servlet.

PAGE 51

43 This servlet handles both aspects of the editing process: displaying a form for the user to change and applying the changes once they are made. If a user wants to edit the metadata for a document, she clicks the edit icon and a request is sent to the web server. Part of this request includes the documentÂ’s ID, which the Edit servlet uses to gather the metadata from the database for the correct document. A HTML form is created and the documentÂ’s current metadata settings are included in the form. This HTML response is returned to the web browser for display to the user. Figure 5.8 Form Used to Edit a DocumentÂ’s Metadata Note that the Edit servlet generates HTML itself. This is a departure from the processing sequence used when creating the views. Since the creation of a HTML form with a preset number of fields is quite simple, it is easier to bypass the process of generating XML and converting that to HTML.

PAGE 52

44 The user is presented with a form, similar to that shown in Figure 5.8, containing the current metadata values. She can change any values she wishes, and then submits the changes with the “Submit Query” button. This directs another request to the Edit servlet. The servlet determines that this second request includes changes that must be applied by querying an “action” parameter. Each field value is extracted from the HTTP request buffer, a SQL query is built, and the database is updated with the new information. If there are no errors encountered during this process, the user is redirected to a view of the system, which should now reflect the applied metadata changes. Delete The Delete action allows (authorized) users to remove documents from DocMan. For a document to be completely removed its metadata information must be erased from the database and the document’s contents (stored as an XML file on the web server) must be deleted. The Delete servlet handles this processing. This action is represented by a trash can icon displayed to the left of the document’s name in all of the views. If a user clicks this icon, an initial request to delete is sent to the Delete servlet. The servlet looks for a parameter called “confirm” in the HTTP request buffer. If this parameter is missing, which is always the case during the initial request, the servlet creates a confirmation screen to display to the user. An example of this screen is shown in Figure 5.9. If the user selects “no,” they are returned to a view of the system. If they select “yes,” another HTTP request is sent to the Delete servlet that includes the “confirm” parameter as well as the document’s ID. The servlet then removes the document from the system.

PAGE 53

45 Figure 5.9 Delete Confirmation Screen First, the database is queried for the filename field of the document that is to be deleted. This filename field points to the documentÂ’s location on disk. When this is returned, the document is removed from the disk. Second, another SQL query is sent to the database to delete the documentÂ’s entry from the spoolcontents table. If these two actions succeed, the document is no longer part of DocMan. Finally, the user is returned to a view of the system contents that no longer include the deleted document. Extending Views One of the main benefits of storing data as XML is the independence of the data from any particular view. If done correctly, XML tags identify each element in the schema and include all the information about the data that is necessary for the intended application. In this situation, different amounts of metadata are included about each

PAGE 54

46 document when the XML buffer is created, but the XML does not provide any indication about how the documentÂ’s metadata is to be displayed. The final display format and medium (HTML) is determined in a separate step, namely the XSLT processing. A disconnection such as this allows for flexibility when displaying the XML data. As noted in the sections above, there are two XSL template files used in DocMan. The view.xsl file contains a template applied to XML data to generate a Basic or Detailed View, and the adminview.xsl file contains a separate template applied to the XML buffer generated for the Administrator View. It is within these templates that the details of the views are determined. For example, the Basic View consists of a series of HTML tables containing information about documents. There is nothing in the XML buffer stating that the metadata should be displayed in HTML tables, but that is how the XSL template is written to process the XML. The XML metadata could just as easily have been converted to plain text without the HTML tags, and this would obviously look different when displayed on a web browser or other display device. The point is that once the metadata is properly wrapped in XML, the method in which to display it is an independent concern. DocMan was written to display each document as a separate table. However, since the data are wrapped in XML, simply modifying the XSL template can change the details of the display. Suppose that the Basic View is to be changed to display only the document names. The view.xsl template file can be modified to allow the document names to pass through to the resulting HTML data while filtering everything else. After this change is applied to the web server, users of DocMan would start seeing the updated Basic View.

PAGE 55

47 This means that absolutely no changes to any of the servlets, the database, or the filter class are necessary to modify the displays. As long as the new XSL template uses the same XML schema and tag names, it can be used without other changes. There are several benefits to using such an approach. If the maintenance of a system is divided between several programmers, and each programmer is responsible for a different layer of the system (e.g., database, business logic in the servlets, display), changes to part of the system can be made independent of the other sections. The programmer charged with maintaining the database is not concerned with the specifics of the HTML display. Changes to the business logic in the servlets such as a security model change should not directly affect the database or the display. As long as a common XML schema is used as an interface between different layers, a level of independence can be achieved.

PAGE 56

48 CHAPTER 6 DOCUMENT VIEWING Any document management system must allow users to see the contents of their documents. DocMan is no different. It offers users the ability to view the contents of their documents through the same web browser interface that they use to see the document listings. This chapter describes the mechanism providing this feature. Document View Creation Chapter 5 describes the various system content views available to a user. One of the three actions available on each document to an authorized user is the View action. Selecting this action provides the contents of the document for the userÂ’s perusal. A sample of this view is shown in Figure 6.1. The process of displaying a document starts with a userÂ’s request. When a user clicks the view icon next to a document, a HTTP request is sent to the web browser. This request is directed to the DocView servlet, which handles the processing. For the DocView servlet to correctly identify which document is to be displayed, a document identifier is sent as part of the HTTP request. This document identifier corresponds to the id field stored in the spoolcontents table of the database. Once the servlet verifies that this information is present in the HTTP request, it proceeds to access the database.

PAGE 57

49 Figure 6.1 Sample Document View. Following the standard database access procedure outlined in Chapter 3, the DocView servlet makes a request to the spoolcontents table for the disk location of the document. As discussed in Chapter 4, when a document is submitted to DocMan, it is converted to an XML format and stored on the disk of the web server. The physical location on the web serverÂ’s disk is recorded in the database entry for the new document. This information is extracted by the DocView servlet and used to locate the XML document. Once the document has been located, the DocView servlet only has to return the contents (which are already in XML form) to the client via the response stream. The ServletOutputStream class is used by the servlet for sending raw data back to the requesting entity. The servlet opens a file reader to the XML data file and reads 16KB of

PAGE 58

50 data at a time, sending it directly to the output stream. When the entire file has been sent, the file and the output stream are closed and the servlet is finished with the request. At this point the XML data has been directed to the output stream that eventually becomes the HTTP response buffer. This stream is normally sent to the requesting client application. However, since the data is still in XML form, the client (a web browser) might not be able to display it properly. So, an XSLT filter intercepts the output stream and converts the XML data to HTML in much the same way that was done with the views discussed in the preceding chapter. The same XSLTFilter class mentioned before is used to perform the actual application of the XSL template to the XML data. As before, the filter must be defined for the web application and a URL pattern mapping must be established. A sample configuration is shown in Figure 6.2. XSLTDocumentFilter filters.XSLTFilter xslt /WEB-INF/classes/xslt/doc.xsl XSLTDocumentFilter /DocView Figure 6.2 Filter Configuration Settings for Viewing a Document This listing shows the use of the XSLTFilter class and the specification of the XSL template file ( doc.xsl ). The URL mapped to this particular filter is /DocView , which covers any requests to the DocView servlet. The XSL template used for this step is significantly different from those used to create the views described in Chapter 5. Creating a view of the system involves extracting the

PAGE 59

51 XML tags that express each document’s metadata settings and placing them into an HTML table. In this situation, the XML tags used to delimit the document itself must be processed and converted to HTML. Internally, each record of a document is expressed in a pair of ... tags and each of these records is extracted. Pages are delimited using entries, and the XSL template uses these to produce a “Page n ” entry and a horizontal page break, as shown in Figure 6.1. Finally, all of the document records and page entries are wrapped in HTML
...
tags to preserve the character and line spacing of the original document. The resulting HTML buffer is returned to the web browser and displayed. Modifying the Display of a Document As with the views described in the previous chapter, the method used to create the display of a documentÂ’s contents was designed to be flexible and accommodating to changes. Since the contents of the document are wrapped in XML and are separate from the processing that dictates the display, modifications to display details can be made without changing any part of the document. This flexibility creates many possibilities for the administrator of the DocMan system. The XML schema used to delimit the contents of each document effectively creates a contract between the data and any process manipulating the data. Since the XSL template is applied as an independent step and as long as all templates adhere to the XML schema, XSL templates can be switched whenever the display is to be changed. Suppose it is desirable for the document text to display in a fixed-length font. A new XSL template is created using this font, the settings for the XSLT filter used to process document views is changed to point to the new template, and any future requests are display in this new format.

PAGE 60

52 Another important benefit of this design is that it can easily handle issues of backward compatibility. As long as the XML schema used to store documents in XML stays the same, any valid XSL template can be applied to a document. Suppose a document is imported into the system at some point in the past and subsequently archived. While the document sits in the archival system, DocManÂ’s default document viewing scheme is changed. When the archived document is extracted from the archival system, it can be plugged right back into DocMan and will be viewable in the new display format. Or, if it is desirable to view the document in the exact same way that it was viewed at the time of archival, the older XSL template in use at that time could be applied again. Either way, the contents of the document stay the same. Administrators of DocMan have the flexibility to make changes as they see fit without running into later compatibility issues. Of course, as indicated above, this is possible as long as the XML schema does not change. But what if it does change for some reason? It turns out that the flexibility of XSL templates provides a solution for this scenario as well. XSL templates are not limited to converting data from XML to HTML. On the contrary, XSL templates can be used to convert between any number of data formats. This means that a conversion from one XML schema to another XML schema could be achieved with a properly-written XSL template. So, if the XML schema itself changes at some point, a processing component (e.g., filter, servlet) could be used to convert the old XML schema to the new XML schema before the standard conversion of XML to HTML occurs. This way, even documents archived in an old schema could be easily displayed when there is a schema translator available. Keep in mind that changes to an XML schema are not generally common or desirable, but if they do become necessary old data does not have to be lost.

PAGE 61

53 Features such as these can save significant time and energy in the maintenance of a system over the systemÂ’s lifetime. This chapter describes the process necessary to view documents in DocMan. The next chapter expands on extensions to DocMan as well as summarizing the results of this thesis.

PAGE 62

54 CHAPTER 7 CONCLUSIONS AND EXTENSIONS Extensions DocMan is a relatively simple prototype created to verify the concepts put forth by the thesis. However, extensions can be added that would increase the feature set and usability of DocMan. Examples of such extensions are given here. More View options DocMan offers a single way to view the contents of documents in the system. However, it is easy to use the XSLT language to perform more complex transformations of the XML data. XSL templates can be written to take the DocMan XML schema and produce PDF, PostScript, or plain text representations of the documents, to name just a few possibilities. Substituting the existing XSL templates used by the Java Filters and changing the Multipurpose Internet Mail Extensions (MIME) data stream type allows the browser to display any of these formats. More Metadata DocMan provides less than two dozen metadata fields to describe documents. The Barr Enterprise Print Server contains documents that might have several hundred such fields. An obvious extension to DocMan would be to expand the number of metadata fields it supports. In doing so, the metadata (and underlying database schema) would be redefined. Fields could be grouped according to relevant criteria (i.e., security settings, owner properties, and application description) and the database tables may be organized differently. Also affected by these changes would be the servlets that query the database

PAGE 63

55 and upload new documents, the XSL templates that produce views, and possibly the XML schema itself. Document management tools Additional features can be added to DocMan to increase the feature set. It may be desirable to have the ability to e-mail documents directly from the system or upload new documents via the File Transfer Protocol (FTP). Additional servlets can be created and added to the system to handle additions such as these. As new features become available, changes to the XSL templates can be made to provide these options to users. Different views of system contents The power of SQL can be harnessed to provide more options for user displays. These options can include the ability to group documents by owner, sort documents by a particular column, or allow users to pick which document fields to display and which to hide. Changing the servlets that access the database would provide these features. Since the data is in a relational database, options such as sorting, grouping, and displaying become much easier to provide. Implement interface to abstract database access Currently, DocMan includes the database access code within the servlets processing the HTTP requests. As the system becomes more complex, this approach would prove problematic because the commingling of database code and HTTP request processing would add unnecessary complexity to the code base. Preferably, the database code would be separated into a different module providing a well-defined interface through which data requests would be made. If different programmers are tasked with each module, this division will prevent sharing problems and other conflicts that could arise. Also, it is simply a cleaner design to divide major functional sections into separate modules.

PAGE 64

56 Thesis Conclusions Recall that the central problem explored by this thesis is: “What is an effective way to maintain metadata in a DMS so that this metadata is readily accessible to the DMS, flexible enough to work with 3rd-party metadata schemas, and robust enough to withstand changes to the DMS system without causing unintended side-effects?” The three parts of this thesis can be addressed individually. Metadata accessibility for the DMS Recall that a major problem affecting the Barr Enterprise Print Server is the high processing cost of metadata manipulation. DocMan seeks to alleviate this problem by moving away from a proprietary flat-file storage mechanism toward a more standard, well-known approach. The use of a relational database provides the alternative medium in which to hold the metadata, and universal acceptance of this type of storage supplies the desired degree of freedom from a proprietary access method. For BEPS, any part of the application that needs access to a document’s metadata has to be aware of the proprietary data structure in which it is stored. Concepts such as concurrency control, transactional integrity and scalability are not handled well by BEPS. Concurrency is not a major problem because the design of the product presents few if any concurrency concerns, but this is a limiting factor for future expansion. Transactional integrity and scalability are more immediate concerns because they can limit the capability of the system to process documents at high volumes. Most relational DMS provide these features and many more. In addition, the common interface to query and manipulate the database provides many more avenues for a system of this type. If it becomes desirable to integrate a 3rd-party application into the overall system that accesses the database, the designers of the application would only need to

PAGE 65

57 know about the underlying database schema. Separate applications are not even required to communicate with the database in the same way. If one wants to use Java with a JDBC driver and the other uses C++ with ODBC, that would be perfectly acceptable, assuming adequate drivers are available for the DMS. Data abstractions techniques can also be employed. One application can access the database through standard SQL calls while the other uses a data encapsulation scheme such as Enterprise JavaBeans™ (EJB), and neither would know the difference. The possibilities go on and on. Using a relational database to store the metadata has opened up a plethora of options for the system designer. Accessibility and scalability have been greatly increased, and the benefits of this approach are demonstrated by DocMan. Flexible with 3rd-party metadata schemes The integration of different computer systems or applications can present significant hurdles, not least of which is the differences between data storage schemes. However, the magnitude of such problems is greatly reduced if the systems both subscribe to a common storage mechanism such as a relational database. The ubiquity with which this type of system is deployed in today’s market makes it even more important to a design that may be integrated at some point. Suppose two document management systems each have a set of metadata included for each document. If the intersection of these metadata sets is empty, it would be unlikely that they would ever be integrated; if the metadata for each document type has no common characteristics, why would they ever be commingled in a single application? However, if there is some overlap, conflicts need to be resolved. Analyzing well-known database schema allows an integration engineer to spot problem areas and create

PAGE 66

58 solutions. DocMan presents just such a database schema (albeit a limited one) and would facilitate integration. As far as approaches to actually integrate two systems, DocMan suggests a few methods. Schema conversion can be performed using XSLT in much the same way the XML-to-HTML conversion was performed. The contents of one system or the other (or both) could be ported to the new common format. Alternatively, additions to the existing schema in the relational database could be created and the new content could be added in that manner. These methods may be appropriate if the two systems are simple or relatively similar. The exact method would be dictated by the systems involved in the integration. It should be noted here that the actual process of schema integration is usually quite complex. The level of difficulty is directly (if not exponentially) proportional to the number of systems involved in the integration, and this area is beyond the scope of this thesis. DocMan employs a relational database storage mechanism, which is a common way of storing data, but it does not provide any facilities to resolve conflicts between metadata schemas. The benefit comes from the fact that there is a relatively small learning curve associated with the well-known storage platform and the increased possibility of a shared platform type between systems. DocMan has an open design using a well-known storage system. If integration requirements present themselves, the system can be modified to accept them without a major redesign. Handling changes to the DMS system The previous section discusses the ramifications of integrating 3rd-party data into an existing system. But what about changes or additions to the feature set of the DMS

PAGE 67

59 itself? If a system has tightly coupled components, this could lead to unnecessary restrictions when adding improvements or simply fixing bugs. Utilizing an open design with loosely-coupled components separated by well-defined interfaces supports future expansion of a software system. DocMan has several interfaces in place that separate the various processing modules. At the lowest level is the database itself, which provides access to the data through SQL queries. Data abstracted behind the SQL interface hides the details of the dataÂ’s storage format from higher-level layers that need not be concerned with it. Further, when data are returned from a SQL query, they are immediately encapsulated in the well-defined XML schema. This allows other parts of the application to interact with the data in a standard way. Changes to the database access method or the database itself need not affect other parts of the system. As long as the XML schema stays the same, the other modules cannot tell that anything has changed. Another abstraction is created by the XSLT processing applied to the XML buffers to produce document and content views. The XML simply describes the data; it does not dictate the view. Changing the XSL templates produces different views of the data but does not affect the data itself or the way it is stored. This disconnection is beneficial when changes or additions to the system are made. In summary, it is evident that metadata stored in a relational database provides many more options both for data manipulation and system flexibility.

PAGE 68

60 LIST OF REFERENCES [HUN01] Hunter, Jason and William Crawford. Java Servlet Programming : Second Edition. Sebastopol, California: OÂ’Reilly & Associates, 2001. [KAY00] Kay, Michael. XSLT: ProgrammerÂ’s Reference . Chicago, Illinois: Wrox Press, 2000. [MAR00] Martin, Didier, Mark Birbeck, Michael Kay, Brian Loesgen, Jon Pinnock, Steven Livingstone, Peter Stark, Kevin Williams, Richard Anderson, Stephen Mohr, David Baliles, Bruce Peat, Nikola Ozu. Professional XML . Chicago, Illinois: Wrox Press, 2000. [NEB95] Nebel, E. and L. Masinter. RFC 1867: Form-based File Upload in HTML . El Segundo, California: Xerox Corporation, 1995. [WYA01] Wyatt, Nat and Ames Carlson. Cloudscape 3.6: A Technical Overview . Oakland, California: Informix Software, Inc., 2001.

PAGE 69

61 BIOGRAPHICAL SKETCH Brennan Folmer was born in Boynton Beach, Florida, on July 4, 1976. He lived in W e st Palm B each, F l orida , for the f i rst 17 y e ars o f his life, after wh i c h he m oved to Gainesville, Florida. He completed his high school studies at Forest Hill High School in West Palm Beach in 1994. He received a dual bachelor’s degree in computer engineering and electrical engineering with Highest Honors from the University of Florida in December 1998. Brennan started working at Barr Systems, Inc., part-time in January 1998 as an Associate Support Analyst while completing his undergraduate studies. Upon graduation in December 1998, Brennan transferred to the Development Department as a Systems Programmer and began work as a team member of the Barr Enterprise Print Server project. While working full-time, Brennan pursued a Master of Engineering degree in c omputer s cience f r om the Universi t y of F l orida. After graduation, Brennan plans to continue to work as a programmer in an industrial environment. He looks forward to learning new skills and working with new technologies.