Integrative and Interactive Approach to Three-Dimensional Programming

Permanent Link: http://ufdc.ufl.edu/UFE0015642/00001

Material Information

Title: Integrative and Interactive Approach to Three-Dimensional Programming
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0015642:00001

Permanent Link: http://ufdc.ufl.edu/UFE0015642/00001

Material Information

Title: Integrative and Interactive Approach to Three-Dimensional Programming
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0015642:00001

This item has the following downloads:

Full Text







Copyright 2006


Hyunju Shim

To my parents and brothers


I would like to express my deepest gratitude to my advisor, Dr. Paul A.

Fishwick, who gave me inspiration and guidance throughout my Ph.D. studies at

the University of Florida. I would also like to give my sincere appreciation to my

Ph.D. committee members, Dr. Abdelsalam A. Helal, Dr. Be:i ,i:niii C. Lok, Dr.

Howard Beck, and Dr. Joachim Hammer, for their precious time and advice for my

research. Special thanks go to Peter J. Dobbins and Jonathan Ohlrich for helping

me to successfully finish the empirical study with their classes. I appreciate all the

colleagues in our research group for sharing valuable ideas. Also, I am grateful to

the National Science Foundation and the Air Force Research Laboratory for their

financial support for my studies. I owe great love to my parents, who encouraged

me throughout my studies and my life.


ACKNOWLEDGMENTS ................... ...... iv

LIST OF TABLES ................... .......... viii

LIST OF FIGURES ..................... ......... ix

ABSTRACT ... .. .. .. ... .. .. .. .. .. .. ... .. .. xii


1 INTRODUCTION .................... ....... 1

1.1 Motivations and ('C! 11, i~; ...................... 1
1.2 Contributions to Knowledge .. ...... ............. 5
1.2.1 Flexible Program Visualization ................ 5
1.2.2 Graphics Approach to Graphics Programming ........ 6
1.2.3 Integrative and Interactive Approach to Programming and
M odeling ............ . . .... 7
1.2.4 Three-dimensional Execution and Debugging . . 9
1.2.5 Evaluation of Visual and Textual Language Notations 9
1.3 Organization of This Dissertation ................. .. 10

2 RELATED WORK ....... ......... ............. 13

2.1 Visual Programming ............... ...... 13
2.2 Metaphorical Software Visualization ................ .. 17
2.3 Differences between Related and Our Work . . 21

VIA DYNAMIC MODELING .................. .... .. 22

3.1 Flexible Approach to Visualization ................. 23
3.1.1 Metaphors in Visualization ................ 23
3.1.2 Aesthetics in Visualization ................. .. 24
3.2 Modeling Programs via Dynamic Models . . ...... 26
3.2.1 Programming versus Modeling ............... .. 26
3.2.2 Dynamic Modeling of Programming Principles ...... ..29
3.3 RUBE Framework .................. ........ 33
3.3.1 Framework Structure ................ .. .. 33
3.3.2 Model Execution in RUBE ................. .. 39

3.4 Modeling Graphics Program .
3.4.1 Requirement Specification
3.4.2 Design .. .........
3.4.3 Implementation .. ..
3.4.4 Execution and Validation
3.5 Modeling State-based Program .


4.1 Modeling Predator-prey
4.1.1 Causal Graph .
4.1.2 Flow Graph .
4.1.3 Simulation .
4.2 Model Visualization .

4.2.1 Model Integration
4.2.2 Model Interaction a
4.3 Discussion .. ......

. . ... . 5 3

system .........

nd Execution .....

TO VISUAL PROGRAMMING .. ...................

5.1 Conceptual Language Description .. ...............
5.1.1 Language Requirements .. ................
5.1.2 Computational Model .. .................
5.1.3 Language Constructors .. ................
5.1.4 Layouts and Programs .. .................
5.1.5 Internal Representation .. ................
5.2 VPTinker Programming .. ....................
5.3 Integrative and Interactive Environment .. ............
5.3.1 Integrative Visualization .. .................
5.3.2 Interaction and Debugging .. ..............
5.4 D discussion . . . . . . . .


6.1 Designing Textual Language .. .................
6.1.1 Mapping between Visual and Textual Languages ......
6.1.2 Tinkertoy Textual Language .. .............. Language constructors and programs ........ Program execution .. ...............
6.2 Em pirical Study . . . . . . .
6.2.1 Hypothesis and Assumptions .. ..............
6.2.2 Experimental Design .. ..................
6.2.3 Experimental Results .. ................. Test result . . . . . . Survey . . . . . . .


6.2.4 Empirical Study Discussion .

7 CONCLUSION ............... ............ ..114

7.1 Summary ............... ............ ..114
7.2 Further Research. .................... ........ ..117


A EMPIRICAL STUDY MATERIALS ........... .. ... .120


B.1 Test Group ............... ......... 128
B.1.1 Material. .................... ......... 128
B.1.2 Quantitative Results ............. .... . 129
B.1.3 Qualitative Results ................ ... 129
B.1.3.1 Coding. .................. .... .. 129
B.1.3.2 Quotes .................. .... 129
B.2 Controls group .................. .......... 133
B.2.1 Material. .................. ........ 133
B.2.2 Quantitative Results ............. .... . 133
B.2.3 Qualitative Results ................ ... 134
B.2.3.1 Coding. .................. .... .. 134
B.2.3.2 Quotes .................. .... 134

REFERENCES ........................ . . 137

BIOGRAPHICAL SKETCH .................. ......... 147


Table page

4-1 Algorithm for converting a causal graph into a set of differential equations 62

6-1 Grammars for Tinkertoy Construction Language ............ ..98

6-2 T-test: Two-sample Assuming Unequal Variances . ..... 106

6-3 Responses for the Multiple-choice Questions on the Survey ...... ..107

6-4 Responses for the VPTinker Programming . . .... 108

6-5 Responses for the Textual Programming ..... . . 108

B-1 Results for the Multiple-choice Questions on the Survey . ... 129

B-2 Responses for the VPTinker Programming . . .... 129

B-3 Results for the Multiple-choice Questions on the Survey . ... 133

B-4 Responses for the Textual Programming . . ..... 134


Figure page

3-1 Aesthetical rendering of program icons ............. .. .. 26

3-2 Data-flow between blocks .................. ....... .. 29

3-3 Dynamic model components for conditional branches . .... 31

3-4 Parallelism in functional block model (FBM) . . ..... 33

3-5 The RUBE programming APIs and the RUBE framework . ... 34

3-6 Snapshot of the blender interface showing a simple program model 35

3-7 MXL representation of the program model in Figure 3-6 . ... 36

3-8 Example of the user-defined Python script code . . 38

3-9 FBM for the bowling alley generation .................. 42

3-10 Layout design for a 3D integrated model ................ .. 43

3-11 Graphics programming icons ............... .... 44

3-12 Snapshot of connecting icons from the blender interface . ... 45

3-13 Program model for the bowling alley using a factory metaphor . 46

3-14 Snapshots of program model for the bowling alley using a factory metaphor
from different angles: a) top view, b) right side view, c) left side view,
and d) close-up view of the bowling alley .... . . 47

3-15 3D bowling alleys generated from the program model in Figure 3-13
with different parameters .................. ....... .. 47

3-16 Algorithm and Finite State Machine(FSM) for comment scanner . 48

3-17 RUBE FSM diagram for comment scanner ............... ..49

3-18 Icons for comment scanner .................. ..... .. 50

3-19 Program model for comment scanner using a chemistry metaphor . 51

3-20 3D graphics generated from the program model in Figure 319 . 52

4-1 Causal graph for rabbit and fox populations . . ..... 55

4-2 Causal graph for fox-rabbit predation .............. .. 56

4-3 Causal graph for the prey-predator model ................ ..57

4-4 Augmented causal graph .................. ....... .. 58

4-5 System dynamics flow graph symbols .................. 59

4-6 System dynamics flow graph for the causal graph in Figure 44 . 60

4-7 Vensim simulation result of the predator-prey model . .... 61

4-8 System dynamics 3D symbols: From left, level, rate, cause-and-effect
arc, and flow arc .................. ........... 64

4-9 3D representations of the predator-prey flow graph .......... ..65

4-10 3D geometry models for the rabbit and fox ............... ..65

4-11 Control panel for model simulation ................ 66

4-12 Integrated model for the predator-prey system ............. ..67

4-13 Model component identification through highlighting . .... 69

4-14 Setting simulation variables ................ ..... 70

4-15 Snapshots of the integrative modeling and simulation . .... 71

5-1 Hole positions on spool ............... ....... .. 75

5-2 Simple Tinkertoy model ............... ....... .. 76

5-3 Data-flow diagram for the construction of the model in Figure 5-2 77

5-4 Icons for Tinkertoy model programming ............. .. 78

5-5 Icons for connecting positions ............... .... 80

5-6 Iconic program model for the model in Figure 5-2 ......... 81

5-7 Tinkertoy model visual program ................ ...... 81

5-8 Example of visual program and internal tree representation ...... ..82

5-9 Tinkertoy programming environment .............. .. .. 84

5-10 Main control panel ............... ......... .. 84

5-11 Snapshots for interface usages: From left, setting trace, setting control
point, and setting break point .................. .. 85

5-12 Model integration for the Tinkertoy program . ..... 87

5-13 Executing Tinkertoy program with breakpoints ............. .91

6-1 Mapping diagram between conceptual, visual, and textual languages .97

6-2 Mapping between visual and textual language constructors ...... ..99

6-3 Textual program model for Figure 5-2 .................. 99

6-4 Tinkertoy model textual program .................. .. 99

6-5 Tinkertoy model textual program 2 ................ 100

6-6 Test materials .................. ............. 104

6-7 Graphs for the scores of subjects .................. ..... 105

A-l Test model 1 .................. ............ .. 120

A-2 Textual program for test model 1 .................. .. 120

A-3 Visual program for test model 1 .................. ..... 121

A-4 Integrated model for test model 1 .................. .. 121

A-5 Test model 2 ...... ............. ............ 122

A-6 Textual program for test model 2 .................. .. 122

A-7 Visual program for test model 2 .................. ..... 123

A-8 Integrated model for test model 2 .................. .. 123

A-9 Test model 3 .................. ....... ...... 124

A-10 Textual program for test model 3 .................. .. 124

A-11 Visual program for test model 3 .................. ..... 125

A-12 Integrated model for test model 3 .................. .. 125

A-13 Test model 4 .................. ............ .. 126

A-14 Textual program for test model 4 .................. .. 126

A-15 Visual program for test model 4 .................. ..... 127

A-16 Integrated model for test model 4 .................. .. 127

Abstract of Dissertation Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Doctor of Philosophy



Hyunju Shim

August 2006

C'!h Ii: Paul A. Fishwick
Major Department: Computer and Information Science and Engineering

This dissertation introduces an integrative and interactive environment for

three-dimensional (3D) programs. In this environment, users can lend themselves

to the creative activity in designing their own icons for the basic building blocks

of programs, and they can also wire them together to construct a program. This

work is viewed as providing a user-customized method of programming while

leveraging the ability to blend both program syntax and semantics within the

same virtual space. The integrative approach has been broadened to simulation

modeling, especially for system dynamics modeling. A new method for modeling

the particular type of a predator-prey system is demonstrated by leveraging the

power of human interaction, as applied to two different models: a system dynamics

model and the geometry model. Both models are represented in three dimensions

and are fully integrated so that interacting with one model's components provides

context for the other model's components through a method of highlighting. With

the integrative modeling approach, the linkages between the related objects from

different model types are visualized by effective interaction means.

As a way to leverage the integrative and interactive environment for visual

programming, a simple visual programming language for constructing 3D graphics,

Visual Programming Tinker (VPTinker), was developed. VPTinker is a data-flow

visual programming language for novices to construct virtual Tinkertoy models. It

is an iconic language using a factory metaphor representation, and its programming

is achieved by wiring the icons together. The integrative environment allows

the output of the VPTinker program to be integrated with the source program

within the same virtual space. This approach to graphics programming enables a

programmer to have a holistic view of the program syntax, the source program,

the semantics, and the program output. In the VPTinker environment, a program

is arranged within the two-dimensional (2D) plane, and the program output

is constructed in the 3D space. New interaction mechanisms among the source

program, the output model, and the programmer have also been designed. The

user interactions include simultaneous highlighting of related components between

the source program and the program output; progressive animation for program

execution and its output construction; the pl ,i pause-and-stop execution modes;

and adjustable speed of the execution and animation. The VPTinker environment

also includes a novel debugging feature, such as setting 3D breakpoints among the

program components for the step-by-step execution.

Classroom assessments were performed in the Spring and Summer 2006

semesters at the University of Florida to evaluate the effect of visual programming

on the problem-solving capability for graphics programming. To evaluate the

graphics programming in VPTinker by comparison, the textual counterpart

language was designed in such a way that it is mapped one-to-one with VPTinker.

The empirical results showed that the subjects who used VPTinker received better

grades than those who used the textual counterpart language. The empirical results

were analyzed based on the cognitive dimensions of visual and textual notations.


1.1 Motivations and Challenges

Visual programming refers to programming that makes use of visual means

in the process of programming. The goal of visual programming is to improve

the problem-solving capability in programming by replacing traditional textual

notations in programming languages with visual ones. Ever since the introduction

of the seminal research in visual programming [1, 2], much research has been

conducted regarding the generation of effective visual representations and tools

for different programming paradigms and tasks [3-5]. Until the late 1980s,

the prevailing techniques among visual programming communities were in

two dimensions [6-8]. It is only recently that three-dimensional (3D) visual

programming was introduced and gained interest [9-12]. Most research in visual

programming focuses on easiness, efficiency, and improvement of programming.

The success of visual programming languages (VPLs), as a general programming

language, is controversial, and empirical studies on the usability test of VPLs

have shown conflicting results [13-17]. Unlike general purpose programming

languages, VPLs have been successful as domain- p ... ''.:' 1,.tliwlt, from various

areas [18-21]. However, leveraging the 3D graphics in the programming process

itself for graphics programming has not been studied widely, and it has remained

an interesting question. Specifically, this research was initiated with the following

five motivations:

* How can creativity increase peoples interest and help their learning of

* Why not leverage 3D computer graphics in the programming process?

* Can a 3D program be visually integrated with its output 3D graphics?

* How can the user leverage 3D execution and debugging for 3D programs?

* Would the visual approach improve problem-solving in graphics programming?

The first motivation is about customizing visual expressions in visual

programming. In designing icons to represent language constructors in visual

programming, primitive objects are commonly used as language expressions. The

layouts are optimized in such a way to minimize line crossings and maximize

symmetry. There has been little freedom for programmers to customize the

appearance of visual expressions, and issues such as stimulation of the user's

creativity and aesthetic perspectives of visualization are often neglected among

visual programming communities [22]. As computational technologies diffuse into

our everydwi lives and aesthetics are discussed as equally important as usability,

the need to consider aesthetics in the design of interactive systems increases [23].

Recently, within the Human-Computer Interaction (HCI) community, an increased

role has arisen in the importance of properly balancing utility with aesthetics.

Aesthetics are now discussed as a distinguishable, measurable construct in the

field of HCI [24, 25]. This dissertation introduces 3D customizable icons in visual

programming to stimulate the user's creativity and aesthetic perspectives. This

approach to programming has particular utility in education where creativity has

been shown to increase subject interest.

Besides the customizable approach to 3D icons, as a second motivation,

we were interested in a visual approach to graphics programming. That is, an

input program and its execution output are in the form of 3D graphics. The

programming code to generate computer graphics artifacts usually involves learning

and coding in a language such as C++ or Java. In the creation of computer

graphics programs, the underlying application programming interfaces (APIs),

such as OpenGL [26, 27] and Java3D [28], provide a set of textual statements.

These statements allow the specification of geometric objects using the provided

primitives, together with a set of commands that control how these objects are

rendered. For generating complex 3D computer graphics, authoring tools such

as Maya, 3D Max, and Blender are used. These approaches are effective, but the

question remains why we are not leveraging the power of 3D computer graphics in

the ] *ji',, Iti,,i,:j process itself.

Having an input program and its output as 3D graphics, our third motivation

was about creating an integrative and interactive environment in which both the

input program and the output graphics co-exist within the same space. A source

program and the execution are two different perspectives of problem-solving. Each

of these perspectives describes a static and dynamic perspective, respectively.

While they are semantically tightly coupled entities, little research has been

conducted from visual programming communities about surfacing the relationships

between the program source and its output in visualization forms. From the HCI

community, several techniques have been introduced to assist comprehension of a

complex system that has more than one conceptual perspective. Multiple views are

a technique that uses more than two distinct views to support the investigation of

a single information system [29]. Several techniques, such as linking and brushing,

are introduced for displaying multidimensional information using multiple views

[30-35]. Multiple views allow viewers to focus on partial information while they

are linked so that the information contained in individual views can be integrated

into a coherent view of the information as a whole. In the computer simulation

community, various model types have been developed and introduced to describe

different aspects of these systems. A system can be modeled by views based on

its logic, dynamics, or physical appearance. Recently, increasing interest has been

shown in integrating different model types of simulation for visualization [36, 37].

The integrative environment allows programmers to have a holistic view on both

the program source and its execution output, and also to experience special

types of interaction between the program source, the execution output, and the


As a fourth motivation, we were interested in creating novel interfaces for

program execution and debugging that leverage an integrative approach. When

writing a complex program, programmers are likely to make syntactic or semantic

mistakes. Debu. t.-i assist programmers to identify and resolve program bugs with

such features as single -I' pp11 n-stepping, trace, breakpoints, and watchpoints.

According to Johnson and Kenney [38] and Johnson [39], a debui-i. vr should

provide more than just information. It should be easy and it should allow users

to focus on examining a program itself rather than controlling the debiu'- vr. User

interaction should be minimized to request information, and the format of reported

data should be quickly interpretable. Using debui i -is, programmers can easily

monitor the status and values of variables and memory, and they can run the

program instruction-by-instruction or by a group of instructions.

With textual languages, most debugging interfaces are in two dimensions.

Interactions with such debiu -ir are provided by streams of strings, menu bars,

drop-down menus, and pop-up windows. Information related to debugging is

delivered in the textual form usually provided by a separate console or window.

Three-dimensional debugging of 3D programs enhances user interaction via

leveraging motions and animation, which are fundamental features of 3D programs.

Recently, increasing research has been conducted in leveraging 3D animation for

illustrating the execution status of programs [10, 12, 40, 41]. With the integrative

environment, programmers construct, execute, examine, and debug 3D programs

within the environment where the program source and the execution output are

presented together. This integrative environment can be further broadened for the


simulation model types in such a way that different model types of the same system

coexist within the same space.

Finally, as a fifth motivation, we were concerned about whether or not

programming for computer graphics (i.e., programs, when executed, would create

geometry and rendered scenes) would be improved by a visual approach. Computer

programs are very complex systems, and learning programming for problem solving

is subject to the characteristics of the language in which the solution is described

[42, 43]. For the same programming task, a programmer constructs different mental

models with different programming languages [44, 45]. In [14], it is shown that

the effectiveness of language is task-dependent VPLs may be easier to learn and

understand than textual languages. The evaluation of a visual programming system

is crucial, but the usability test of visual programming is not trivial. One way of

evaluating visual programming is to compare it with a traditional approach, which

is programming via a textual language. We were interested in investigating the

problem solving capability in visual programming for graphics programming by

comparing a visual language to its textual counterpart language.

1.2 Contributions to Knowledge

1.2.1 Flexible Program Visualization

Until recently, due to the high cost graphics hardware, visual expressions

in visual programming language were adhered to primitive objects. Qualitative

perspectives in visualization, such as aesthetics, creativity, and desirability,

therefore, have been ignored from the visual programming community. However,

with tod wi's low cost and high performance graphics resources, the user's

creativity and aesthetic perspectives are becoming new features in visualization.

Three-dimensional representations provide a greater number of degrees of freedom

not present in two dimensions, or at least extending what is available in two

dimensions for aesthetics and expression. The degrees of freedom for three

dimensions include one extra positional dimension; two additional degrees of

rotation; an addition of presence and immersion; an increased engagement, which is

the ability to interact with a scene; an ability to texture and add rendering effects;

and a creativity "index," that is, the ability to extend the creative potential beyond

two dimensions.

This research introduces a customized approach to visual programming that

enables novices to learn programming through a high degree of creativity by

expanding the aesthetic range of program representation. We express this creativity

in our program constructors through the use of customized 3D icons. The broader

view encapsulates two ideas: form and function have equal footing and we should

strive beyond pure optimal functionality toward environments that balance look

and feel with pure usability. With this approach, users are exposed not only to the

programming environment in which they can wire basic building blocks together

for programming, but also can design their own icons, which lend themselves to

the creative activity. A framework has been developed for programmers to leverage

their own designed icons in programming. In this framework, dynamic model types,

such as the Functional Block Model (FBM) and Finite State Machine (FSM), are

applied in the construction and execution of programs to capture the nature of

programming principles.

1.2.2 Graphics Approach to Graphics Programming

For programming tasks that require spatial reasoning, a visual approach

provides close mappings between the problem domain and the program domain,

therefore enhancing problem solving capabilities. A visual approach allows

programmers to focus better on the flow of data and spatial reasoning than

sequential and textual approaches allow. This dissertation introduces a 3D

visual programming language, Visual Programming Tinker (VPTinker), for

computer graphics programming that leverages the data-flow paradigm and a

factory metaphor for its conceptual model and representation. The VPTinker

language programming applies a visual programming approach to learning

and practicing data-flow graphics programs. The use of a factory metaphor in

program visualization naturally and intuitively reveals the concepts of a data-flow

paradigm and surfaces it in the program configuration. The use of a factory

metaphor naturally lends VPTinker programs to the 2.5-dimensional (2.5D) layout

where icons are laid on the floor. The limited movement of 2.5 dimensions is less

confusing, and it provides intuitive mapping to two-dimensional (2D) interaction

devices, such as a mouse. One advantage of the 2.5D layout for VPTinker is that

novices can easily grab and move icons using a mouse while freely navigating the

3D program space without special training. Using this language, the construction

of a Tinkertoy model is represented as a data flow network. The visual approach

to data-flow programming surfaces the flow of data in the program's configuration,

which makes it easy to learn and easy to practice data-flow paradigms.

1.2.3 Integrative and Interactive Approach to Programming and

In this dissertation, we introduced an integrative approach to visualization

where interrelated systems are presented within the same virtual space. An

integrative environment for VPTinker programs blends both program syntax and

semantics within the same virtual space, and it enhances the interaction among

the source programs, the program execution outputs, and the programmer. Within

this environment, an approach to graphics programming begins with the user

constructing a 3D graphics program. Execution of the program generates 3D

animated graphics, which are juxtaposed with the input program. The integrative

approach allows programmers to navigate the scene that contains both the program

source and the output and to view the scene from various viewpoints. With

traditional graphics programming approaches, the source program and the output

graphics are two distinctive entities. With the integrative approach, the program

source and the output graphics can be treated as coexisting entities. Using the

integrative environment, programmers can watch the output graphics being built

as the input program is executed and animated. This si r-.- I 1 mode of operation

integrates syntax with semantics, where semantics is loosely specified to be

equivalent to l',' -'Ja" behavior. As the program execution progresses, semantically

related objects from the program source and the output are simultaneously

highlighted. This type of visualization provides a holistic view of program

execution. With the integrative approach, the power of human interaction is

also applied to the program source and the output in such a way that interaction

with one model's components provides context for the other model's components

through a method of visual highlighting. The dual interaction between both

models is a form of integrative modeling where interrelationships among model

components are surfaced in the user interface. This dual interaction allows free

form exploration of both models and their equivalences. With dynamic scenes that

offer users interaction and response, users become more immersed in experiencing a

scene. In visualization, interactions help users to comprehend abstract concepts or

models. The interactive visualization can also be used to reveal hidden information

on users' demand or to stimulate users' interest, which is an important factor in


This dissertation further leverages the integrative and interactive environment

for simulation model types. With the integrative approach, different model types

coexist and they are linked in the same virtual space. The integrative visualization

reduces efforts for context switch between different model types. The explicit

component identification for related model elements supports the understanding of

the mappings between different model types presented in the visualization.

1.2.4 Three-dimensional Execution and Debugging

An intelligent learning environment has been developed for programming

and debugging 3D programs. Within this environment, 3D programs are

progressively constructed and interactively examined in 3D space. The status

of the partly or completely executed program is demonstrated by visual means

or animation. Programmers are allowed to place metaphoric walls among the

program constructors to set breakpoints. Animation of program execution

allows programmers to visually trace the execution of a program for each

breakpoint. Within the integrative environment, the related components from

the source program and the execution output are simultaneously highlighted on

a programmer's demand. Debugging is enhanced by pl li pause-stop execution

modes that let programmers stop the execution and resume the execution for

inspecting partly constructed execution outputs. Combined with component

identification, setting breakpoints and the pl il pause-and-stop execution modes

allow programmers to check a program fragment easily at any point of program

execution. This type of the progressive debugging environment is especially

beneficial to novices. A programmer's control over the program execution

is amplified via adjustable speed of program execution. Three-dimensional

progressive debugging makes it practical to: monitor the program execution

intuitively; do early testing and demonstration; and make corrections sooner in the

software development cycle.

1.2.5 Evaluation of Visual and Textual Language Notations

One of the main interests of visual programming languages is the evaluation

of their problem-solving capabilities. One way to evaluate this is to compare

it with the traditional approach, which is the problem-solving capability via

a textual language. For the evaluation of VPTinker by comparison, a textual

counterpart language is designed in such a way that it is mapped one-to-one

with the iconic language. Designing the textual language starts by specifying the

mapping criteria between the visual and textual languages. Since the focus of

designing the textual language is for comparison with VPTinker, the designing

mappings between them are crucial issues. For an accurate comparison, they

must provide the identical capabilities for problem solving, except for their

notations. Everything must therefore be identical except that the syntax for one

language is in visual and the syntax for the other language is in textual. For

complete mapping between VPTinker and the textual language, a formalism must

occur for the mapping transformation process. The mapping criteria between

VPTinker and its textual counterpart language are that the mapping should

be unambiguous, complete, consistent, and equivalent. Classroom assessments

were performed in the Spring and Summer 2006 semesters at the University of

Florida to evaluate the effect of VPTinker programming on the problem solving

capability for geometry manipulation problems against its textual version of

language. The empirical results showed that the subjects who used VPTinker

received better grades than those who used the textual counterpart language.

The empirical results are analyzed based on the cognitive dimensions (CDs) of

visual and textual notations. Based on the CD analysis, visual notations fit more

closely to the mental representations than textual ones for the tasks that require

spatial reasoning, and visual programming would better improve problem solving

in graphics programming than textual programming.

1.3 Organization of This Dissertation

Two foci of this dissertation are 1) customization in model and software

visualization and 2) the integrative approach to 3D modeling and programming.

Previously, a Web and XML based framework, the RUBE framework, was

developed by former members of our research group. The RUBE framework

provides a necessary environment for the construction, visualization, and execution

of simulation models. We expanded the RUBE framework as a programming

environment. The RUBE programming environment allows users to lend themselves

to the creative activity in designing their own icons to represent the basic building

blocks of programs. Users can also construct a program as a data-flow model

or as a state-transition model using their own designed icons. Customization in

visualization allows users to leverage metaphors and to expand the aesthetic range,

which can stimulate their interest and productivity in programming.

The second focus of this dissertation is on the integrative and interactive

approach to 3D modeling and programming. The main concern with the integrative

approach is to blend multiple visualizations, which describe different perspectives

of the same system, within the same space. The integrative approach provides a

holistic view and intuitive interactions between the multiple visualizations. We

applied the integrative approach to system dynamics modeling, which is a special

type of simulation modeling, and to the 3D programming for graphics applications.

In modeling, different perspectives of the same system are different model types,

such as the dynamic and geometry models. In programming, different perspectives

are the source program and its execution. Within the integrative environment,

semantically related components from multiple visualizations are fully connected.

The relationships between the components are also surfaced by dual interactions

via the user interface.

The rest of this dissertation explains details about our approach to visual

programming. C'!i lpter 2 discusses related research. ('!i lpter 3 introduces the

customizable approach to software visualization. C'! lpter 4 presents the integrative

and interactive approach to simulation model types, especially for a system

dynamics model. C'! Ilpter 5 provides an integrative and interactive approach

to graphics programming using an iconic language, VPTinker. ('! Ilpter 6 offers

designing of a textual counterpart language of VPTinker and an empirical study


comparing the problem-solving capability of VPTinker with its textual counterpart

language. C'!i pter 7 concludes this dissertation with possible future research.


This chapter introduces related works from the area of visual programming

and software visualization using metaphors.

2.1 Visual Programming

ConMan [6] is a high-level visual programming language that lets users

dynamically build and modify graphics applications. By interactively connecting

simple components in ConMan, users construct a complete graphics application.

A data-flow metaphor is used in ConMan, and users construct and modify

applications by creating components that are interconnected via input and

output ports. Components that have been developed for the ConMan data-flow

environment include the sweep component to create a surface by sweeping; the tape

component for recoding; the render component that supports different rendering

qualities from wire-frame to ray tl iI ii.: the mixer component to interpolate

shapes and color; and the watch component for the inspection of data flowing

across the screen. Developers are encouraged to break monolithic applications

into functional components that communicate with each other using high level

data structures. The current application field of this research is the same as

ConMan, which is graphics programming. However, while ConMan is a 2D visual

programming environment that takes a data-flow metaphor, this research provides

a 3D programming environment deploying different principles from dynamic model

types and customizable icons.

Pictorial Janus (PJ) [7, 8] is a 2D visual programming language for parallel

logical programming. Programs in PJ are drawings in which the execution is

defined as the animation of the drawings. The basic elements of a PJ program

are closed contours and directed and undirected connections between them.

Geometrical representations of PJ objects, such as shapes, size, and color, have no

semantics. Objects of PJ are constants, list elements, links, functions, agents, and

rules. PJ programs are defined by combining PJ objects. Agents in the PJ program

communicate with each other via message passing. The behavior of the agent is

determined by the preconditions of the rules in case they match the corresponding

input patterns of the agent. Janus In Motion (JIM) provides interactive editing,

animating, and debugging environment for PJ programs.

Lingua Graphica [9] is a visualization of procedural textual languages. It

defines a visual 3D syntax for C++ programs that allow users to inspect and

modify the virtual reality simulation code without having to leave the virtual

environment. Graphical primitives used in Lingua Graphica are color, translucence,

shape, size, associative links, co-location, text, sound, and motion. Associative

links are used for representing class inheritance, data-flow binding, and function

calling or definition sequence. Co-location refers to the concepts of location,

containment, and intersection of one object with respect to another object.

Solid Agents in Motion (SAM) [10] is a visual 3D programming language,

visualization, and environment for parallel system. A SAM program is mainly given

as a set of 3D objects with no separate textual description. Three-dimensional

objects in SAM are 3D messages, agents with ports, and rules with a precondition

and a sequence of actions. Input and output ports are distinguished by the

direction of the representing cone. In SAM, the order by which rules are elaborated

is not defined. The execution of a SAM program is based on a synchronous

communication. The SAM execution cycle has two phases: agent execution and

communication. The execution of a SAM program can be observed by animation.

The SAM program allows users to build the abstract visual representations and

the corresponding concrete graphical representations for programs. In the concrete

representations, additional static 3D objects that are not part of the program can

be included as a background to give a more realistic environment.

3D-Visualan [11] is a 3D rewritable, rule-based programming language

in which both programs and data are expressed by 3D-bitmaps. Programs of

3D-Visualan are the ordered set of pattern-replacing rules. The priority of rules is

determined by their locations in the 3D-bitmaps, such as the rear rules which have

higher priority than the frontal rules. The behavior of the program is described by

means of before-after rules, which define how 3D-bitmaps change over time.

VisuaLinda [12] is a framework and a system which makes use of 3D

visualization to help the programmer debug parallel programs, and it also di- -'v1

both the relation between the Linda server and client and their execution. In this

framework, processes are laid out in an XY-plane and a Z-axis represents time.

The 3D layout of processes and time lets both the process-relation diagram and

the process-time diagram be visualized in the same 3D space. This integrated

visualization supports a simultaneous understanding of a process relation and a

time flow. Within the 3D space, it is also possible to display a large number of

processes and communication links between them without crossing in space.

Programming virtual object behavior in virtual reality Program (PiP) [46] is

a virtual environment system in which a user can create, modify, test, and save

object behaviors. The PiP program extends the visual programming approach to

3D multimodel programming to represent the behavior of a program or object by

manipulating it through the 3D space. A PiP virtual object model consists of its

forms, functions, and behaviors. The most typical behaviors of objects are specified

by demonstration. For example, to specify a condition that an object must be in a

certain position and orientation, the user simply manipulates and moves the object

into the desired configuration. Using PiP, users can program or specify the motion

or animation intuitively through a better understanding of direct programming in

3D space.

In Computer Animation Environment Language (CAEL) [47], visual

syntax-directed programming within a 3D computer animation context can take

place. Language grammar in CAEL augments a subset of Pascal with procedures.

Program representation subclasses utilize 2D depictions which are positioned in

3D space. In CAEL-3D, users are allowed to alter the appearance of the geometry

and/or color of the p" Iv.-.onal 3D objects, which brings a level of user-specifiability

into the representation. A user is free to create 2D and 3D graphical polygonal

elements for representing program constructs. The incorporated menu-based

interaction methodology and tools of the 3D computer animation system let users

act upon and maneuver through the programming constructs.

3D-PP [41, 48] applies the direct manipulation of operations to the 3D

program elements. 3D-PP is based on the concurrent logic programming language

Guarded Horn Clauses (GHC). The visual program of 3D-PP is composed of

a combination of hierarchical nesting boxes of pictorial programming elements,

such as atom, list, id_data, goal, and built-in goal. An extended drag-and-drop

technique is used for describing the program structure of nesting boxes. The

semi-transparent representation, using nesting level filtering and double-click

browsing, improves the spatial problem in visual programming caused by a small

screen. Debugging of a 3D-PP program is supported by animated execution.

Animated execution animates state transitions while execution of the program

proceeds. The animation is di-p1 .1, ,1 in a similar manner with the programs,

thus the programmer can easily read the animation. Three kinds of functions are

provided in combination with animated execution: 1) rewriting for speeding up

testing and modification of processes; 2) marking data for tracing data; and 3)

marking rules for narrowing suspicious implementation.

A Visual Programming Environment for Python (ViPEr) [49] is a Python and

Tkinter-based visual programming environment for composing networks of Python

scripts for molecular data processing and visualization. It allows non-programmers

to interactively build a network of computational streams and visualize their data

without writing code. In ViPEr, nodes are lightweight wrappers of functionality

available in Python and organized in libraries. Drag-and-drop adds nodes into the

graphical network editor. Input and output ports of nodes are connected using the

mouse to define a directed graph. This directed graph is used to propagate data

and execute nodes. Subnetworks can be encapsulated into macro nodes, allowing

nesting of networks. Tooltips and balloons provide runtime information about node

functions, inputs, and outputs. Data flowing through nodes can be interactively

monitored and introspected. Users can specify the appearance of the port icon

using different colors or shapes, which provide visual hints for connecting the

proper outputs with the proper inputs.

2.2 Metaphorical Software Visualization

The CyberNet [50, 51] project uses 3D metaphoric worlds to visualize the

network data. It was developed with adaptive navigation features that are

dependent on the 3D metaphor used for visualizing the information and on

the user's task. The network data are depicted metaphorically using the visual

parameters of the virtual world. Different metaphors are used to visualize the

information according to the type of the user's task. The choice of a metaphor

depends on the type of information that is relevant for that task; the best

mapping between the data values; and the metaphor's visual parameters and

user preferences. Metaphors such as a building, a city, a solar system, a cone

tree, or a landscape have been designed. The task-dependent navigation, called

metaphor-aware navigation, was approached. The metaphor-aware navigation

allows users to navigate in the virtual world with the mechanism most suited to

that particular metaphor.

The Information Cube [52] is a 3D visualization technique of hierarchical

information that uses 3D cubes. With the cube metaphor, information is

represented by a container or a box. XN -i 1 cubes represents hierarchical

information in such a way that the outermost cube represents the top level data

and the next level data are enclosed inside of the outermost cube. Each cube is

rendered in semi-transparent color so users can easily see the inside of the cubes.

The semi-transparent rendering also lets inner level boxes fc.--., '1 out gradually and

complex hierarchical information systems maintained reasonably.

Rilling and Mudur [53] introduced the use of metaballs to provide 3D visual

representations of software systems. Metaballs are a 3D object modeling technique

which blends and transforms an assembly of particles with associated shapes into

more complex 3D shapes. They are also called metablobs, soft objects, and point

clouds, and they are extensively used in representing complex organic shapes and

structural relationships in biology and chemistry. Rilling and Mudur applied the

metaball metaphor to visualize software entities and their dynamics. Particles in

a metaball metaphor are mapped into software structures, with blobs representing

an object or a function. The potential energy around blobs is mapped into the

strength of the coupling between entities. Dynamic aspects of software are visually

depicted by animating the appearance properties of blobs. By mapping software

entities or parameters in the software slices with metaball models, Rilling and

Mudu created a 3D virtual space where users can walk around for examination of

the model.

Panas et al. [54] introduced a 3D city metaphor in visualizing complex software

for the purpose of maintenance and reverse engineering. They mapped static and

dynamic properties of software to the city environment. The size of the buildings

represents the amount of lines of code, and the density of buildings shows the

amount of coupling between components. Structures and qualities of buildings

represent the quality of the system's implementations. Cars moving through a

city indicate a program run. Dense traffic indicates heavy communication between

various components. Speed and the type of cars represent performance and

priority. Car crushing shows program exceptions. In addition to these visualization

for reverse _,.-ir. iii: they applied a cost-focused metaphor over the 3D city

metaphor to visualize additional business-related information for different types

of stakeholders, such as developers, designers, vendors, and project managers. For

example, the components currently being modified are colored in yellow. Parts

of the system that are no longer needed are colored in brown. Frequently used

components are surrounded by fire. Frequently modified components are visualized

as a building with flashes.

The Zeugma system [55-57] is an environment for constructing and experiencing

Analogical Representation of Programs (ARP). Zeugma allows users to visualize

Lisp programs using analogy or metaphor, such as a solar system, cities, and

spiders. Given a structured schemata describing relations between program

analysis and representation features, Zeugma automatically constructs analogical

representation for the given programs. With the Zeugma system, it is possible

to switch from one representation to the other for programs. The combined and

simultaneous views of both the visualization and the control-flow view provide

links between program components and and their visualizations.

Knight and Munro [58] introduced the use of a world metaphor as a tangible

object for software visualization within a virtual reality environment. They

introduced the world metaphor in which the entire software is mapped into the

world, source files are mapped into cities, classes are mapped as districts, and

methods are mapped into buildings. Colors of the buildings are used to represent

access controls, such as public, protected, and private. The height of a building

represents the lines of code. Parameters of a method are represented by doors. To

assist the navigation and orientation of the user's visualization system, a central

garden and enclosing fence are included in the visualization, and block structures

are used for layout.

Hopkins and Fishwick [59] introduced an architectural and an agent-based

metaphor in the representation of an MINIX-like OS with four l. ri? of user

processes, server processes, input/outout (I/O) tasks, and process management.

In this representation, the operating system (OS) is represented by a building,

and each li. r of the OS is mapped into one floor of the building. Synthetic

human agents are used to represent each task. Message passing between the OS

components is represented by a transporting courier agent with a briefcase. The OS

devices are mapped into an agent standing behind a desk. Waiting lines of agents

are used to represent a device queue. To represent the dynamics of OS which can

be modeled by Finite State Machine (FSM), a dynamic traffic light metaphor is


Software Landscapes [60] visualizes the structure of large software systems

using the landscape metaphor. Three-dimensional landscape elements, customized

layouts, and hierarchical network structures are used to represent program entities,

their hierarchy, and relationships, respectively. The landscape metaphor, with 3D

objects on a 2D layout, facilitates navigation and comprehension of visualizations.

The hierarchy of software packages is represented by nested spheres. Spheres

contained within another sphere are arranged on a 2D circle. Transparency of

spheres are dynamically managed by the distance of the viewer. Spheres are opaque

when the viewer is very far and completely transparent when the viewer is very

close. Classes are represented by circular discs. Methods and attributes of a class

are represented by cuboids on the disc. For clarity, the 3D routing lines are used

to represent the relations between software entities according to their hierarchical


2.3 Differences between Related and Our Work

We can differentiate our work from related research as follows:

* Customizable approach to visual programming: Traditionally, expressions
in visual languages have been limited to primitive objects. Recently, several
research including [55-57] and CyberNet [50, 51], introduced the use of v ,'iiv
themes and metaphors in software visualization. In these systems, however, a
user's freedom in representation is still limited to the pre-existing themes and
metaphors. We introduced a customized approach to visual programming that
enables programmers to design their own icons to represent the basic building
blocks of their programs.

* Visual approach to graphics application: Conman [6] introduced a 2D
data-flow approach to 3D graphics construction. We introduced a 3D
data-flow approach to 3D graphics construction.

* Integrative visualization: In this work, we introduced an integrative approach
where the program source and its execution output co-exist and fully
connected within the same space. This visualization approach has not been
introduced to the visual programming community.

* Three-dimensional debugging of 3D programs: Unlike debugging textual
programs, debugging 3D programs has not been studied widely. Debugging 3D
programs, however, is getting more of interest. VisuaLinda [12] introduced 3D
debugging by visualizing erroneous components during the program execution.
Our work introduced pll i pause-and-stop execution modes and setting
breakpoints for 3D programs that allow programmers to check a program
fragment easily at any point of program execution.

* Progressive animation of program execution: Increasing research in visual
programming has leveraged animation to demonstrate the execution status
of programs [7, 8, 10, 41, 48]. Besides demonstrating the execution status,
we leveraged animation in such a way that the program output, which is 3D
graphics, is being constructed as the input program is being executed and


In this chapter, a visual programming environment, which facilitates a

customized 3D application programming interfaces (APIs) using dynamic model

types, is presented. The environment introduced in this chapter is developed for

users to lend themselves to the creative activity in designing their own icons for

the basic building blocks of programs and to wire them together to construct a

program. The environment also integrates the output graphics to be placed with

the 3D input program. RUBE, a Web and XML-based modeling and simulation

framework, provides a necessary environment for the construction, visualization,

and execution of program models. In the RUBE framework, users can construct

their program models, which will be stored in XML format and translated into

an executable program by the translation engines. The RUBE framework makes

use of a 3D tool, Blender [61], to provide a modeling environment. Users can use

pre-existing icons or create their own to represent program modules.

The unification of 3D dynamic model types with the program construction

brings several benefits in programming:

* Capturing dynamics of programs using simulation model types

* Use of metaphoric icons, which increases understandability and productivity of
novice programmers

* Use of metaphors to allow greater flexibility and freedom in model representation

* Stimulating user's creativity in the design of program icons

* Leveraging aesthetics aspects

* Juxtaposing program models with program outputs in the same space

3.1 Flexible Approach to Visualization

Designing notations of language constructors is one of the main issues in visual

programming. For software and information visualization systems, it is common

to use metaphors for the representation of abstract concepts [50-60]. This section

introduces a new approach to visual programming in which users are allowed to

create their own visual notations for language constructors.

3.1.1 Metaphors in Visualization

The word "metaphor" is derived from the Greek word iH i,-1i. ." The

metaphor's primary function is understanding new concepts from familiar ones.

The main functionality of metaphor is in its usability. Lakoff and Johnson [62]

demonstrated the pervasiveness of a metaphor in all aspects of human activities

not only as a matter of a language but also as a principal way of reasoning and

learning. A metaphor can benefit both a novice and an expert in some system.

A metaphor provides insights to novice users regarding the nature of a function

or application. A metaphor turns to a dead metaphor when it has occurred so

often and become a new meaning of the expression. Dead metaphors have a

communicative value for experts [63]. Visual programming can take advantage

of metaphoric icons, especially in their visual representations. Icons were first

introduced in programming when Smith constructed a special user interface for

operations in programming language, PYGMALION. He used icons to subsume

the notions of variables, reference, data structure, and functions [2]. Inspired

by Smith's work, Xerox introduced the 8010 "SI o1 Information System with

the desktop metaphor icons as a user interface [64]. Several research leveraging

metaphors in visualization were introduced in C'! lpter 2.

The conjunction of metaphors in visual programming could generate a

synergy effect on learning programming. The use of metaphoric icons, from

which their functionality can be visually referred to, would improve productivity

and understandability of programming for novice programmers. A i- r and

Masterson [65] claimed that the use of a factory metaphor for the data-flow

programming is simpler than flowcharts. Properly coupled with object oriented

concepts, a wider group of programmers-those whose minds more easily grasp

the factory metaphor-will appreciate visual programming. Some arguments exist

that metaphors should not be treated as a necessary criterion in designing visual

representations. Blackwell and Green [66] conducted an experiment about the

usability of metaphors in visual languages and claimed that metaphors do not

particularly assist visual languages.

Examples of metaphoric icons, which might be suitable to be used in visual

programming, include a warehouse metaphor for database applications where data

are stored and retrieved frequently; a plumber metaphor or a factory metaphor

for data-flow programming where data flow from one functional unit to another;

and a chemistry metaphor for state-based applications where the current state of a

system is determined based on the external or internal conditions.

3.1.2 Aesthetics in Visualization

In addition to usability, aesthetics is another dimension in visualization.

Aesthetics in visual programming has traditionally tended toward the manifestation

of optimality criteria, such as minimal line crossings and the efficiency of layout.

However, several recent publications included in [67] --.:: -1 that aesthetics for

products, including the visual programming products, should be defined to more

broadly cover both form and function or visual appearance and pure usability. This

more balanced approach is common in product design [68, 69], and since programs

are human products, such as houses and automobiles, we advocate extending

aesthetics in their visualization.

Fogarty et al. [23] claimed, as computational technologies move beyond

the confines of the work environment and into the rest of our lives, that the

need to consider aesthetics, or, more generally, desirability in the design of

interactive systems, will increase. Recently, within the HCI community, an

increased role has arisen in the importance of properly balancing utility with

aesthetics. A successful product is one that achieves both function as well as

form. The existence of aesthetics is discussed as a distinguishable, measurable

construct in the field of HCI, and it is advocated to better balance 1 ii' ', design

dimensions regarding usability and aesthetic considerations [24, 25]. The use of

personalized and customized metaphors in 3D visual programming and software

visualization can stimulate users' interest and creativity. This lets users tailor the

appearance of program components while preserving their functionality. The design

of personalized and customized 3D icons is one 1 ,---r where aesthetic aspects can

be integrated within software visualization. Three-dimensional representations

provide a greater number of degrees of freedom for aesthetics and expression [70].

Customization and personalization in HCI might be one solution to remove some

limitations in aesthetic designs caused by their dependency on personal experiences,

language, and culture. Another 1-,-,-r of aesthetics in software visualization is

applying different textures and rendering methods to the model constructors and

the model scene. Using metaphoric icons and adding graphical effects, such as

transparency and shading, we can build 3D dynamic systems or program models

that might be aesthetically expressed and appreciated.

Figure 3-1 shows the use of customized icons in a 3D program model

developed using the RUBE framework. Icons shown in this figure, from the

left-hand side, are Texture, Duplicate, Transform, and Di'plmv. Different materials,

textures, and rendering methods are applied in the creation of this figure. The

left figure in Figure 3-1 is a scene from a 3D program with colored materials and

Figure 3-1. Aesthetical rendering of program icons

different textures. The right figure shows the same program and metaphor with no

textures but with transparent materials and the wire frame rendering provided by


3.2 Modeling Programs via Dynamic Models

A framework, called RUBE, has been developed for programmers to leverage

their own designed icons in programming. In this framework, dynamic model

types such as the Functional Block Model (FBM) and Finite State Machine (FSM)

are applied in the construction and execution of programs to capture the nature

of programming principles. Before introducing the RUBE framework, we will

examine the principles of modeling and programming as a way to leverage modeling

techniques in programming.

3.2.1 Programming versus Modeling

Programming is the process of creating a computer program with instructions

that a computer can interpret to solve human problems. In general, programming

is classified into two categories, declarative programming and imperative programming,

depending on the model of computation. Programming languages are further

categorized into paradigms that include functional, data-flow, logic, Von Neumann,

and object-oriented [71]. Each paradigm reflects different perspectives of a model

of computations from which a programmer can conduct problem-solving in more

logical and efficient v--i.

Modeling in general refers to the creation of the representation for a certain

system which involves reasoning about the system. The goal of modeling is to come

up with a representation of a system that is easy to use for describing the system in

a mathematically consistent manner [72]. Following efforts to discover concepts and

representations of models, the first general purpose simulation languages, General

Purpose Simulation System (GPSS) [73] and SIMSCRIPT [74], were introduced in

the early 1960s. Ever since the early 1960s, concepts of modeling and simulation

have been reviewed to promote a more coherent representation and to capture more

logical and precise dynamic behaviors of a system. As the principles of modeling

and simulation are refined, the focus of simulation model development has shifted

from programming to modeling [75].

Although computer programming and modeling have gone in two different

directions, they share a common interest: problem-solving. However, while

problems in computer programming refer to general problems, those in computer

simulation are usually related to an activity about making decisions. For this

reason, we can -,iv that modeling is a particular way of problem-solving, and the

field of modeling and simulation has developed different computational models from

those of programming [76].

While the field of modeling and simulation departed from general purpose

programming as it matured, much research has also been done relating programming

with modeling to improve or supplement programming. SIMULA is the first

object-oriented language and a good example of how modeling and simulation

principles can improve programming. The concepts of class, subclass, and

inheritance in the object-oriented paradigm were first introduced in SIMULA.

Bloss [77] claimed that traditional imperative languages are poorly suited for

modeling the concurrent logic in simulation, and the absence of an explicit

time-flow mechanism in the functional programming would fit nicely into the

simulation modeling where time flow is an implicit part. Dynamic model types,

such as Petri Net and FBM with synchronous and .i-v chronous properties, are

more suitable to represent and practice logical and/or concurrent programs than

traditional imperative languages. Recently, enterprise level software is getting

complex from one system to multi system, from one programming language

to multi programming language, and from isolated network into interoperable

networks. The Object Management Group (OMG) announced Model Driven

Architecture (Ml )A) as a solution for effective development of complex software. As

its name implies, MDA is a model-centric development approach, which encourages

efficient use of system models in the software development process.

Models are abstractions of physical systems which allow engineers to

reason about the system under study. They can focus on relevant factors while

ignoring unnecessary details. Modeling software enhances modularity of software

components while decoupling unrelated components. Models from a classical

systems theory include sets of inputs, outputs, and states, along with the transition

functions. Defining software with inputs and outputs, along with the transition

functions, sets the boundary of software and reduces unexpected errors. Modeling

computer programs also open a possibility for computer simulation to be integrated

into the process of software development, especially for execution and debugging.

The model-based debugging can result in interactive execution and getting early

feedback. With this approach, we can also execute and analyze different scenarios

against our model. Based on the analysis of the simulation results, such as how

different external factors and design decisions affect the system's outputs, we

can select the best external factors and designs that meet our objectives. Visual

demonstration or animation of the program execution is also easily accomplished.

process process process

Figure 3-2. Data-flow between blocks

Animation is a powerful tool to visualize the internal process of a dynamic system,

and it has a growing role in computer-aided algorithm design, documentation and

debugging, education, performance monitoring, and engineering control.

Unlike most visual programming languages that visually mimic the behavior of

textual programs, we applied dynamic model types, such as the Functional Block

Model (FBM) and Finite State Machine (FSM), in the construction and execution

of programs to capture the nature of programming principles. Unifying modeling

with programming brings several benefits to programming: capturing dynamics

of programs using simulation model types; getting early feedback by running and

modifying models; applying metaphors to allow greater flexibility and freedom

in model representation; and storing model components in an ontological model

structure [78].

3.2.2 Dynamic Modeling of Programming Principles

The previous section discussed modeling programs at a higher level. This

section focuses on modeling of principles and elements of programming and

discusses issues in representing programming constructors using dynamic models.

Representing data flow. The concept of data flow is explained as data move

around an information system or processor. Visualizing data flow elucidated its

concept clearly than textual explanation. In Figure 3-2, arrows represent data

flowing into and flowing out of the process unit.

Representing control flow. In imperative textual languages, control

flow is often designed to be sequential so that each statement is executed in the

order in which it appears statically in the written program. In this case, the

sequential composition of statements is the normal style of programming. The

sequential composition, however, becomes unclear when it comes to 3D space. In

3D space, the sequence of statements must be specified for all three dimensions,

which is not a simple task. Since the control flow in the declarative programming,

especially in the logic programming, is implicit, the representation of the control

flow in declarative visual languages is not a big issue. However, for 3D languages

wherein the control flow must be specified explicitly, there must be explicit v--v,

such as using lines or arrows to represent the flow of control between program

units. Dynamic model types in simulation have syntactically and semantically

well-defined connections between operational units. Use of dynamic model types

in 3D programming has advantages with regard to these well-defined connectors,

which represent the control flow of programs.

Handling branches and loops. In a structured program, the flow of

execution is changed by the evaluation of the conditional expressions in branches

and loops, such as "if," -'.--ich," "for," and !lle" statements. The conditional

branches and loops in programming can be modeled in more than one way using

dynamic model types [79]. Semantic similarities exist between conditional branches

in programming and FSM in dynamic models. The Finite State Machine (FSM)

takes input and decides the next state to move by comparing the input values

against the condition of transitions that are going out of the current state. The

next state selected by the true transition becomes the current state and the output

of FSM. In the RUBE framework, conditional branches can be modeled using a

multimodel block that has FSM inside. The condition on each transition in the

FSM can model the conditional expression in the branch statements. The input

values of the FSM are used to decide which transitions must be taken. This process

is identical to the evaluation of the conditional expression of if, switch, for and

while statements based on the evaluation variables. Another way of modeling is

having FBM with a block that can produce different results for different output

c o n d i^io n rt r u e w 1 if -
true f [tre---*

condition start -* condition

false else 1 | ---- |
S -false-* l -*

FSM if-else FBM if-else

-- do

do while xit- continue

do-while while-do

Figure 3-3. Dynamic model components for conditional branches

ports. Loops can be modeled with FBM that have a circular connection between

blocks. To prevent infinite execution of the circularly connected blocks in a

program model, the exit condition must be specified in at least one block among

the circularly connected blocks. The exit condition of loops can be modeled with

output-selective blocks or multimodel blocks with FSM inside. The diagrams

in Figure 3-3 show dynamic models for "if-else", "dowhile", and v.! ile-do"

statements, respectively.

Modularization and parameter passing. Program modularization makes

large programs more manageable. Fix et al. [80] claimed that the skill recognizing

the links tying the separate program modules together can contribute to the

formation of the mental representation of a program. With visual programming

languages, it is very convenient to represent the program modules and the

relationship among them provided by visual scoping and explicit means of

parameter passing. In our research, predefined and user-defined functions are

stored in separate files. A program is constructed by combining separately defined

program modules, which are represented by 3D icons, into a single program

model using the 3D modeling environment called the blender interface in RUBE.

Parameter passing is explicitly defined by connecting input ports to output ports

between 3D icons.

Level of detail. Usually language constructors in 3D programming require

more physical space than textual constructors. In visual programming, the level of

details of program modules determines how dense or coarse the program appears

in 2D or 3D space. Determining the most effective-but still best representative

level of details for program modules-is a very tricky problem. This is because

it depends on the complexity of the applications' domains and the skill of the

programmers. For visual programming where physical space is a critical factor,

different technologies, such as nesting, zooming in, and fish-eye views, provide

efficient v--~ to utilize program space.

Visual execution. One of the advantages in visual programming is the

visual execution of programs. Since a program is visually represented, it is easy

to animate its execution during runtime. The animation of a program execution

can be done by adding a code that changes the appearance of visual program

constructors into each of the program modules. In this way, when the specific

program module is executed, the appearance of representative icons is changed.

When applied to a graphics program, the execution of a program is animated while

the result of the program execution, which is the construction of graphics objects,

appears. Using the blender interface, a 3D program and its output graphics are

co-existed and animated together in the same 3D space.

Concurrency. A program is said to be "concurrent" if more than one

execution threads of control exists in the program. For some applications,

such as servers and graphical applications, concurrency is a logical structure.

Concurrency also arises to handle independent physical devices or to increase

system performance. Visualization and animation are also effective methods for

no-input bki
block I
nok-input | multiple
merge o
no- input
block "

Parallel execution Parallel execution

Figure 3-4. Parallelism in functional block model (FBM)

understanding and debugging parallel programs [7, 8, 81]. A line connecting two

points is suitable to indicate the transmission and receipt of a message. Zooming

or scrolling aids users to comprehend a potentially cluttered display. Animation

is good for observing the instantaneous state of the parallel system. The 3D

visualization of VisuaLinda helps the programmer in debugging parallel programs

and in displaying both the relation between the Linda server and multiple clients

and their execution [12]. The use of dynamic model types, such as FBM and

Petri-Net, in programming can also facilitate the visualization of concurrent

threads and their execution. Figure 3-4 shows concurrent execution of blocks in


3.3 RUBE Framework

This section introduces the RUBE framework as a visual programming

environment, which is developed with approaches of customizable visualizations

and programming via dynamic model types.

3.3.1 Framework Structure

Originally, RUBE was developed as a Web and XML based modeling and

simulation framework for geometry and dynamic models. The RUBE framework

includes a Python-based user interface called the blender interface to provide the

complete 3D modeling and simulation environment for users [82]. We expanded

the RUBE framework as a programming environment by supplying 3D APIs


Python code
Customized iconsthon code
RUBE fragments
--RUBE- ---
PythonAPI Factory' Chemistry Graphics Math
theme ] theme codes codes

---------------------- i----------------

______ Blender __

3D program MXLoDXL
Without translator
E dynamics
Framework I .DXL


3D program with

Figure 3-5. The RUBE programming APIs and the RUBE framework

as modeling elements. The 3D APIs in the RUBE framework are composed of

program code fragments and their representative icons. Figure 3-5 shows the

RUBE APIs on top of the RUBE framework. The detailed translation processes

in the RUBE framework are also depicted in this figure. The red dashed-line

rectangle in Figure 3-5 shows my contributions for the programming using the

RUBE framework.

A user can start programming from the RUBE framework by running the

blender interface written in Python from Blender. Once the blender interface

is started, users can import the predefined or user-created program modules to

construct their program models. This is done by clicking appropriate buttons

that represent program modules from the blender interface. When a user locates

LL11L .2J0 Y'.J J-AnTa I I I--'a. -Ii IL -

Figure 3-6. Snapshot of the blender interface showing a simple program model

and imports a program module, a representative icon and a piece of Python code

associated with the module are also imported into the system. Figure 3-6 shows

the snapshot of the blender interface with a very simple program model, which

creates, duplicates, and di-p l'i a sphere object using factory metaphor icons. The

code fragments for the user created program model are scattered throughout the

code library, and they are not glued yet to form an executable program.

Internal representations of models in RUBE are in eXtensible Markup

Language (XML). By clicking the Simulate button from the blender interface,

the model components and structure defined in the Blender 3D window are stored

in a textual XML file. The user-defined model is stored in Multimodel eXchange

Language (\!X I), an XML modeling language developed by the modeling and

simulation research group at the University of Florida. Figure 3-7 shows the MXL

representation of the program model defined in Figure 3-6. In MXL, predefined or