Vision-Based Control for Flight Relative to Dynamic Environments

Material Information

Vision-Based Control for Flight Relative to Dynamic Environments
Causey, Ryan S
Place of Publication:
[Gainesville, Fla.]
University of Florida
Publication Date:
Physical Description:
1 online resource (164 p.)

Thesis/Dissertation Information

Doctorate ( Ph.D.)
Degree Grantor:
University of Florida
Degree Disciplines:
Aerospace Engineering
Mechanical and Aerospace Engineering
Committee Chair:
Lind, Richard C.
Committee Members:
Crane, Carl D.
Dixon, Warren E.
Slatton, Kenneth C.
Graduation Date:


Subjects / Keywords:
Aircraft ( jstor )
Cameras ( jstor )
Coordinate systems ( jstor )
Geometric planes ( jstor )
Homography ( jstor )
Moving images ( jstor )
Remotely piloted vehicles ( jstor )
Simulations ( jstor )
State estimation ( jstor )
Velocity ( jstor )
Mechanical and Aerospace Engineering -- Dissertations, Academic -- UF
autonomous, camera, estimation, homography, moving, tracking, vision, visual
City of Miami ( local )
bibliography ( marcgt )
theses ( marcgt )
government publication (state, provincial, terriorial, dependent) ( marcgt )
born-digital ( sobekcm )
Electronic Thesis or Dissertation
Aerospace Engineering thesis, Ph.D.


The concept of autonomous systems has been considered an enabling technology for a diverse group of military and civilian applications. The current direction for autonomous systems is increased capabilities through more advanced systems that are useful for missions that require autonomous avoidance, navigation, tracking, and docking. To facilitate this level of mission capability, passive sensors, such as cameras, and complex software are added to the vehicle. By incorporating an on-board camera, visual information can be processed to interpret the surroundings. This information allows decision making with increased situational awareness without the cost of a sensor signature, which is critical in military applications. The concepts presented in this dissertation facilitate the issues inherent to vision-based state estimation of moving objects for a monocular camera configuration. The process consists of several stages involving image processing such as detection, estimation, and modeling. The detection algorithm segments the motion field through a least-squares approach and classifies motions not obeying the dominant trend as independently moving objects. An approach to state estimation of moving targets is derived using a homography approach. The algorithm requires knowledge of the camera motion, a reference motion, and additional feature point geometry for both the target and reference objects. The target state estimates are then observed over time to model the dynamics using a probabilistic technique. The effects of uncertainty on state estimation due to camera calibration are considered through a bounded deterministic approach. The system framework focuses on an aircraft platform of which the system dynamics are derived to relate vehicle states to image plane quantities. Control designs using standard guidance and navigation schemes are then applied to the tracking and homing problems using the derived state estimation. Four simulations are implemented in MATLAB that build on the image concepts present in this dissertation. The first two simulations deal with feature point computations and the effects of uncertainty. The third simulation demonstrates the open-loop estimation of a target ground vehicle in pursuit whereas the four implements a homing control design for the Autonomous Aerial Refueling (AAR) using target estimates as feedback. ( en )
General Note:
In the series University of Florida Digital Collections.
General Note:
Includes vita.
Includes bibliographical references.
Source of Description:
Description based on online resource; title from PDF title page.
Source of Description:
This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Thesis (Ph.D.)--University of Florida, 2007.
Adviser: Lind, Richard C.
Statement of Responsibility:
by Ryan S Causey.

Record Information

Source Institution:
Rights Management:
Copyright Causey, Ryan S. Permission granted to the University of Florida to digitize, archive and distribute this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.
Resource Identifier:
662596751 ( OCLC )
LD1780 2007 ( lcc )


This item has the following downloads:







































































































































































Full Text


The proliferation of autonomous systems is generating a demand for smarter, more complex

vehicles. The motivation behind these concept vehicles is to operate in urban environments

which requires a number of complex systems. Video cameras have been chosen as sensors to

facilitate topics such as obstacle detection and avoidance, target tracking and path planning.

These technologies have stemmed from two communities in the literature: (i) image processing

and computer vision and (ii) performance and control of autonomous vehicles. This chapter

will focus on the research applied to autonomous systems and describe the current state of this

research, problems that have been addressed, some difficulties associated with vision, and some

areas in need of contribution. In particular, the review will cover the topics of most relevance to

this dissertation and highlight the efforts toward autonomous UAV

The block diagram shown in Figure 1-6 illustrates the components of interest described

in this dissertation for state estimation and tracking control with respect to a moving object

which involves object motion detection, object state estimation, and object motion modeling and

prediction. The literature review of these topics is given in this section.

2.1 Detection of Moving Objects

In order to track and estimate the motion of objects in images using a monocular camera

system, a number of steps are required. A common first step in many image processing

algorithms is feature point detection and tracking. This step determines features of interest,

such as comers, in the image that usually correspond to objects of interest, such as windows, in

the environment. The famous feature point tracker proposed by Lucas and Kanade [17, 18] has

served as a foundation for many algorithms. This technique relies on a smoothness constraint

imposed on the optic flow that maintains a constant intensity across small base-line motion of

the camera. Many techniques have built upon this algorithm to increase robustness to noise and

outliers. Once feature tracking has been obtained, the next process involves segmenting the

image for moving objects. The need for such a classification is due the fact that standard image

image where the optical axis penetrates the image plane to the upper left hand corner of the

image. This translation is done using the terms o,, and ov, given in units of pixels. The skew

factor is another intrinsic parameter which accounts for pixels that are not rectangular and is

defined as so. The ideal perspective transformation now takes the general form given in Equation

3-11, where pixel mapping, origin translation, and skewness are all considered.

p' fs,, fse o,, 1 00

The perspective transformation obtained in Equation 3-11 is rewritten to Equation 3-12.

rlzx' = Knorl (3-12)

The 3 x 3 matrix K is called the intrinsic parameter matrix or the calibration matrix while

the 3 x 4 constant matrix Ho defines the perspective projection, and finally x' represents the

homogeneous image coordinates [p', v', 1]' that contain pixel mapping and skew.
3.2.3 Extrinsic Parameters

In order to achieve this transformation to image coordinates, both intrinsic and extrinsic

parameters must be known or estimated a priori through calibration. The extrinsic parameters

of the camera can be described as the geometric relationship between the camera frame and the

inertial frame. This relationship consists of the relative position, T, and orientation, R, of the

camera frame to an inertial frame. By defining the position vector of a feature point relative to

an inertial as 5 = [t, y,(, ]'l, transformations can map the expression found in Equation 3-12

to obtain a general equation that maps feature points in the inertial frame to coordinates in the

image plane for a calibrated camera.

p' fs,, fse o,, 1 0
Be v' 0 fsv ov 0 0 R (3-13)

1 0 1 001

x 10


200 400
Index (counts)

200 400
Index (counts)

Figure 10-19. Norm error for A) relative translation and B) relative rotation

Figures 10-20 and 10-21 show the relative translation and rotation decomposed into their

respective components and expressed in the body frame, B. These components reveal the relative

information needed for feedback to track or home in on the target of interest.



Index (counts)

Index (counts)

Index (counts)

Figure 10-20. Relative position states: A) X, B) Y, and C) Z


Index (counts)

Figure 10-21. Relative attitude states: A) Roll, B) Pitch, and C) Yaw



00O 00 200 400 600
Index (counts)

Index (counts)


Ryan Scott Causey was born in Miami, Florida, on May 10, 1978. He grew up in a stable

family with one brother in a typical suburban home. During his teenage years and into early

adolescence, Ryan built and maintained a small business providing lawn care to the local

neighborhood. The tools acquired from this work carried over into his college career. After

graduating from Miami Killian Senior High School in 1996, Ryan attended Miami Dade

Community College for three years and received an Associate in Arts degree. A transfer student

to the University of Florida, Ryan was prepared to tackle the stresses of a university aside from

the poor statistics on transfer students. A few years later, he received a Bachelor of Science in

Aerospace Engineering with honors in 2002 and was considered in the top three of his class.

Ryan soon after chose to attend graduate school back at the University of Florida under Dr. Rick

Lind in the Dynamics and Controls Laboratory. During the summertime, Ryan interned twice at

Honeywell Space Systems as a Systems Engineer in Clearwater, FL and once at The Air Force

Research Laboratory in Dayton, OH. Vision-based control of autonomous air vehicles became

his interest and he is now pursuing a doctorate degree on this topic. Ryan was awarded a NASA

Graduate Student Research Program (GSRP) fellowship in 2004 for his proposed investigation on

this research.

fo, and uncertainty bounded by size of Ay E R. A similar expression in Equation 4-4 presents
the range of values for radial distortion.

f = { fo $ f : If || < Ay} (4-3)

d = {do+-t8d 116 8d ~d} (4-4)

The variations of feature points due to the camera uncertainties can be directly computed.

The uncertain parameters given in Equation 4-3 and Equation 4-4 are substituted into the

camera model of Equation 4-1 and Equation 4-2. The resulting expressions for feature points are

presented in Equations 4-5 and 4-6.

pl = foil~lx~ 1+d fo24_ r 2872 (4-5)
+3doffi, +3dlo +doi +fid+3f d + 3 fod8yd

vl = fo 1+off + + xE28(46
+3dof rl +3dfo +di +f id +3f8df+3f yd d

These~~~~~~ eqain deosrt a opiae eaiosi ewe neranyi etr
points and uncrtainty in camra~ ,6 paaees The featur poits3,6, actual vary lieal with

unerait in foa eghfracmr ihu ail itrinoeeteicuino

u=~~~~~~ {u 8, : ,| -t~lof2v = {vgo~f +L~- fiv : t 3fi~ | < Av } (4-8)f


1.1 Motivation

Autonomous systems are an enabling technology to facilitate the needs of both military

and civilian applications. The usefulness of autonomous systems ranges from robotic assembly

lines for streamlining an operation to a rover exploring the terrain of a distant planet. The main

motivation behind these types of systems is the removal of a human operator which in many

cases reduces operational cost, human errors, and, most importantly, human risk. In particular,

military missions consistently place soldiers in hazardous environments but in the future could

be performed using an autonomous system. The federal sector is considering autonomous

vehicles, specifically, to play a more prominent role in several missions such as reconnaissance,

surveillance, border patrol, space and planet exploration over the next 30 years [1]. This increase

in capability for such complex tasks requires technology for more advanced systems to further

enhance the situational awareness.

Over the past several years, the interest and demand for autonomous systems has

grown considerably, especially from the Armed Forces. This interest has leveraged funding

opportunities to advance the technology into a state of realizable systems. Some technical

innovations that have emerged from these efforts, from a hardware standpoint, consist mainly

of increasingly capable microprocessors in the sensors, controls, and mission management

computers. The Defense Advanced Research Projects Agency (DARPA) has funded several

projects pertaining to the advancement of electronic devices through size reduction, improved

speed and performance. From these developments, the capability of autonomous system has been

demonstrated on vehicles with strict weight and payload requirements. In essence, the current

technology has matured to a point where autonomous systems are physically achievable for

complex missions but not yet algorithmically capable.

The aerospace community has employed many of the research developed for autonomous

systems and applied it to Unmanned Aerial Vehicles (UAV). Many of these vehicles are currently

a single camera setup, known as monocular vision, and a two camera setup, known as stereo

vision. For monocular vision, a sequence of images are taken over time whereas stereo vision

uses two images taken by different cameras at the same time. Motion estimates, using monocular

vision, has been solved for the cases associated with the movement of the camera relative to a

stationary objects and the reverse problem involving movement of objects relative to a stationary

camera. The process of determining camera motion from stationary objects is commonly referred

to as localization. Conversely, determining the motion or position of an object in space from a

pair of images is known as structure from motion. For fixed objects, simultaneous localization

and mapping (SLAM) can be employed to estimate the camera motion in conjunction with the

object's locations. Meanwhile, the use of stereo vision allows one to estimate the motion of

objects while the camera is also moving. Solutions to these methods are well established in

the computer science community and the mathematical details regarding these techniques are

provided in Chapter 3. This dissertation will focus on the monocular camera configuration to

address the state estimation problem regarding moving targets.

The advantage of these techniques becomes more apparent to UAV when applied to

guidance, navigation, and control. By mounting a camera on a vehicle, state estimation of

the vehicle and objects in the environment can be achieved in some instances through vision

processing. Once state estimates are known, they can then be used in feedback. Control

techniques can then be utilized for complex missions that require navigation, path planning,

avoidance, tracking, homing, etc. This general framework of vision processing and control has

been successfully applied to various systems and vehicles including robotic manipulators, ground

vehicles, underwater vehicles, and aerial vehicles but there still exists some critical limitations.

The problematic issues with using vision for state estimation involves camera nonlinearities,

camera calibration, sensitivity to noise, large computational time, limited field of view, and

solving the correspondence problem. A particular set of these image processing issues will be

addressed directly in this dissertation to facilitate the control of autonomous systems in complex


Abstract of Dissertation Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Doctor of Philosophy



Ryan Scott Causey

August 2007

Chair: Richard C. Lind
Major: Aerospace Engineering

The concept of autonomous systems has been considered an enabling technology for a

diverse group of military and civilian applications. The current direction for autonomous systems

is increased capabilities through more advanced systems that are useful for missions that require

autonomous avoidance, navigation, tracking, and docking. To facilitate this level of mission

capability, passive sensors, such as cameras, and complex software are added to the vehicle.

By incorporating an on-board camera, visual information can be processed to interpret the

surroundings. This information allows decision making with increased situational awareness

without the cost of a sensor signature, which is critical in military applications. The concepts

presented in this dissertation facilitate the issues inherent to vision-based state estimation of

moving objects for a monocular camera configuration. The process consists of several stages

involving image processing such as detection, estimation, and modeling. The detection algorithm

segments the motion field through a least-squares approach and classifies motions not obeying

the dominant trend as independently moving objects. An approach to state estimation of moving

targets is derived using a homography approach. The algorithm requires knowledge of the

camera motion, a reference motion, and additional feature point geometry for both the target and

reference objects. The target state estimates are then observed over time to model the dynamics

using a probabilistic technique. The effects of uncertainty on state estimation due to camera

calibration are considered through a bounded deterministic approach. The system framework

focuses on an aircraft platform of which the system dynamics are derived to relate vehicle states

operational and have served reconnaissance missions during Operation Iraqi Freedom. The

Department of Defense (DoD) has recorded over 10,000 flight hours performed by UAV in

support of the war in Iraq since September 2004 and that number is expected to increase [1].

Future missions envision UAV to conduct more complex task such as terrain mapping,

surveillance of possible threats, maritime patrol, bomb damage assessment, and eventually

offensive strike. These missions can span over various types of environments and, therefore,

require a wide range of vehicle designs and complex controls to accommodate the associated


The requirements and design of UAV are considered to enable a particular mission

capability. Each mission scenario is the driving force of these requirements and are dictated

by range, speed, maneuverability, and operational environment. Current UAV range in size from

less than 1 pound to over 40,000 pounds. Some popular UAV that are operational, in testing

phase, and in the concept phase are depicted in Figure 1-1 to illustrate the various designs. The

two UAV on the left, Global Hawk and Predator, are currently in operation. Global Hawk is

employed as a high altitude, long endurance reconnaissance vehicle whereas the Predator is

used for surveillance missions at lower altitudes. Meanwhile, the remaining two pictures present

J-UCAS, which is a joint collaboration for both the Air Force and Navy. This UAV is described

as a medium altitude flyer with increased maneuverability over Global Hawk and the Predator

and is considered for various missions, some of which have already been demonstrated in flight,

such as weapon delivery and coordinated flight.

The advancements in sensors and computing technology, mentioned earlier, has facilitated

the miniaturization of these UAV, which are referred to as Micro Air Vehicles (MAV). The scale

of these small vehicles ranges from a few feet in wingspan down to a few inches. DARPA has

also funded the first successful MAV project through AeroVironment, as shown in Figure 1-2,

where basic autonomy was first demonstrated at this scale [2]. These small scales allow highly

agile vehicles that can maneuver in and around obstacles such as buildings and trees. This

capability enables UAV to operate in urban environments, below rooftop levels, to provide

-04, -0 /-0


Figure 10-7. Optical flow for nominal (black) and perturbed (red) cameras for A) f = 1.1 and
d = 0, B) f = 1.0 and d = 0.01, and C) f = 1.1 and d = 0.01

The vectors in Figure 10-7 indicate several effects of camera perturbations noted in

Equations 4-5 and 4-6. The perturbations to focal length scale the feature points so the

magnitude of optic flow is uniformly scaled. The perturbations to radial distortion have larger

effect as the feature point moves away from the center of the image so the optic flow vectors

are altered in direction. The combination of perturbations clearly changes the optic flow in both

magnitude and direction and demonstrates the feedback variations that can result from camera


The optic flow is computed for images captured by each of the perturbed cameras. The

change in optic flow for the perturbed cameras as compared to the nominal camera is represented

as 6Sy and is bounded in magnitude, as derived in Equation 4-14, by Ay. The greatest value of Sy-

presented by these camera perturbations is compared to the upper bound in Table 10-6. These

numbers indicate the variations in optic flow are indeed bounded by the theoretical bound derived

in Chapter 4 and indicate the level of flow variations induced from the variations in camera


Table 10-6. Effects of camera perturbations on optic flow
Perturbation Analyze Analyze Analyze
Set only with 87 only with 8d with 8f and 6d

87 = -0.2 and 6d = -0.02 0.0476 0.0476 0.0040) 0.0040) 0.0496 0.0543
87 = -0. 1 and 8d -0.01 0.0238 0.0476 0.0020 0.0040 0.0252 0.0543
87 = 0.1 and 8d 0.01 0.0238 0.0476 0.0020 0.0040 0.0264 0.0543
87 = 0.2 and 6d = 0.02 0.0476 0.0476 0.0040 0.0040 0.0543 0.0543



























Optic flow measurements for example 1 .....

Virtual environment for example 2 ......

Feature points across two image frames .....

Uncertainty in feature point .......

Uncertainty results in optic flow ......

Nominal epipolar lines between two image frames

Uncertainty results for epipolar geometry ....

Nominal estimation using structure from motion .

Uncertainty results for structure from motion ..

Vehicle trajectories for example 3 ......

Position states of the UAV with on-board camera

Attitude states of the UAV with on-board camera

Position states of the reference vehicle ......

Attitude states of the reference vehicle ......

Position states of the target vehicle .......

Attitude states of the target vehicle .......

Norm error .......

Relative position states ......

Relative attitude states ......

Virtual environment .......

Inner-loop pitch to pitch command Bode plot ..

Pitch angle step response ......

Altitude step response .......

Inner-loop roll to roll command Bode plot ....

Roll angle step response ......

Heading response .......

. .126

..... .. .. .127

....... .. .128

........ .. .128

........ .. .129

. .. ... .. .130

. ...... .. .131

.... .. .. .132

. ... .. .. .133

........ .. .134

.... .. .. .135

.... .. .. .135

....... .. .135

....... .. .136

........ .. .136

........ .. .136

......... .. .. 137

.......... .. 137

.......... .. 137

........ .. .138

. ... .. .. .141

........ .. .141

.......... .. 142

. .... .. .. .143

.......... .. 144

......... .. .. 144

10-29 Open-loop estimation of target's inertial position

checking the condition that NTe3 = n3 > 0, where e3 is in the direction of the optical axis normal

to the image plane.

3.6.4 Structure from Motion

Structure from motion (SFM) is a technique to estimate the location of environmental

features in 3D space. This technique utilizes the epipolar geometry in Figure 3-4 and assumes

that the rotation, R, and translation, T, between frames is known. Given that, the coordinates of

rll and r12 can be computed. Recall, the fundamental relationship repeated here in Equation 3-62.

r12 = Rrll + T (3-62)

The location of environmental features is obtained by first noting the relationships

between feature points and image coordinates given in Equation 3-7 and Equation 3-8. These

relationships allow some components of rlx and rly to be written in terms of pu and v which are

known from the images. Thus, the only unknowns are the depth components, rl1,z and rl2,z, fOr

each image. The resulting system can be cast as Equation 3-63 and solved using a least-squares

2 (R11 z -+R12" +1IR13) Tx
z- (R21+R2 +R23 r2,z =I T (3-63)

1 (R31 9+-tR32" +1IR33)41 Tz
This equation can be written in a compact form as shown in Equation 3-64 using z =

[r12,z, r1,z] as the desired vector of depths.

Az = T (3-64)

The least-squares solution to Equation 3-64 obtains the depth estimates of a feature point

relative to both camera frames. This information along with the image plane coordinates can be

used to compute (Tll,x,Tl1,y) and (r12,x,r12,y) by substituting these values back into Equations 3-7

and 3-8. The resulting components of rll can then be converted to the coordinate frame of the

second image and it should exactly match r12. These values will never match perfectly due to

profile and is provided in Equation 8-11. This acceleration time history is computed implicitly

through the position estimates obtained from the homography algorithm

[ai (t -1) ,ai (t -2),...,ai (t N+-2),ai (t N+-1)] (8-10)

ai (t j) = vi (t j) vi (t j 1) (8-11)

The motion profiles given in Equations 8-8 and 8-10 provide the initial motion state

description that propagates the Markov transition probability function. The form of the Markov

transition probability function is assumed to be a Gaussian density function that only requires

two parameters for its representation. The parameters needed for this function include the mean

and variance vectors for the acceleration profile given in Equation 8-12. Note, during this chapter

pu (x) is the mean operator and not the vertical component in the image plane. Likewise, 02 (X) iS

referred to as the variance operator.

[p (ai (t +t j)) G2 (ai (t + j))] j ,1 ,k(8-12)

The Markov transition function is defined in Equation 8-13, where the arguments consist of

the mean and variance pertaining to the estimated acceleration.

P (ai (t + j)) = xn Cp (ai (t + j)) 02 (ai (t + j))) (8-13)

The initial mean and variance for acceleration are computed in Equations 8-14 and 8-16

for the transition function. The functions f, and f, are chosen based on the desired weighting of

the time history and can simply be a weighted linear combination of the arguments. These initial

statistical parameters are used in the prediction step and updated once a new measurement is


p (ai (t)) = fe (ai (t 1) ai (t 2) ,. .ai (t N)) (8-14)

a2 Si F)) f (a ir l ) i (t-2))2 (8-15)

P (x)






















3 h

Probability density function

Essential matrix

Relative rotation

Rotational transformation from body-fixed to camera-fixed coordinates

Rotational transformation from Earth-fixed to body-fixed coordinates

Rotational transformation from Earth-fixed to reference coordinates

Rotational transformation from Earth-fixed to target coordinates

Rotational transformation from Earth-fixed to virtual coordinates

Rotational transformation from reference-fixed to virtual coordinates

Rotational transformation from target-fixed to virtual coordinates

Rotational transformation from camera-fixed to virtual coordinates

Relative translation

Target-fixed coordinate frame

Position of camera along {$1,82, ~3} axes

Position of aircraft along {&1l, e^2, 83 } axes

Control input vector

Classification group of features to an independently moving object

Virtual coordinate frame

Search window in the image

Vector of aircraft states

Vector of initial aircraft states

Feature point measurements in the image plane

Camera parameter of the k camera

Horizontal angle for field of view


- (Xb, Yb ,Zb)

The same model was also used for the tanker or reference vehicle. The tanker was exactly

trimmed at the same conditions and airspeed as the receiver aircraft and given a specified

trajectory to follow. Initially the tanker's position was offset from the receiver's position at the

start of the simulation. The values of this offset are described relative to receiver's coordinate

frame and are as follows: 500 ft in front (+tX direction), 20 ft to the side (+tY direction),

and 100 ft above (-Z direction). The trajectory generated for the tanker aircraft prior to the

simulation was a straight and level flight with a slight drift toward the East direction. This lateral

variation was added to the trajectory to incorporate all three dimensions into the motion to test in

all directions.

On the other hand, the modeling of the drogue is much more difficult to characterize and is

of much interest in the research community. The stochastic nature of its motion is what makes

the modeling so challenging. The flow field affecting the drogue consist of many nonlinear

excitations including turbulence due to wake effects and vortex shedding from the tanker aircraft.

For this drogue model an offset trajectory of the tanker's motion was used as the drogue's general

motion. The offset of the drogue is initially at 200 ft in front +tX direction), O ft to the side (+tY

direction), and 70 ft above (-Z direction) relative to the receiver aircraft. More complicated

motions of the drogue were considered during testing but resulted in a diverging trajectory for

the receiver. This deviation from the desired path was due high rate commands saturating the

actuators. Low passing filtering can be incorporated to alleviate this behavior.

10.4.2 Control Tuning

The control architecture described in Chapter 9 is integrated and tuned for the nonlinear

F-16 model to accomplish this simulation. It was assumed that full state feedback of the aircraft

states were measurable including position. The units used in this simulation are given in ft and

deg which means the gains determined in the control loops were also found based on these units.

First, the pitch tracking for altitude controller is considered. The inner-loop gains for this

controller are given as ke = -3 and kg = -2.5. The bode diagram for pitch command to pitch

angle is depicted in Figure 10-23 for the specified gains. This diagram reveals the damping

The relationship shown in Equation 3-51 can be extended to image coordinates through

Equation 3-53.

x2 = Hxl (3-53)

A similar approach as used in the eight-point algorithm can be used to solve for the entries

of H. Multiplying both sides of Equation 3-53 with the skew symmetric matrix xi results in the

planar homography constraint shown in Equation 3-54.

xiHxy = 0 (3-54)

Since H is linear, linear algebra techniques can be used to stack the entries of H as a column

vector h and, therefore, Equation 3-54 can be rewritten to Equation 3-55,

a'h = 0 (3-55)

where a is the Kronecker product of xi and xl. Each feature point correspondence between

frames provides two constraints in determining the entries of H. Therefore, to solve for a

unique solution of H, Equation 3-55 requires at least four feature point correspondences.

These additional constraints can be stacked to form a new constraint matrix ?, as shown in

Equation 3-56.

'Y= aX1,a2,8 3, --, nn T (3-56i)

Rewriting Equation 3-55 in terms of the new constraint matrix results in Equation 3-57.

Wh = 0 (3-57)

The standard least-squares estimation can be used to recover H up to a scale factor.

Improvements can be made to the solution when more than four feature point correspondences

are used in the least-squares solution. The scale factor is then determined as the second largest

singular value of the solution H [102, 103], shown in Equation 3-58 for the unknown scaler h.

|1| = G2 (H) (3-5 8)

and incur cost and time delays. These additional expenses add complexity and eliminate the

attractiveness of low cost autonomous systems. Meanwhile, the current appeal of these systems

has been the use of low cost off-the-shelf components, such as cameras that are easily replaced.

Maintaining a low cost product is a goal for UAV that can be accomplished by considering a

vision system. If future operations require a stockpile of thousands of UAV or MAV ready to

deploy, then the capability to switch out or replace components in a timely fashion with little cost

is a tremendous functionality. Therefore, this dissertation describes a method that would enable

cameras to be replaced rapidly and without the need for extensive calibration.

1.5 Contributions

The goal of the work presented in this dissertation is to establish a methodology that

estimates the states of a moving object using a monocular camera configuration in the presence

of uncertainty. The estimates will provide not only critical information regarding target-motion

estimation for autonomous systems but also retain confidence values through a distribution

around a target's estimate. Previous work has investigated many problems and issues related

to this topic but has neglected several key features. In particular, this thesis addresses (i) the

physical effects of camera nonlinearities on state estimates, (ii) a multi-layered classification

approach to object motion based on visual sensing that determines the confidence measure in the

estimates, and (iii) the relationships between vehicle and sensor constraints coupled with sensor

fusion in an autonomous system framework.

The main contribution of this dissertation is the development of a state estimation process of

a dynamic object using a monocular vision system for autonomous navigation. In addition to the

main contribution, there exists some secondary contributions solved in the process of facilitating

the main goal. The contributions presented in this dissertation consist of the following:

*A homography approach to state estimation of moving objects is developed through a virtual
camera to estimate the relative pose of the target relative to the true camera. This virtual
camera facilitated the estimation process by maintaining a constant homography relative to a
known reference object.


Vision-based feedback can be an important tool for autonomous systems and is the primary

focus of this dissertation in the context of an unmanned air vehicle. This dissertation describes

a methodology for a vehicle, such as a UAV, to observe features within the environment and

estimate the states of a moving target using various camera configurations. The complete

equation of motion of an aircraft-camera system was derived in its general form that allows

multiple cameras. Camera models were summarized and the effects of uncertainty regarding the

intrinsic parameters was discussed. Expressions for worse-case bounds were derived for varies

vision processing algorithms on a conservative level. A classification scheme was summarized

to discern between stationary and moving objects within the image using a focus of expansion

threshold method. The homography derivation proposed was the main contribution of this

dissertation where the states of a moving target were formulated based on visual information.

Some underlining assumptions were imposed on the features and the system to obtain feasible

estimates. The two critical assumptions imposed on the features were the planar constraint and

the requirement of the distance to a feature on the reference and target vehicles be known and

equal. An additional assumption was placed on the system which involved a communication link

that allows the vehicle to have access to the states of the reference vehicle. The modeling of the

target position attempted to anticipate future locations to enable a predictive capability for the

controller and to provide estimates when the features are outside the field of view. The approach

summarized here consisted of a Hidden Markov method which has limitations for general 6-DOF

motion due to incomplete motion models. Lastly, a standard control design is tuned for an aircraft

performing waypoint navigation to use in closed-loop control where commands are generated

from the state estimator.

Simulations were presented to validated the proposed algorithms and to demonstrate the

applications for autonomous vehicles. The first simulation verified the feature point and optic

flow computation for a aircraft-camera system containing multiple cameras with time varying

during reconstruction.

R = UR ( + VT) T = URz( +\ EUT (3-48)

0 +1 0

whr Ry"2+0- 41 0 0

0 01

The eight-point algorithm fails with a non-unique solution when all points in 3D space lie on

the same 2D plane [102, 103]. When this situation occurs one must use the planar homography

approach, which is the topic of the next section.

3.6.3 Planar Homography

The homography approach can be used to solve the degenerate cases of the eight-point

algorithm. For instance, a very common case where the feature points of interest all lie on the

same 2D plane in 3D space causes the algorithm to produce nonunique solutions. This case, in

particular, is a crucial part of enabling autonomous systems to navigate in urban environments.

Manmade structures such as buildings, roads, bridges, etc. all contain planar characteristics

associated with their geometry. This characteristic also applies especially to aerial imagery at

high altitudes where objects on the ground are essentially viewed as planar objects. Therefore,

this section describes the planar case to estimating motion from two images of the same scene as

shown in Ma et al. [102, 103]. Figure 3-5 depicts the geometry involved with planar homography.

The fundamental relationship expressing a point feature in 3D space across a set of images is

given through a rigid body transformation shown in Equation 3-49.

r12 = Rrll + T (3-49)

Recall that rla and rll are relative position vectors describing the same feature point in space

with respect to camera 2 and camera 1, respectfully, and R and T are the relative rotation and

translation motion between frames.

2.2.1 Localization

Localizing the camera position and orientation relative to a stationary surrounding has been

addressed using a number of methods. An early method presented by Longuet-Higgins [38, 39]

used the coplanarity constraint also known as the epipolar constraint. Meanwhile, the subspace

constraint has also been employed to localize camera motion [40]. These techniques have

been applied to numerous types of autonomous systems. The mobile robotic community has

applied these techniques for the development of navigation in various scenarios [41-45]. The

applications have also extended into the research of UAV for aircraft state estimation. Gurfil

and Rotstein [46] was the first to extend this application in the framework of a nonlinear aircraft

model. This approach used optical flow in conjunction with the subspace constraint to estimate

the angular rates of the aircraft and was extended in [47]. Webb et al. [48, 49] employed the

epipolar constraint to the aircraft dynamics to obtain vehicle states. The foundation for both

of these approaches is a Kalman filter in conjunction with a geometric constraint to estimate

the camera motion. Some applications for aircraft state estimation have involved missions for

autonomous UAV such as autonomous night landing [50] and road following [51].

2.2.2 Mapping

Location estimation of stationary targets using algorithms such as structure from motion

has been extensively researched for non-static cameras with successful results. The foundation

of these techniques still rely on the geometric constraints imposed on stationary targets. The

decoupling of structure from the motion has been characterized in a number of papers by

Soatto et al. [52-58]. These approaches employ the subspace constraint to reconstruct feature

point position through an extended Kalman filter. Several survey papers have been published

describing the current algorithms while comparing the performance and robustness [59-62].

Robust and adaptive techniques have been proposed that use an adaptive extended Kalman filter

to account for model uncertainties [63]. In addition, Qian et al. [64] designed a recursive Hoo filter

to estimate structure from motion in the presence of measurement and model uncertainties while


Figure 5-2. Camera-fixed coordinate frame

Similar to the body-fixed coordinate frame, a transformation can be defined for the mapping

between the body-fixed frame, B and the camera frame, I as seen in Equation 5-5

i; by

i2. = RBI b2. (5-5)

13 b3B

where RBI is the relative rotation between frame B and I, respectfully. This transformation

is analogous to the aircraft's roll-pitch-yaw, where now these rotation angles define the roll,

pitch and yaw of the camera relative to the aircraft's orientation. The coordinate rotation

transformation, RBI, can be decomposed as a sequence of single-axis Euler rotations as seen in

Equation 5-6, similar to the body-fixed rotation matrix. The orientation angles of the camera are

required to determine the imaging used for vision-based feedback. The roll angle, #c, describes

rotation about ;;3, the pitch angle, 8c, describes rotation about 12 and the yaw angle, c,, describes

rotation about ii.

RBI= l(cOle 2e>le Oc)3c)] (5-6)

The matrix RBI in Equation 5-6 will transform a vector in body-fixed coordinates to
camera-fixed coordinates. This transformation is required to relate camera measurements to
on-board vehicle measurements from inertial sensors. The matrix again depends on the angular

9.2 Controller Development .. . .... .. 118
9.2.1 Altitude Control . .... ... . 18
9.2.2 Heading Control ......... .. .... .. 119
9.2.3 Depth Control . . .. ..... .21

10 SIMULATIONS ............. ..............123

10.1 Example 1: Feature Point Generation ..... .... .. .. 123
10.2 Example 2: Feature Point Uncertainty .... .... . .. 126
10.2.1 Scenario . .. .... .. .26
10.2.2 Optic Flow ......... ... .. .. 128
10.2.3 The Epipolar Constraint . .... .. .. 130
10.2.4 Structure From Motion . ... .. .. 132
10.3 Example 3: Open-loop Ground Vehicle Estimation .. . 133
10.3.1 System Model . ... ..... .. .34
10.3.2 Open-loop Results . .. .. .. .. 135
10.4 Example 4: Closed-loop Aerial Refueling of a UAV .. .. .. .. .. 138
10.4.1 System Model ... . ..... .. .39
10.4.2 Control Tuning . ..... .. . 140
10.4.3 Closed-loop Results . ... .... .14
10.4.4 Uncertainty Analysis ....... ... .. .. 148

11 CONCLUSION ............. ..............151

REFERENCES ......... . ..... .. 154

BIOGRAPHICAL SKETCH ......... .. ... .. 164

The expressions for features points, given in Equation 4-7 and Equation 4-8, can

be substituted into Equation 4-11 to introduce uncertainty. The resulting expression in

Equation 4-12 separates the known from unknown elements.

J = +(4-12)
Vog Votv -v

A range of variations are allowed for optic flow due to the uncertainty in feature points.

The expression for IJ can thus be written using nominal, IJo, and uncertain, Sy, terms as in

Equation 4-13 where the uncertainty is bounded by Ay E R.

g = {$o+8y 6~ : |Sy|

The amount of uncertainty in optic flow depends on the uncertainty in each feature point.

The maximum variation in velocity for a given point, determined by rl, is given in Equation 4-14.

The actual bounds on the feature points, as noted in Equation 4-9 and Equation 4-10, varies

depending on the location of each feature point so bounds of A,, and A,, are given for each

vertical component and Av, and Av, are given for each horizontal component. As such, the bound

on variation is noted in Equation 4-14 as specific to the rll and rl2 used to gather feature points in

each image.

Ay= max || (8,,, 8)2+ _6v 8,2 | (4-14)

V1? I a,1

4.3 Epipolar Geometry

State estimation using epipolar geometry, computed as a solution to Equation 3-44, requires

a pin-hole camera whose intrinsic parameters are exactly known. Such a situation is obviously

not realistic so the effect of uncertainty can be determined. A non-ideal camera will lose the



Feature point location on reference vehicle realtive and expressed in

Earth-fixed coordinates

Feature point location on target vehicle expressed in Earth-fixed


Variance operator of a vector x

Gradient threshold

Attitude of aircraft about {$1, 82, ~3) aXeS

Attitude of camera about {81l, c^2, 83) aXeS

Roll command

Heading command

Angular rates of aircraft about {$1, 82 -3) aXeS

Angular rates of camera about {FIl, c^2, c^3) aXeS

Radial distortion uncertainty bound

Focal length uncertainty bound

Uncertainty bound in the entries of the planar homography matrix

Uncertainty bound in the entries of the essential matrix

Uncertainty bound in depth components

Lateral deviation between vehicle and target

Uncertainty bound in optic flow

Uncertainty bound in pu

Uncertainty bound v

Two-view feature point matrix using the planar homography matrix

Nominal two-view feature point matrix using the planar homography

(#, 6, y)

S= (p, q, r)

me = (Pc, 4c, re)



8.1 Introduction

Once state estimation of a moving target has been obtained the next step is to record these

estimates over time to try and leamn the object's general motion. The purpose of understanding

these motions are useful for prediction and allows for closed-loop control for applications such as

autonomous docking and AAR. In essence, this prediction step provides the tracking vehicle with

future state information of the target which assists the controller in both the tracking and docking

missions. This chapter describes a probabilistic method that employs the time history estimates

of the target's motion to determine future locations. In addition to providing state predictions, the

modeling scheme also provides position updates when features are outside the field of view.

Linear modeling is not sufficient for prediction in this situation, where the motion is

stochastic. Linear techniques that estimate a transfer function, such as ARX, require that the

inputs and outputs of the system are known. Although this is the case for many systems, it

doesn't apply in this scenario because the inputs (i.e. the forces) on the target are assumed to be

unknown. For example, in the AAR mission the target, or drogue, interacts with a flow field that

is potentially turbulent due to the effects of the surrounding aircraft (i.e. tanker and receiver)

and difficult to model. The drogue is also tethered by a flexible boom that applies reaction forces

which are dictated from the tanker aircraft and the aerodynamic forces on the boom. These

factors make the modeling task challenging to accurately represent the motion of a general target

with unmodeled dynamics and disturbances. Therefore, the method considered in this dissertation

will consist of a probabilistic approach to account for general motions with stochastic behavior.

8.2 Dynamic Modeling of an Object

There are numerous modeling schemes in the research community. The probabilistic

approaches can be separated into two main categories consisting of supervised and unsupervised

learning algorithms. Supervised algorithms require training data that determines trends apriori

and classifies the the motion under consideration to the trends observed during training.

Currently, several universities have a research facility dedicated to the investigation of

MAV, including Brigham Young University (BYU), Stanford University, Georgia Institute

of Technology, and the University of Florida. The autonomous capabilities demonstrated by

BYU incorporated an autopilot system for waypoint navigation that integrated traditional IMU

sensors [3, 4]. Meanwhile, Stanford has examined motion planning strategies that optimize flight

trajectories to maintain sensor integrity for improved state estimation [5]. The work at Georgia

Tech and BYU has considered corporative control of MAV for autonomous formation flying [6]

and consensus work for distributed task assignment [7]. Alternatively, vision based control has

also been the topic of interest at both Georgia Tech and UF. Control schemes using vision have

been demonstrated on platforms such as a helicopter at Georgia Tech [8], while UF implemented

a MAV that integrated vision based stabilization into a navigation architecture [9, 10]. The

University of Florida has also considered MAV designs that improve the performance and agility

of these vehicles through morphing technology [11-13]. Fabrication facilities at UF have enabled

rapid construction of design prototypes useful for both morphing and control testing. The fleet of

MAV produced by UF are illustrated in Figure 1-3 where the wingspan of these vehicles range

from 24 in down to 4 in.

Figure 1-3. The UF MAV fleet

There are a number of current difficulties associated with MAV due to their size. For

example, characterizing their dynamics under flight conditions at such low Reynolds numbers

is an extremely challenging task. The consequence of increased agility at this scale also gives

rise to erratic behavior and a severe sensitivity to wind gust and other disturbances. Waszak

et al. [14] performed wind tunnel experiments on 6 inch MAV and obtained the required

stability derivatives for linear and nonlinear simulations. Another critical challenge toward MAV

Camera Feature Point Moving Object State Sohsi
x(0) = 0Model Tracker Distection Estimation Dermnti

LIIIIIImage 13Processing

Controller Moc eling

Figure 1-6. Closed-loop block diagram with visual state estimation

simple image differencing, where the stationary background is segmented out; however, this

approach does not apply to moving cameras. In the case of a moving camera, the background

is no longer stationary and it begins to change over time as the vehicle progresses through the

environment. Therefore, the images taken by a moving camera contain the motion due to the

camera, commonly called ego-motion, and the motion of the object. Techniques that involve

camera motion compensation or image registration have been proposed to work well when there

exists no stationary objects close to the camera which cause high parallax. This dissertation

will establish a technique to classify objects in the field of view as moving or stationary while

accounting for stationary objects with high parallax. Therefore, with a series of observations of a

particular scene, one can determine which objects are moving in the environment.

Knowing which objects are moving in the image dictates the type of image processing

required to accurately estimate the object's states. In fact, the estimation problem becomes

infeasible for a monocular system when both the camera and the object are moving. This

unattainable solution is cause by a number of factors including 1) inability to decouple the

motion from the camera and target and 2) failure to triangulate the depth estimate of the object.

For this configuration, relative information can be obtained and fused with additional information

for state estimation. First, decoupling the motion requires known information regarding motion

of the camera or the motion of the object, which could be obtained through other sensors such

The homography solution is then decomposed into its rotational and translational

components through a similar technique used in the eight-point algorithm. This approach

uses SVD to rewrite the homography matrix, as shown in Equation 3-59.

HTH = VEVT (3-59)

The matrix C = diag [o0a~ 21 -- ] nd ,mthe vetor V,- alt~~re nn otn orma nl eigenvector c~~ol-~rresodn to

the singular values of E. The columns of the matrix V can be written as V = [vl, v2, v3]. Defining

two other unit-length vectors, shown in Equation 3-60, that are preserved in the homography

mapping and will facilitate in the decomposition process.

vi +v3 vi-v3
U11= /2. = (3-60)

Furthermore, defining the matrices shown in Equation 3-61 will establish a homography

solution expressed in terms of these known variables.

Ui = [v2,111slT21/2] Wi = [HyI2-Hui~H 2Hul
[ ] (3-61)
U2 = 7Z2:l 82-1T22 W2 = [Hy2Hu(II2,H 2H2

The four solutions are shown in Table 3-1 in terms of the matrices given in Equations 3-61,

3-60 and the columns of the matrix V. Notice the translation component is estimated up to a 2

scale factor. This is the same scale ambiguity associated with the eight-point algorithm, which is

caused by the loss of depth during the image plane transformation.

Table 3-1. Solutions for homography decomposition
R1 = W1 Uz R3 = R1
Solution 1 NI1 = itul Solution 3 N3 = -NI1
T = (H-R1)Nl ci73 --2T1
R 2 = W2 U2 R 4 = R 2
Solution 2 N2~ = Vi2 Solution 4 N4 q
gT2 = (H -R2) N2 I f4 = -gf2

A unique solution for the homography is then found by imposing the positive depth

constraint, which is associated with the physically possible solution. This imposition involves

* A new approach to detecting moving objects in a sequence of images is developed. This
method computes estimates for the focus of expansion (FOE) and then classifies each
feature point into their respective motions through an iterative least-squares solution. The
decision scheme for classification maintains a cost function, which determines if a feature
point obeys a particular FOE, under a desired threshold. The dominant motion assumption is
then used to determine which FOE class is considered stationary objects in the environment
and which are associated with moving objects.

* The nonlinear dynamics for an aircraft-camera system are derived for a general camera
configuration and model. This structure allows multiple cameras with time varying positions
and orientations within the derivation to compute image plane quantities such as feature
point position and velocity.

* A new method for obtaining error bounds on the target state is established to provide a
region of where the estimate can lie from the effects of uncertainty. This method can be
described as a deterministic framework that computes upper bound uncertainty and was
implemented to describe variations to image plane coordinates and propagated through
vision based algorithms. Although this upper bound or worse-case approach to uncertainty
is a conservative technique, it provides a fast implementation scheme to account for
inaccurate camera calibration.

* The implementation of the homography of a moving target along with a model prediction
scheme will be incorporated into a controls framework to enable closed-loop tracking of an
unknown moving object of interest.

The first chapter of this thesis describes the motivation for this research, some current

objectives and limitations to address followed by a summary of the contribution and descriptions

of potential applications for this research.

Chapter 2 describes the related work and literature review that applies to this particular

research topic.

Chapter 3 introduces the foundation of computer vision and image processing. First the

camera geometry is described along with the projection model followed by the constraints used to

facilitate the estimation process. Lastly, traditional algorithms which estimate both the 3D motion

of the camera and the motion of targets are described.

Chapter 4 quantifies the effects of uncertainty in state estimation from variations in feature

point position caused from camera calibration and feature point tracking.

-1 )(

Time (sec) Time (sec)


Time (sec)

Figure 10-16. Attitude states of the reference vehicle (pursuit vehicle): A) Roll, B) Pitch, and C)

Time (sec)


Figure 10-17. Position states o




0 20 40 60
Time (sec)




Time (sec)

f the target vehicle (chase vehicle): A) North, B) East, and C)


20 40 60 10
Time (sec)

20 40 60 0
Time (sec)

20 40
Time (sec)

Figure 10-18. Attitude states of the target vehicle (chase vehicle): A) Roll, B) Pitch, and C) Yaw

motion from the UAV to the target of interest. The norm error of this motion are depicted in

Figure 10-19. These results indicate that with synthetic images and perfect tracking of the

vehicles nearly perfect motion can be extracted. Once noise in the image or tracking is introduced

the estimates of the target deteriorate quickly even with minute noise. In addition, image artifacts

such as interference and drop outs will also have an adverse affect on homography estimation.


'Yv Vertical angle for field of view

87 A variation in focal length

6d A variation in radial distortion

8,, A variation in pu

6v A variation in v

8 y A variation in optic flow

Sc A variation in the two-view feature point matrix

Sq A variation in the entries of the essential matrix

Sq, A variation in the two-view feature point matrix using the planar

homography matrix

Sh A variation in the entries of the planar homography matrix

6A A variation in the two-view feature point matrix using structure from


8: A variation to the depth components in two-view camera geometry

rl Position vector of feature point relative to and expressed in camera

coordinate frame I

rlF,n Feature point location on reference vehicle realtive and expressed in

camera-fixed coordinates

grl,, Feature point location on target vehicle realtive and expressed in

camera-fixed coordinates

TIVF,n Feature point location on reference vehicle realtive and expressed in
virtual coordinates

TIV,,, Feature point location on target vehicle realtive and expressed in virtual

pu Vertical coordinate in the image plane

pu (x) Mean operator of a vector x

Acceleration of the target in E

Body-fixed coordinate frame components

Position vector of camera center in camera-fixed coordinated frame

Radial distortion

Nominal radial distortion

Earth-fixed coordinate frame components

Focal length

Nominal focal length

Altitude state

Stacked column vector of the entries of the planar homography matrix

Altitude command

Nominal entries of the planar homography matrix

Image motion model

Camera-fixed coordinate frame components

Proportional gain on altitude error

Proportional gain on pitch rate

Proportional gain to roll rate

Proportional gain on the lateral position error

Integral gain on the lateral position error

Proportional gain on pitch

Proportional gain to roll

Proportional gain to heading error

Epipolar line in image i

Translation from camera-fixed to reference-fixed coordinates expressed

relative camera-fixed coordinates

a (t)



{&1l, e2, 83}

















The outer-loop that connects altitude to pitch commands is considered. The gains for the

inner-loop pitch tracking remained the same while the gain in altitude error was set to k = 1.25.

The final compensation filter is given in Equation 10-1 and was designed in Stevens et al. [110].

A step response for this controller is illustrated in Figure 10-25 that shows a steady climb with

no overshoot and a steady-state error of 2 ft. This response is realistic for an F-16 but not ideal

for autonomous refueling mission where tolerances are on the cm level. The altitude transition is

slow due to the compensator but one may consider more aggressive maneuvers for missions such

as target tracking that may require additional agility.

s2 + 0.35s +t 0.015
Gs2+-t2.41s+-t0.024 (01





0 20 40 60 80 100
Time (sec)

Figure 10-25. Altitude step response

The next stage that was tuned in the control design was the heading controller. The

inner-loop gains were chosen to be kg = -5.7 and kp = -1.6 for the roll tracker. The bode

diagram for this controller of roll command to roll angle is shown in Figure 10-26 which shows

attenuation in the lower frequency range. This attenuation removes any high frequency response

from the aircraft which is desired during a refueling mission, especially in close proximity.

Meanwhile, the coupling between lateral and longitudinal states during a turn was counteracted

is related to q as in Equation 3-44. The matricx C, shown in Equation 3-45, is a nx 9 matrix of

stacked feature points matched between two views.

Cq = 0 (3-44)

#1l,192,1 V1,19u2,1 #u2,1 91l,192,1 V1,192,1 V2,l #1l,1 V1,1 1

u1,2iu2,2 V1,2iu2,2 #u2,2 #u1,2V2,2 V1,2V2,2 V2,2 #u1,2 V1,2 1 3-5

i#1,niU2,n V1,niU2,n iU2,n i#1,nV2,n V1,nV2,n V2,n i#1,n V1,n 1
A unique solution for Equation 3-44 exists using a linear least-squares approach only if

the number of matched features in each frame is at least 8 such that rank(C) = 8. Additionally,

more feature points will obviously generate more constraints and, presumably, increase accuracy

of the solution due to the residuals of the least-squares. In practice, the least-squares solution to

Equation 3-44 will not exist due to noise, therefore, a minimization is used to find an estimate of

the essential matrix, as shown in Equation 3-46.

min||Cq||, ||q|| =1 (3-46)

Once an estimate of the essential matrix is found, the next step is to decompose this matrix

into its translational and rotational components. This decomposition is obtained through singular

value decomposition (SVD) of the essential matrix, and is shown in Equation 3-47.

Q = UEV' (3-47)

where E = diag {ol, 2, 03 } are the singular values. In general, this solution is corrupted

from noise and needs to be projected onto the essential space. This projection is performed

by normalizing the singular values to C = diag {1, 1,0} and adjusting the corresponding U

and V. The motion decomposition can now be obtained through Equation 3-48, where the

translation T is found up to a scaling factor. These four solutions, which consist of all possible

combinations of R and Tx, are checked to verify which combination generates a positive depth

in terms of the relative position with a lens offset, c, relative to the camera frame.

p = x x (3-5

If the origin of the camera frame is placed at the lens, (i.e., c = 0), Equations 3-5 and 3-6

reduce to the very common pin-hole camera model and is represented by Equations 3-7 and 3-8.

p = f Ex(3-7)

v =f (3-8)

This projection is commonly written as a map H:

H : RW3 ,2; X x (3-9)

The ideal perspective projection given in Equations 3-7 and 3-8 can be expressed in

homogeneous coordinates and is shown in Equation 3-10.

pu f 0 0 rlx

Ezv =0 f 0 Try (3-10)

1 0 0 1 rlz

3.2.2 Intrinsic Parameters

The image plane that is acquired from physical cameras is more complicated than the ideal

projection given in Equation 3-10. First, the image plane is discretized into a set of pixels,

corresponding to the resolution of the camera. This discretization is based on scale factors that

relate real-world length measures into pixel units for both the horizontal and vertical directions.

These scaling terms are defined as s, and sv which have units of pixels per length, where the

length could be in feet or meters. In general, these terms are different but when the pixels are

square then s, = sy. Second, the origin of the image plane is translated from the center of the

(R, T)

Figure 3-5. Geometry of the planar homography

If an assumption is made that the feature points are contained on the same plane, then a new

constraint involving the normal vector can be established. Denote N = [121,122,13 T as the normal

vector of the plane containing the feature points relative to camera frame 1. Then the projection

onto the unit normal is shown in Equation 3-50, where D is the projected distance to the plane.

N rll = nlrl,x+/t 2291,y ft l23T1,z= (3-50)

Substituting Equation 3-50 into Equation 3-49 results in Equation 3-51,

82=R+TNI qir (3-51)

where the planar homography matrix is defined to be the following

H = R t _TNT (3-52)

by an aileron-elevator connect. This connection involved a proportional gain of k, = 0.35

multiplied to the roll angle and added to the elevator position.

Bode Diagram
Gm = 15.9 dB (at 43.4 rad/sec) Pm = 179 deg (at 0.0583 rad/sec)
From Bankcmd(pt 1) To r2d(pt 1)

r -100


a, -90

10-4 10-2 100 102
Frequency (rad/sec)

Figure 10-26. Inner-loop roll to roll command Bode plot

The step response for this bank controller is illustrated in Figure 10-27. The tracking

performance is acceptable based on a rise time of 0.25 see, an overshoot of 6% and less than a

3% steady-state error.

The outer-loop tuning for heading controller consisted of first tuning the gain on heading

error. A gain of kw = 1.5 was chosen for this mission which demonstrated acceptable

performance. Figure 10-28 shows the heading response using this controller for a right turn.

The response reveal a steady rise time, no overshoot, and a steady-state error of less than 2 deg.

Finally, the loop pertaining to lateral deviation was tuned to k,, = 0.5 and kyi = 0.025 which

produced reasonable tracking and steady error for lateral position.

The final stage of the controller involves the axial position. This stage was designed to

increase thrust based on a velocity command once the lateral and altitude states were aligned.

A proportional gain was tuned based on velocity error to achieve a slow steady approach speed

image relative to an aircraft and then employing the moving object detection algorithm shown in

Chapter 6. Once moving objects in the image are detected, the homography estimation algorithm

proposed in this chapter is implemented for target state estimation.

7.2 State Estimation

7.2.1 System Description

The system described in this paper consists of three independently moving vehicles or

objects containing 6-DOF motion. To describe the motion of these vehicles a Euclidean space is

defined with five orthonormal coordinate frames. The first frame is an Earth-fixed inertial frame,

denoted as E, which represents the global coordinate frame. The remaining four coordinate

frames are moving frames attached to the vehicles. The first vehicle contains two coordinate

frames, denoted as B and I, to represent the vehicle's body frame and camera frame, as described

in Chapter 5 in Figure 5-1. This vehicle is referred to as the chase vehicle and is instrumented

with an on-board camera and GPS/IMU sensors for position and orientation. The second vehicle,

denoted as F, is considered a reference vehicle that also contains GPS/IMU sensors and provides

its states to the chase vehicle through a communication link. Lastly, the third vehicle, denoted

as T, is the target vehicle of interest in which unknown state information is to be estimated. In

addition, a fictitious coordinate frame will be used to facilitate the estimation process and is

defined as the virtual coordinate system, V.

The coordinates of this system are related through transformations containing both rotational

and translational components. The rotational component is established using a sequence of

Euler rotations in terms of the orientation angles to map one frame into another. Let the relative

rotation matrices REB, RBI, REF, REV, RIV, RFV, RTy and RET denote the rotation from E to B,

B to I, E to F, E to V, I to V, F to V, T to V, and E to T. Secondly, the translations are defined

as TEB, F, XV, XT, F,n, T,n, TBI, XIV, mFI mIT, TF,n N1T,n, W1VF, FVT, TVF,n, and rlvr,n which

denote the respective translations from E to B, E to F, E to V, E to T, E to the 12th feature point

on the reference vehicle and target vehicles all expressed in E, B to I expressed in B, I to V, I to

F, I to T, I to the 12th feature point on the reference and target vehicles expressed in I, V to F, V


Image processing and computer vision refers to the process of acquiring and interpreting

2-dimensional visual data to achieve awareness of the surrounding environment. This information

is used to infer spatial properties of the environment that are necessary to perform essential tasks

such as guidance and navigation through unfamiliar environments. An important breakthrough in

computer vision occurred when algorithms were able to detect, track, and estimate locations of

features in the environment.

This dissertation relies on feature points as the foundation for any vision-based feedback.

The term "features" allows one to establish a relationship between the scene geometry and

the measured image. These points generally correlate to items in the environment of special

significance. Some examples of items that often constitute feature points are corners, edges

and light sources. Such feature points can provide information about the overall object in the

sense that a set of corners can outline a building. Feature points do not necessarily provide

enough information to completely describe an environment but, in practice, they usually provide

sufficient information for target tracking and position estimation. To understand the algorithms

that use feature points, an establishment of the fundamental equations governed by the physics of

a camera will be described.

3.1 Camera Geometry

A camera effectively maps the 3-dimensional environment onto a 2-dimensional image

plane. This image plane is defined as the plane normal to the camera's central axis located a focal

length, f, away from the origin of the camera basis. The geometry provided by a pin-hole camera

lens is described in Figure 3-1. The vector, rl, represents the vector between the camera and a

feature point in the environment relative to a defined camera-fixed coordinate system, as defined

by I. This vector and its components are represented in Equation 3-1.

For feature points that are stationary in the environment, the translational optic flow

induced by the camera motion is constraint to radial lines emanating from the FOE, as shown

in Figure 6-2. Consequently, feature points that violate this condition can be classified as

independently moving objects. This characteristic observed from static features will be the basis

for the classification scheme.

1 r
r ,r
I ( I
I ~
r'r r


~, r
/* ir



Figure 6-2. FOE constraint on translational optic flow for static feature points

The residual optical flow may contain independently moving objects within the environment

that radiate from their own FOE. An example of a simple scenario is illustrated in Figure 6-3 for

a single moving object on the left and a simulation with synthetic data of two moving vehicles

on the right. Notice the two probable FOEs in picture on the left, one pertaining to the static

environment and the other describing the moving object. In addition, the epipolar lines of the two

distinct FOEs intersect at discrete points in the image. These properties of moving objects are

also verified in the synthetic data shown in the plot on the right. Thus, a classification scheme

must be designed to handle these scenarios to detect independently moving objects. The next

The modeling scheme presented in Chapter 8 provides a method to estimate targets in

Euclidean space when features do exit the image. This method works well for short periods of

time after the target has left; however, the trust in the predicted value degrades tremendously

as time increases. Consequently, when a feature leaves the image the controller can rely on the

predicted estimates to steer the aircraft initially but may resort to alternative approaches beyond a

specified time. As a last resort, the controller can command the aircraft to slow down and regain a

broader perspective of the scene to recapture the target.

differences between the axes in each coordinate system and the sequence of single-axis rotations.

In particular, the rotation order used for this transformation was a 3-2-1 sequence.

cos(e,) cos(Ve) sin(Qc) sin(ec) cos(Ve,) cos(Qc) sin(Ve,) cos(Qc) sin(ec) cos(Ve,) + sin(Qc) sin(Ve,)
RBI = cos(8c) sin(Vec) sin(#c ) sin(8c) sin(Ve,) + cos(#c) cos(Ve,) cos(#c) sin(8c) sin(Ve) sin(#c) cos(Ve,)

sin(ec) sin(Qc) cos(6c) cos(Qc) cos(6c)

The rates of change of these orientation angles are again required for coordinate frame

transformations. The roll rate, pe, is the angular velocity about 13, the pitch rate, q,, describes

rotation about i2, and the yaw rate, re, described rotation about ii. The vector, me, is given in

Eq. 5-8 to represent these angles.

OWe = rcil +t qc 2 tPc 3


5.2 System Geometry

The fundamental scenario involves an aircraft-mounted camera and a feature point in the

environment. This scenario, as outlined in Figure 5-3, thus relates the camera and the aircraft to

the feature point along with some inertial origin.

2, ~

Feature Point

Figure 5-3. Scenario for vision-based feedback

The bound on error, Az, can be expressed using Equation 4-29. This bound notes that the
bound on variations in feature points, and ultimately the bound on solutions to structure from
motion, depends on the location of those feature points.

Az n

|| (Ao + BA)-1( T -(Ao + BA) o) ||


Is1, P la
8#2 I < 2,

test the system under practical conditions. Additionally, incorporating the modeling scheme

presented in Chapter 8 into the refueling simulation will help the controller by providing state

estimate when the target exits the field of view.

Similarly to the previous example, vision-based feedback is generated using a flight

simulation. The overall setup of this example is the same where a nonlinear model of an F-16 is

used to fly through a cluttered environment while capturing images from an on-board camera.

Camera settings, such as focal length and field of view, are kept the same from the previous

example. The actual environment has been normalized based on the aircraft velocity so units are

not presented.

A set of obstacles, each with a feature point, are randomly distributed throughout the

environment and are not the same same as the previous example. This environment is shown in

Figure 10-4 along with a pair of points indicating the locations at which images will be captured.

The aircraft is initially straight and level then translates forward while rolling 4.0 deg and yawing

1.5 degp at the final location.


*. *
-10 00* ** o "
-1000 *0 0 00 50 2

0 **

Figure 10-4. Virtual environment of obstacles (solid circles) and imaging locations (open circles)
A) 3D view and B) top view

A single camera is simulated at the center of gravity of the aircraft with line of sight aligned

to the nose of the aircraft. The intrinsic parameters are chosen such that fo = 1.0 and do = 0.0

for the nominal values. The images for the nominal camera associated with the scenario in

Figure 10-4 are presented in Figure 10-5 to show the variation between frames.

The vision-based feedback is computed for a set of perturbed cameras. These perturbations

range as 87 E [-0.2, 0.2] and 6d E [-0.02, 0.02]. Obviously the feature points in Figure 10-5 will

vary as the camera parameters are perturbed. The amount of variation will depend on the feature

angles of the aircraft, (#, 6,11). The velocity of the aircraft's center of mass is vb and is defined in

Equation 5-27. As stated in Equation 5-27, the aircraft's velocity is expressed in the body-fixed

coordinate frame. Each of these parameters will appear explicitly in the aircraft- camera


vb = ub1 t +vb2+ twb3 (5-27)

The first six equations represent the force and moment equations, while the remaining

equations are kinematic relationships. The aerodynamic parameters consist of both the

aerodynamic forces, {K,, F, FZ}", on the aircraft and the aerodynamic moments, {L,1M, N}",

which are all contained in the force and moment equations. Although these equations do not

contain control inputs explicitly, the aerodynamic parameters are directly effected by the position

of the control surfaces on the aircraft. In other words, when the control surface deflections are

changed the flow over that surface also changes. This flow change over a surface results in

changes of the aerodynamic forces, such as lift and drag, which directly produce forces and

moments that roll, pitch, and yaw the aircraft and are described by the stability derivatives for

each aircraft. Therefore, controlled maneuvers are accomplished by changing these aerodynamic

parameters through the control surfaces.

An alternative approach to solving the nonlinear equations is to linearize these equations

about a trim condition using a Taylor series expansion. By linearizing these equations about a

level flight condition, the aircraft equations become decoupled into two planar motions. This set

of equations, although easy to solve, have limitations outside the chosen trim state, especially for

smaller more maneuverable aircraft. The choice of what set of aircraft equations to use depends

primarily on the aircraft and the application.

5.4 Aircraft-Camera System

The preliminary definitions established in the previous sections will now be used to

formulate the aircraft-camera system by using the systems described in this chapter. Here the

dependence of image plane position and velocity on the aircraft states along with the kinematic

3.6 Two-View Image Geometry ........ .. .. 56
3.6.1 Epipolar Constraint ........ .. .. 57
3.6.2 Eight-Point Algorithm ....... .. .. 59
3.6.3 Planar Homography .. . .... .. 61
3.6.4 Structure from Motion . ...... ... .. 65


4. 1 Feature Points ......... . .. .. 67
4.2 Optical Flow ......... ... .. 70
4.3 Epipolar Geometry ......... . .. .. 71
4.4 Homography ........ . ... .. 73
4.5 Structure From Motion ....... ... .. 75

5 SYSTEM DYNAMICS ......... . ... .. 77

5.1 Dyanmic States ......... . ... .. 77
5.1.1 Aircraft ............. ........... 77
5.1.2 Camera ............. ........... 79
5.2 System Geometry ......... . ... .. 81
5.3 Nonlinear Aircraft Equations ....... .. .. 83
5.4 Aircraft-Camera System ......... ... .. 84
5.4.1 Feature Point Position ....... .. .. 85
5.4.2 Feature Point Velocity ..... ... .. 85
5.5 System Formulation ........ .... .. 86
5.6 Simulating ....._.. . ... .. 89


6. 1 Camera Motion Compensation . ...... ... .. 90
6.2 Classification ......... . ... .. 95


7.1 Introduction ......... . ... .. 98
7.2 State Estimation . . ...... .01
7.2.1 System Description .............10
7.2.2 Homography Estimation .............10


8.1 Introduction .............. ............ 111
8.2 Dynamic Modeling of an Object.............11
8.2.1 Motion Models .............12
8.2.2 Stochastic Prediction .............13

9 CONTROL DESIGN ..............17

9. 1 Control Objectives .............17

subject to Equation 3-22. One important limitation of this criterion occurs when the window in

both images contains relatively constant intensity values. This results in the aperture problem

where a number of solutions for h are obtained. Therefore, during the feature selection process

it's beneficial to choose features that contain unique information in this window.

hi=argm~in ||li(i-I2((hSi))|2 (3-23)

There are two common techniques to solve Equation 3-23 for small baseline tracking: (1)

using the brightness consistency constraint and (2) applying the sum of squared differences

(SSD) approach. Each of these techniques employs a translational model to describe the image

motion. Therefore, if one assumes a simple translational model then the general transformation is

shown in Equation 3-24.

h(x) = x +t Ax (3-24)

The brightness consistency constraint is derived by substituting Equation 3-24 into

Equation 3-22 while initially neglecting the noise term. Applying the Taylor series expansion

to this expression about the point of interest, x, while retaining only the first term in the series

results in Equation 3-25.
aI dpu aI dv aI
aSp dt av dt tat (-5
This equation relates the spatial-temporal gradients to the pixel motion assuming the

brightness remains constant across images. Rewrting Equation 3-25 in matrix form results in

Equation 3-26.

AITu +t It = 0 (3-26)

where u = [f ].

Equation 3-26 constitutes 1 equation with 2 unknown velocities; therefore, another

constraint is needed to solve this problem. A unique solution for the velocities can be determined

by enforcing an additional constraint on the problem, which entails restraining regions to a local

window that moves at constant velocity. Upon these assumption one can minimize the error

- -Command

0 5 10 15 2(
Time (sec)

step response







d 10


Figure 10-27. Roll angle

0 20 40 60
Time (sec)

80 100

Figure 10-28. Heading response

to the target. A gain of ks = 3.5 was determined for this loop which generates the desired

approach. Lastly, to help limit the number of times the feature points exit the field of view a

limit was imposed on the pitch angle. This limit was enforced when the approach achieve a

specified distance. For this example, the distance was set to within 75 ft in the axial position of

the body-fixed frame which was determined experimentally from the target's size.

10.4.3 Closed-loop Results

The state estimation performance of the target drogue during this simulation was similar to

the previous simulation regarding the tracking of a ground vehicle. The estimated target states


The control strategy considered in this dissertation uses the computed relative states found

between a moving camera and a moving target of interest as shown in Chapter 7. Effectively,

these quantities are the error signals used for control to track the moving camera toward a desired

location based on the motion of the target. The framework presented here will use aircraft and

UAV navigation schemes for the aerial missions described in Chapter 1. Therefore, the control

design described in this chapter focuses on the homing mission to facilitate the AAR problem,

which involves tracking the position states computed from the homography.

Various types of guidance controllers can be implemented for these types of task once

the relative position and orientation are known. Depending on the control objectives and how

fast the dynamics of the moving target are, low pass filtering or a low gain controller may be

required to avoid high rate commands to the aircraft. In the AAR problem, the success of the

docking controller will directly rely on several components. The first component is the accuracy

of estimated target location which during AAR needs to precise. Secondly, the dynamics

of the drogue are stochastic. This causes the modeling task to be impractical in replicating

real life so the controller is limited to the models considered in the design. In addition, the

drogue's dynamics may not be dynamically feasible for the aircraft to track which may further

reduce performance. Lastly, the controller ideally should make position maneuvers in stages by

considering the altitude as one stage, the lateral position as another stage, and the depth position

as the final stage. In close proximity, the controller should implement only small maneuvers to

help maintain the vehicles in the FOV.

9.1 Control Objectives

The control objectives for the AAR mission is to track and home on the target drogue and

successfully dock with the receptacle. This controller is designed using a tracking methodology

that regulates the relative distance to within a specified tolerance. For example, the tolerance

required for aerial refueling is on the centimeter scale [15].

For a stationary feature point in space, the position vector, 5, is constant in magnitude and

direction and is expressed in the inertial frame; therefore, this time derivative is zero. Likewise,

the position vector of the aircraft's center of mass, TEB, is also expressed in the inertial basis and,

therefore, the time derivative just becomes ~TEB. Meanwhile, the Derivative Theorem is employed

on such terms as rl and TBI to express these terms in the moving frame. By applying this theorem

and solving for feature point velocity with respect to the camera frame, Equation 5-29 can now

be rewritten to Equation 5-30 for a non-stationary feature point.

'd(r) = ( ~TEB Bd(TBI) a~x TBI Eo Ix r (5-30)
dt dt

This equation can be reduced further if the cameras are constrained to have no translation

relative to the aircraft so 3(TEI) = 0. Alternatively, this term is retained in the derivation to allow

this degree of freedom in the camera setup. The angular velocity, E I, can be further decomposed

using the Addition Theorem. The final step implements Equations 5-5 and 5-13 to transform

each term into the camera frame. After some manipulation, the expression for the velocity of a

feature point relative to the camera frame results in Equation 5-31.

Ti = RBIREB ( TEBR RBITBI RBI( (x TBI) ((RBIO+ Wc) x 8) (5-31)

The image plane velocity of a feature point relative to the camera frame is finalized by

substituting both equations for position and velocity derived in Equation 5-28 and 5-31 into

Equations 3-32 and 3-33. This result will provide a description of the optical flow for each

feature point formed by either the camera traveling through the environment or the motion of the

feature points themselves. To incorporate radial distortion effects into the optic flow computation

requires the additional substitution into Equations 3-34 and 3-35.

5.5 System Formulation

The derivation of the aircraft-camera equations can be easily extended to systems with

multiple cameras all of which have their own position and orientation relative to the aircraft while

acquiring numerous feature points in each image. Although this adds computational complexity,

as GPS and IMUs. Second, the depth estimate can be acquired if some information is known

regarding the target geometry (e.g. a fixed distance on the target). For the case of stereo vision,

depth estimates can be obtained for each time step which is suitable for estimating the states of a

moving object. Although this particular configuration addresses the depth estimation, additional

issues involving the correspondence solution emerge when introducing multiple cameras [5].

Furthermore, the accuracy of the state estimates becomes poor for small baseline configurations,

which occurs for MAV using stereo vision. These issues regarding target state estimation will be

considered in this dissertation to show both the capabilities and limitations toward autonomous

control and navigation.

Another important task involved with target estimation is to determine a pattern (if

any) in the object's motion based on the time history. The objects can then be classified into

deterministic and stochastic motions according to past behavior. With this information, prediction

models can be made based on previous images to estimate the position of an object at a later time

with some level of confidence. The predicted estimates can then be used in feedback for tracking

or docking purposes. For stochasticly classified objects, further concerns regarding docking or

AAR are imposed on the control problem.

The primary task of state estimation, for both the vehicle and objects in the environment,

relies on accurate knowledge of the image measurements and the associated camera. Such

knowledge is difficult to obtain due to uncertainties in these measurements and the internal

components of the camera itself. For instance, the image measurements contain uncertainties

associated with the detection of objects in the image, in addition to noise corruption. These

drawbacks have prompted many robust algorithms to increase the accuracy of feature detection

while handling noise during the estimation process. Alternatively, many techniques have been

used to accurately estimate the internal parameters of the camera through calibration. The

parameters that describe the internal components of the camera are referred to as intrinsic

parameters and typically consist of focal length, radial distortion, skew factor, pixel size, and

optical center. This calibration process can become cumbersome for a large number of cameras

3.2.4 Radial Distortion

Other nonlinear camera effects that are not accounted for in the pin-hole model, such as

radial distortion, can be addressed through additional terms. A standard lens distortion model is

considered to account for such nonlinearities in the camera. The general distortion term, given in

Equation 3-14, requires an infinite series of terms to approximate the value.

d = dr r2+-td2r4+-td3r,6+--- HOT (3-14)

The distortion model, shown in Equations 3-15 and 3-16, maps an undistorted image,

(p', v'), which is not measurable on a physical camera, into a distorted image, (p'd d&), which

is observable [104]. This distortion model only considers the first term in the infinite series to

describe radial distortion and excludes tangential distortion. This approximation in distortion has

been used to generate an accurate description of real cameras without additional terms [105],

p'd = v'(1+t dr2) (3-15)

vid = p'(1+ dr2) (3-16)

where r2 __ /1 C1)2 /t _V 2)2 and d is the radial distortion parameter of the camera. Assuming

the origin of the camera frame is placed at the lens, then this term becomes r2 __ 1u2 +t v'2.

In addition, the radial distortion parameter, d, which is not described in Figure 3-1, attempts

to model the curvature of the lens during the image plane mapping. This distortion in the image

plane varies in a nonlinear fashion based on position. This effect demonstrates an axisymmetric

mapping that increases radially from the image center. An example can be seen in Figure 3-3B

and 3-3C which illustrates how radial distortion changes feature point locations of a fixed

pattern in the image by comparing it to a typical pin-hole model shown in Figure 3-3A. Notice

the distorted images seem to take on a convex or concave shape depending on the sign of the


vector [102]. The same constraint holds for the image coordinates as well but also introduces an

unknown scale factor. Employing this constraint, estimates of relative motion can be acquired

for both camera-in-hand and fixed camera configurations. This dissertation deals with the

camera-in-hand configuration while assuming a perfect feature point detection and tracking

algorithm. This assumption enables the performance of the vision based state estimation to be

tested before introducing measurement errors and noise.

The homography constraint requires a few assumptions based on the quantity and the

structure of the feature points. The algorithm first requires a minimum of four planar feature

points to describe each vehicle. This requirement enables a unique solution to the homography

equation based on the number of unknown quantities. The reference vehicle will have a minimum

of four pixel values in each image which will be defined as pF,n = CUF,n-,VF,n] Vn2 feature points.

Likewise, the target vehicle will have four pixel values and will be defined as pr,n = [PT,n, VT,n] Vn2

feature points. This array of feature point positions are computed at 30 Hz which is typical for

standard cameras and the frame count is denoted by i. The final requirement is a known distance

for both the reference and target vehicle. One distance represents the position vector to a feature

on the reference vehicle in Euclidean space relative to the local frame F and the second distance

represents the position vector to a feature on the target vehicle in Euclidean space relative to

the local frame T. In addition, the length of these vectors also must be equal which allows the

unknown scale factor to be determined. The vector describing the reference feature point will be

denoted aS SF expressed in F, while the vector describing the target feature point is referred to as

sT expressed in T. These feature point position vectors are also illustrated in Figure 7-2.

The feature points are first represented by position vectors relative to the camera frame,

I. The expressions for both the reference and target feature points are given in Equations 7-3

and 7-4. These vector components are then used to compute the image coordinates given in

Equations 7-1 and 7-2. The computation in Equation 7-4 requires information regarding the

target which is done solely to produce image measurements that normally would be obtained

from the sensor. Remaining computations, regarding the homography, will only use sensor

myr Translation from camera-fixed to target-fixed coordinates expressed

relative camera-fixed coordinates

mVF Translation from virtual to refemce-fixed coordinates expressed relative

virtual coordinates

my, Translation from virtual to target-fixed coordinates expressed relative

virtual coordinates

op Vertical image offset from center to upper left corner in pixel units

ov Horizontal image offset from center to upper left corner in pixel units

p (t) Position of the target in E

pVF Image coordinates in the virtual camera of the reference vehicle

pyr Image coordinates in the virtual camera of the target vehicle

q Stacked column vector of the entries of the essential matrix

qo Nominal entries of the essential matrix

sqc Vertical unit length to pixel scaling

sv Horizontal unit length to pixel scaling

so Image skew factor

u Time rate of change of (p, v)

v (t) Velocity of the target in E

Vb = (u, v, w) Velocity of the body-fixed frame (velocity of the aircraft in body-fixed


vc = (uc, Ve, Wc) Velocity of the camera-fixed frame along {$1,82, 3 } axes

w (t) Random vector

I Subset image specified by W

xrv Translation from camera-fixed to virtual coordinates expressed in

camera-fixed coordinates

0.4 ~ '* 0.4

0.2 0.2

-0.4~ 0.4

-0.6~ 0.6
-0.6 -0.4 -0.2 0 0.2 0.4 0.6 -0.6 -0.4 -0.2 0 0.2 0.4 0.6
v v


Figure 10-5. Feature points for A) initial and B) final images

point, as noted in Equations 4-9 and 4-10, but the effect can be normalized. The variation in

feature point given nominal values of Po = vo = 1 is shown in Figure 10-6 for variation in both

focal length and radial distortion. This surface can be scaled accordingly to consider the variation

at other feature points. The perturbed surface shown in Figure 10-6 is propagated through three

main image processing techniques for analysis.

d 2 0.

10.2.2 Otic Flo

Figure 10-6 t foceral lngth and ftrada doistorin ersnaiecmaio fotcfo o

the nominal camera and a set of perturbed cameras is shown in Figure 10-7.

image is a feature point which indicates some pixel of particular interest due to, for example,

color or intensity gradient near that pixel. These intensity variations correlate well to physical

features in the environment such as comers and edges which describe the character of buildings

and vehicles within a scene as described in Chapter 3. Among the techniques that utilize feature

points, the approach related to this paper involves epipolar geometry [39, 112]. The purpose of

this technique is to estimate relative motion based on a set of pixel locations. This relative motion

can describe either motion of the camera between two images or the relative distance of two

objects of the same size from a single image.

The 3D scene reconstruction of a moving target can be determined from the epipolar

geometry through the homography approach described in Chapter 3. For the case described in

this chapter, a moving camera attached to a vehicle observes a known moving reference object

along with an unknown moving target object. The goal is to employ a homography vision-based

approach to estimate the relative pose and translation between the two objects. Therefore, a

combination of vision and traditional sensors such as a global positioning system (GPS) and

an inertial measurement unit (IMU) are required to facilitate this problem for a single camera

configuration. For example in the AAR case, GPS and IMU measurements are available for both

the receiver and tanker aircraft.

In general, a single moving camera alone is unable to reconstruct the 3D scene containing

moving objects. This restriction is due to the loss of the epipolar constraint, where the plane

formed by the position vectors relative to two camera positions in time to a point of interest

and the translation vector is no longer valid. Techniques have been formulated to reconstruct

moving objects viewed by a moving camera with various constraints [35, 113-116]. For instance,

a homography based method that segments background from moving objects and reconstructs

the target's motion has been achieved [117]. Their reconstruction is done by computing a virtual

camera which fixes the target's position in the image and decomposes the homography solution

into motion of the camera and motion caused by the target. This decomposition is done using a

planar translation constraint which restricts the target's motion to a ground plane. Similarly, Han

can be used. First, two solutions can be eliminated by using the positive depth constraint. The

decision regarding the remaining two solutions is more difficult to decipher unless the normal

vector is known or can be estimated, which in this case is known. Recall the normal vector,

n, describes the plane containing the feature points of the reference vehicle. As a result, the

homography solution is determined uniquely.

The final step in this development is to use the homography solution to solve for the relative

translation and rotation from T to B. The resulting equation for the rotation uses a sequence of

transformations and is shown in Equation 7-31.

RTB = RBIREBRETVRRBI (i 1) REB (i 1) RETF (i 1) (7-31)

The translation is found through a series of scalings followed by a vector sum. The relative

translation, xh, is first multiplied by D to scale distance which is given in Equation 7-29 to obtain

x. Secondly, x is then divided by oc to scale the depth ratio resulting in the final x expressed in I.

This result in conjunction with R is then used in Equation 7-22 to solve for myr. The next step is

to compute the relative translation from I to V which is given in Equation 7-32.

my=RE V EB+ETBTI (7-32)

The relative translation from T to B is then given in Equation 7-33.

xTB = REBRETV (mvT mIV) (7-33)

In conclusion, Equations 7-31 and 7-33 represent the relative motion between the camera

vehicle and the target vehicle. This information is valuable for the control tasks described earlier

involving both tracking and homing applications. The next section will implement this algorithm

in simulation to verify the state estimator for the noise free case.


[1] Secretary of Defense, "Unmanned Aircraft Systems Roadmap 2005-2030," website:
http:/uay. navair:navy, mil/roadmap05/roadmap. htm

[2] Grasmeyer, J. M., and Keennon, M. T., "Development of the Black Widow Micro-Air
Vehicle," 39th Aerospace Sciences 1Meeting and Exhibit, AIAA 2001-0127, Reno, NV,
January 2001.

[3] Beard, R., Kingston, D., Quigley, M., Snyder, D., Christiansen, R., Johnson, W., Mclain,
T., and Goodrich, M., "Autonomous Vehicle Technologies for Small Fixed Wing UAVs,'
AIAA Journal ofAerospace Computing, Information, and Communication, Vol. 2, No. 1,
January 2005, p. 92-108.

[4] Kingston, D., Beard, R., McLain, T., Larsen, M., and Ren, W., "Autonomous Vehicle
Technologies for Small Fixed Wing UAVs, AIAA 2nd Unmanned Unlimited Systems,
Technologies, and Operations-Aerospace, Land, and Sea Corkreticcr, and Workshop and
Exhibit, AIAA-2003-6559, San Diego, CA, September 2003.

[5] Frew, E., "Observer Trajectory Generation for Target-Motion Estimation Using Monocular
Vision," PhD Dissertation, Stanford University, August 2003.

[6] Sattigeri, R., Calise, A. J., Soo Kim, B., Volyanskyy, K., and Nakwan, K., "6-DOF
Nonlinear Simulation of Vision-based Formation Flight," AIAA Guidance, Navigation and
Control C~r~r,rirtic and Exhibit, AIAA-2005-6002, San Francisco, CA, August 2005.

[7] Beard, R., Mclain, T., Nelson, D., and Kingston, D., "Decentralized Cooperative Aerial
Surveillance using Fixed-Wing Miniature UAVs," IEEE Proceedings: Special Issue on
IMulti-Robot Systems, Vol. 94, Issue 7, July 2006, pp. 1306-1324.

[8] Wu, A. D., Johnson, E. N., and Proctor, A. A., "Vision-Aided Inertial Navigation for
Flight Control," A1AA Guidance, Navigation, and Control ~r-rTkreticcl and Exhibit, AIAA
2005-5998, San Francisco, CA, August 2005.

[9] Ettinger S. M., Nechyba, M. C., Ifju, P. G., and Waszak, M., "Vision-Guided Flight
Stability and Control for Micro Air Vehicle," IEEE/RSJ international C~rr r, rrtic on
Intelligent Robots and System, Vol. 3, September/October 2002, pp. 2134-2140.

[10] Kehoe, J., Causey, R., Abdulrahim, M., and Lind, R., "Waypoint Navigation for a Micro
Air Vehicle using Vision-Based Attitude Estimation," AIAA Guidance, Navigation, and
Control C~r~r,rirtic and Exhibit, AIAA-2005-6400, San Francisco, CA, August 2005.

[11] Abdulrahim, M., and Lind, R., "Control and Simulation of a Multi-Role Morphing
Micro Air Vehicle," AIAA Guidance, Navigation and Control ~r-rTkreticcl and Exhibit,
AIAA-2005-6481, San Francisco, CA, August 2005.

[12] Abdulrahim, M., Garcia, G., Ivey, G. F., and Lind, R., "Flight Testing of a Micro Air
Vehicle Using Morphing for Aeroservoelastic Control," A1AA Structures, Structural
Dynamics, and Materials Cr itreticc,. AIAA-2004-1674, Palm Springs, CA, April 20)04.

20 1 1 0 20 20 1 0 2

5 > 5 **5 **-0.0

Fiogur 3-3. Radifal DisgtortionEfcs foria dsA)c f s = ls 0.5 te di = 0,liea B)ai i fo = d=-0005 n

Theia cameora is Ceffectivey modeled usin the fetr oa lngth and radial distoration aong wither

thae ohrterms. dhpesr ibe d in s Eqa tiohn 311Auch those farmers arun ed termeny oad theinrsc

itis parameters and arebise f ounde thouh albr tion. Ant feature point mut eanlze ith respett

these ~ ~ ~ ~ ~ inrni aaeest nuepoersatue estimatin DTherdaldsanefrmafetr

pit t he cent ter of the esimage is dependent on bot the relatv oit t e ions o caeraand features

alohngwt the focwable lniength This radial distance isalso related s viat anonlineare rltinhip s toth

raia distcortison.nCleaprobe fly any nlsso h fatcuraely poins wireuirsiation otie of thlecamera

peatraeers Chapoter 4 wnille dicussatertchnwiqetharset consider bondedn uncertainty toard the

inrisicy pramonie tersuu an esalshqnes af bounded conditionong thfatue poinestht psitios.y

Thes fritrst stepi thoe estimabutiohn pobemi rqirtes teaiity trcoor dtct intcerestig features,

and curves. These features usually correlate to objects of interest in the environment such as

buildings, vehicles, bridges, etc. Although this gradient-based criterion is good at detecting

these features, it also produces a large number of detections from highly textured surfaces that

The next model considered involves a random motion model. The assumed acceleration is

shown in Equation 8-4 and is characterized by a random vector, w (t) and is scaled by a constant,

p. The velocity corresponding to this acceleration is described in Equation 8-5. This model

attempts to capture the stochastic behaviors by utilizing a probabilistic distribution function.

a (t) = pw (t) (8-4)

v (t) = v (t A) + p w (T) d (8-5)

Alternatively, the model shown in Equation 8-4 can be modified to incorporate some

dependence on the previous acceleration value. This dependence is achieved by weighting the

previous acceleration in the model and is shown in Equation 8-6. The benefit to this type of

model as oppose to Equation 8-4 requires some knowledge of the target; namely, that the target

cannot achieve large abrupt changing in acceleration. The resulting velocity expression for this

model is given in Equation 8-7.

a (t) = poa (t At) +t p w (t) (8-6)

v (t) = v i+ JN(t A) + a( -A) w(T) dt (8-7)

8.2.2 Stochastic Prediction

The image sequence obtained from the camera are processed by the homography to obtain

the position estimates of the target. These position estimates are then used to compute a velocity

profile of the target, as shown in Equation 8-8 for the ith target and N image frames. The velocity

profile is computed using a backwards difference method and is given in Equation 8-9.

[vi (t -1) ,vi(t -2) ,...,vi (t N+1 ),vi (t -N)] (8-8)

vi (t j) = pi (t j) pi (t j 1) (8-9)

Similarly, an acceleration profile, defined in Equation 8-10, is obtained from the velocity

profile given in Equation 8-8. The same backwards difference method is used to compute this

10.2.4 Structure From Motion

The images taken during the simulation are analyzed using structure from motion to

determine the location of the environmental features. The initial analysis used the nominal

camera to ensure the approach is able to correctly estimate the locations in the absence of

unknown perturbations. The actual and estimated locations are shown in Figure 10-10 to indicate

that all errors were less than 10-6

a Estimate

0- ..
N. *
-500 *

1000\ -

Figure 10-10. Nominal estimation using structure from motion

The depths are also estimated using structure from motion to analyze images from the

perturbed cameras. A representative set of these estimates are shown in Figure 10-11 as having

clear errors. An interesting feature of the results is the dependence on sign of the perturbation to

focal length. Essentially, the solution tends to estimate a depth larger than actual when using a

positive perturbation and a depth smaller than actual when using a negative perturbation. Such a

relationship is a direct result of the scaling effect that focal length has on the feature points.

Estimates are computed for each of the perturbed cameras and compared to the nominal

estimate. The worst-case errors in estimation are compared to the theoretical bound, given in

Equation 4-29, to these errors. These numbers shown in Table 10-8 indicate the variation in

structure from motion depends on the sign of the perturbation. The approach is actually seen

to be less sensitive to positive perturbations, which causes a larger estimate in depth, than to

negative perturbations. Also, the theoretical bound was greater than, or equal to, the error caused

by each camera perturbation.

processing algorithms such as SFM are not valid for moving objects viewed by a moving camera.

This limitation is caused by the epipolar constraint no longer maintaining a coplanar property

across image sequences; consequently, research has evolved for the detection of moving objects

in a scene viewed by a non-stationary camera.

The detection of moving object in an image sequence is an important step in image analysis.

For cases involving a stationary camera, simple image differencing techniques are sufficient in

determining moving objects [19-21]. Techniques for more realistic applications involve Kalman

filtering [22] to account for lighting conditions and background modeling techniques using

statistical approaches, such as expectation maximization and mixture of Gaussian, to account

for other variations in real-time applications [23-28]. Although these techniques work well for

stationary cameras, they are insufficient for the case of moving cameras due to the motion of the

stationary background.

Motion detection using a moving camera, as in the case of a camera mounted to a vehicle,

becomes significantly more difficult because the motion viewed in the image could result from

a number of sources. For instance, a camera moving through a scene will view motions in the

image caused by camera induced motion, referred to as egomotion, changes in camera intrinsic

parameters such as zoom, and independently moving objects. There are two classes of problems

considered in literature for addressing this topic. The first considers the scenario where the 3D

camera motion is known a priori then compensation can be made to account for this motion to

determine stationary objects through an appropriate transformation [29, 30]. The second class of

problems does not require knowledge of the camera motion and consists of a two stage approach

to the motion detection. The first stage involves camera motion compensation while the last stage

employs image differencing on the registered image [31] to retrieve non-static objects.

The transformation used to account for camera motion is commonly solved by assuming the

majority of image consists of a dominant background that is stationary in Euclidean space [32,

33]. This solution is obtained through a least-squares minimization process [32] or with the

use of morphological filters [34]. The transformations obtained from these techniques typically

where the individual single-axis rotations are commonly referred to as 3-2-1, or roll-pitch-yaw,

[el(#) 2 6) e3(w)] respectfully. The full rotation matrix is represented by Equation 5-3.

cos(0) cos(y) sin( ) sin(0) cos(W) cos( ) sin(W) cos( ) sin(0) cos(W) + sin( ) sin(W)

REB = COS(8) Sin(W) Sin( ) Sin(8) Sin(W) COS( ) COS(W) COS( ) Sin(8) Sin(W) Sin( ) COS(W)

sin(0) sin( ) cos(0) cos( ) cos(0)

The rates of change of these orientation angles also require a coordinate transformation. The

roll rate, p, is the angular velocity about bl, the pitch rate, q, describes rotation about b2, and the

yaw rate, r, describes rotation about $3. The vector, m,, is given in Equation 5-4 to represent these


o,= p61+-tq62 -t r3 (5-4)

5.1.2 Camera

The camera is also described using a right-handed coordinate system defined using

orthonormal basis vectors. The axes, as shown in Figure 5-2, use the traditional choice of i3

aligning through the center of view of the camera. The remaining axes are usually chosen with

12 aligned right of the view and it aligned out the top although some variation in these choices

is allowed as long as the resulting axes retain the right-handed properties. The direction of the

camera basis vectors are defined through the camera's orientation relative to the body-fixed

frame. This framework is noted as the camera-fixed coordinate system because the origin is

always located at a fixed point on the camera and moves in the same motion as the camera.

The camera is allowed to move along the aircraft through a dynamic mounting which

admits both rotation and translation. This functionality enables the tracking of features while

the vehicle moves through an environment. The origin of the camera-fixed coordinate system is

attached to this moving camera, consequently, the camera-fixed frame is not an inertial reference.

A 6 degree-of-freedom model of the camera is assumed which admits a full range of motion.

Figure 5-2 also illustrates the camera's sensing cone which describes both the image plane and

the field of view constraint.

the ith row of this matrix can then be written as Equation 4-21.

6iu = ~I,;+2-tpU 11 pillig V16p, -t26v1 8vi,; 8,;,?~ L (4-21)

U16,v V28pi -t ivz V18vz V26vl -t vl vz

A solution to Equation 3-57, when including the uncertain matrix in Equation 4-20, will

exist, however, that solution will differ from the true solution. Essentially, the solution can be

expressed as the nominal solution, ho, and an uncertainty, Sh, as in Equation 4-22.

('Fo +t 8,) (ho +t Sh) = 0 (4-22)

The solution vector, h = ho +t Sh, for Equation 4-22 has variation which will be norm

bounded by Ah as in Equation 4-23.

h = {ho+8 h 16 8h ~h} (4-23)

The size of this uncertainty, which reflects the size of error in the state estimation, can

be bounded using Equation 4-24. This bound uses the relationship between uncertainties in

Equation 4-21 through the constraint in Equation 4-22. Also, the size of this uncertainty depends

on the location of each feature point so the bounds is noted as specific to the rll and rl2 obtained

from Figure 3-4.

Ahx = I (Fo 6v)-1 who|| (4-24)

8v, < av,

Iav1 < Av2

The maximum variation of the entries of h = ho +t Ah, determined through Equation 4-24,

can then be used directly to compute the variation in state estimates. The entries of h are first

arranged back into matrix form to construct the new homography matrix that includes parameter

5.3 Nonlinear Aircraft Equations

The equations of motion of an aircraft can be represented in several different fashions.

The most general form of the aircraft equations are the nonlinear, highly coupled equations of

motion. These equations of motion are the standard equations which have been derived in a

typical aircraft mechanics book [108-110] and are repeated in Equation 5-14 to 5-26 for overall


P' mg sin O = m(ui + qw ry) (5-14)

F, + mg cos 6 sin = m(v + ru pw) (5-15)

F, + mg cos ecos = m(wi,+ py qu) (5-16)

L= A -Grqr(z -l) -1-!4(5-17)

M =Ir,q+rp(l,- I,) + ,(P2_ -2) (5-18)

N = -IeI + Ir~+ pq(1-I,-) + zqr (5-19)

p = #- isine (5-20)

q = O cos #+ ~icos 8sin (5-21)

r = Qicos ecos @- 0sin (5-22)

0= qcos -rsin~ (5-23)

S= p +q9sin tan O+ rcos ~tan 8 (5-24)

i = (q sin + r cos #) sec 0 (5-25)

=xd CeS, S4SecwcS, +CC, CSeS, -S4C, v (5-26)

dZb -Se S4Ce C4Ce

The shorthand notation for Sw sinW, Cw cosy, So sin6, Co cos6, and Sq sing,

C4 cos # is used in Equation 5-26.

The aircraft states of interest for the camera motion system consist of the position and

velocity of the aircraft's center of mass, TEB and Vb, the angular velocity, co,, and the orientation

Ci = {(Oli, Vi) if A (iPi, Vi) < Jmax }



Hi={(ii)if J2 (Pui,Vi) > Jmax} (6-13)

After all n feature points have been examined under this criterion, a set of m feature points

are classified to the static background, C. Meanwhile, a set of n m feature points are classified

as objects disobeying the static trend, 'L, and are considered moving objects. The class of moving

objects can be further classified into distinct objects through a clustering method. This method

removes all static features and uses the intersections of the epipolar lines pertaining to moving

objects as data points in the clustering algorithm. The resulting data will produce distinct clusters

around the FOEs pertaining to moving objects. The threshold Jax is a design parameter that

segments the feature points into their respective classes and needs to be tuned to account for

measurement noise.

Chapter 5 derives the system dynamics for an aircraft-camera configuration by formulating

the differential equations and observations into a controls framework.

Chapter 6 describes a method that utilizes image processing techniques to detect and

segment moving objects in a sequence of images.

Chapter 7 formulates a homography technique that estimates relative position and

orientation with respect to a moving reference object. The method fuses traditional guidance

and navigation sensors with the computed homography to obtain relative state estimates of the

unknown object with respect to the moving camera. This process applies directly to solving a

significant portion of the AAR problem.

Chapter 8 summaries a modeling technique for moving objects to predict the target's motion

in inertial space.

Chapter 9 discusses a control design scheme that exploits vision-based state estimates to

track and home on a desired target. The control framework will be generalized for many mission

scenarios involving autonomous UAV but will be discussed in the context of the AAR problem.

Chapter 10 will implement in MATLAB the vision algorithms for both open and closed-loop

architectures to demonstrate and verify the purposed methods.

Chapter 11 discusses concluding remarks and proposes future research directions for this


for the first iteration. It is assumed for the first iteration that the two features are static. The

least-squares solution is then given in Equation 6-9 for the FOE coordinates (Pfoetv,Vfoe) (for

the first iteration a least-squares solution is not necessary because two lines intersect at a single

point) .

= ar mm | b ||2 (6-9)


1M= I # ##2l Prz Vi+1 (6-10a)

-1 -1 -1

b ~ ~ ~ t = z- p 2 2 "" Vi+1 A Pi 1 (6-10b)

The next iteration adds another feature into the system of equations and a new potential FOE

point is obtained. If the new feature point is a static feature, then the new estimated FOE will be

near the static FOE, which is found in the first iteration, causing a small residual. Alternatively,

if the feature is point is due to a moving object then the epipolar line will not intersect the

static FOE and shift the solution causing a large residual. Defining the new FOE coordinates as

(pfoue27 1982). A cost function is then checked to verify if the new feature point contains a similar

motion to that of the static background by checking the residual. This residual is defined as the

Euclidean distance from the two FOE solutions found before and after adding the next feature.

If cost is higher than some maximum threshold Jmax then the feature point is discarded into a set

of points classified as moving, fl; else, the feature point is classified into the static FOE solution,

C. This process is repeated until all n feature points have been checked using this cost function

which is shown in Equation 6-11 for the ith iteration. Mathematically, the classification scheme

for the ith iteration is given in Equations 6-12 and 6-13.

12 PVi Pfei -Pfoi-1 2 foei-Vfei-) 2(6-11)

drogue, especially moments before the docking phase. The aerodynamic data acquired in these

experiments confirmed several dependencies on turbulence, flight conditions, and geometry.

2.4 Uncertainty in Vision Algorithms

The location of environmental features can be obtained using structure from motion. The

basic concepts are mature but their application to complex problems is relatively limited due to

complexities of real-time implementation. In particular, the noise issues involved with camera

calibration and feature tracking cause considerable difficulties in reconstructing 3-dimensional

states. A sampling-based representation of uncertainty was introduced to investigate robustness

of state estimation [82]. Robustness was also analyzed using a least-square solution to obtain an

expression for the error in terms of the motion variables [83].

The uncertainty in vision-based feedback is often chosen as variations within feature

points; however, uncertainty in the camera model may actually be an underlying source of

those variations. Essentially, the uncertainty may be associated with the image processing

to extract feature points or with the camera parameters that generated the image. The proper

characterization of camera uncertainty may be critical to determine a realistic level of feature

pomnt uncertainty.

The analysis of camera uncertainty is typically addressed in a probabilistic manner. A

linear technique was presented that propagates the covariance matrix of the camera parameters

through the motion equations to obtain the covariance of the desired camera states [84]. An

analysis was also conducted for the epipolar constraint based on the known covariance in the

camera parameters to compute the motion uncertainty [85]. A sequential Monte Carlo technique

demonstrated by Qian et al. [86] proposed a new structure from motion algorithm based on

random sampling to estimate the posterior distributions of motion and structure estimation. The

experimental results in this paper revealed significant challenges toward solving for the structure

in the presence of errors in calibration, feature point tracking, feature occlusion, and structure


10.2.3 The Epipolar Constraint

State estimation is performed by considering the epipolar constraint to relate the pair of

images. The evaluation of images generated using the nominal camera for this simulated case is

able to estimate the correct states. An investigation of the epipolar lines shown in Figure 10-8

shows the quality of the estimation. Essentially, the epipolar geometry requires a feature point

in one image to lie along the epipolar line. This epipolar line is constructed by the intersection

between the plane formed by the epipolar constraint and the image plane at the last measurement.

The data in Figure 10-8 show the features in the second image do indeed lie exactly on the

epipolar lines.

0.4~ '* + 0.4~ .

-0.2 ~ -0.2

-0.4~ 0.4

-0.6 -0.4 -0.2 0 0.2 0.4 0.6 -0.6 -0.4 -0.2 0 0.2 0.4 0.6
v v

Figure 10-8. Epipolar lines between two image frames: A) initial frame and B) final frame with
overlayed epipolar lines for nominal camera

The introduction of uncertainty into the epipolar constraint will cause variations in the

essential matrix which will also propagate through the computation of the epipolar line. These

variations in the epipolar line are visual clues of the quality of the estimate in the essential

matrix. These variations can occur as changes in the slope and the location of the epipolar line.

Figure 10-9 illustrates the epipolar variations due to perturbations on 87 = 0.1 and 6d = 0.01 to

the camera parameters. The feature points with uncertainty and the corresponding epipolar line

was plotted along with the nominal case to illustrate the variations. The key point in this figures

is the small variations in the slope of the epipolar lines and the significant variations in feature

[25] Sheikh, Y., and Shah, M., Bayesian Object Detection in Dynamic Scenes," IEEE
Computer Society C~rreakican on Computer Vision and Pattern Recognition, San Diego,
CA, June 2005.

[26] Stauffer, C., and Grimson, W. E. L., "Adaptive Background Mixture Models for Real-Time
Tracking," IEEE CriC irthe, on Computer V/ision and Pattern Recognition, Fort Collins,
CO, June 1999, pp. 246-252.

[27] Toyama, K., Krumm, J., Brumitt, B., and Meyers, B., "Wallflower: Principles and Practice
of Background Maintenance", International C~r rthrn ,l on Computer Vision, Corfu,
Greece, September 1999.

[28] Zhou, D., and Zhang, H, "Modified GMM Background Modeling and Optical Flow for
Detection of Moving Objects," IEE International C~r ratherrn on System, Man, and
Cybernetics, Big Island, Hawaii, October 2005.

[29] Nelson, R. C., "Qualitative Detection of motion by a Moving Observer," International
Journal of Computer Vision, Vol. 7, No. 1, 1991, pp. 33-46.

[30] Thompson, W. B., and Pong, T. G., "Detecting Moving Objects," International Journal of
Computer Vision, Vol. 4, 1990, pp. 39-57.

[31] Odobez, J. M., and Bouthemy, P., "Detection of Multiple Moving Objects Using
Multiscale MRP With Camera Motion Compensation," IEEE International C~rrTkratec, on
Image Processing, Austin, TX, November 1994, pp. 245-249.

[32] Irani, M., Rousso, B., and Peleg, S., "Detecting and Tracking Multiple Moving Objects
Using Temporal integration," European C~rr rthrrn l on Computer Vision, Santa Margherita
Ligure, Italy, May 1992 pp. 282-287.

[33] Torr, P. H. S., and Murray, D. W., "Statistical Detection of Independent Movement from a
Moving Camera," Image and Computing, Vol. 11, No. 4, May 1993, pp. 180-187.

[34] Gandhi, T., Yang, M. T., Kasturi, R., Camps, O., Coraor, L., and McCandless, J.,
"Detection of Obstacles in the Flight Path of an Aircraft," IEEE Transactions on
Aerospace and Electronic Systems, Vol. 39, No. 1, January 2003, pp. 176-191.

[35] Irani, M., and Anandan, P., "A Unified Approach to Moving Object Detection in 2D and
3D Scenes," IEEE Transactions on Pattern Analysis and 1MAchine Intelligence, Vol. 20,
No. 6, June 1998.

[36] Sharma, R., and Aloimonos, Y., "Early Detection of Independent Motion from Active
Control of Normal Flow Patterns," IEEE Transactions on Systems, 1Man, and Cybernetics,
Vol. 26, No. 1, February 1996.

[37] Frazier, J., and Nevatia, R., "Detecting Moving Objects from a Moving Platform," IEEE
International C~rrTk ratec, on Robotics and Automation, Nice, France, May 1992, pp.









Open-loop estimation of target's inertial attitude .

Norm error for target state estimates ......

Closed-loop target position tracking ......

Position tracking error ......

Target attitude tracking ......

Tracking error in heading angle .......

Target's inertial position with uncertainty bounds

Target's inertial attitude with uncertainty bounds .

.. .. ... . 145

........ .. .146

........ .. .146

.......... .. 147

.......... .. 147

........ .. .147

.... .. .. .149

. .... ... .. .149

epipolar constraint enables the relationship between image frames to compute the features of the

target through Equation 7-15

PvT,n (xyTVRIV) pr,n= (7-15)

where xyjv is the skew symmetric representation of the relative translation from I to V expressed

in I and the new pixel coordinates determined from the virtual camera are denoted as pVF,n =

[PVF,n, VVF,n] Vn2 for the reference vehicle and pyr,n = [PVr,n, Wa~n] Vn2 for the target vehicle. As

a result of the virtual camera, the desired property is obtained regarding pixels of the reference

vehicle computed from the camera at i 1 are equal to the pixels generated by the virtual camera

at i. Mathematically, this property is expressed in Equation 7-16 which relies on the relative

motion remaining constant to maintain the reference stationary in the image.

PF,n ( 1) = PVF,n (i) (7-16)

With this virtual camera in place and the reference pixels stationary, the computation of

the homography between the reference and target vehicles is considered. First, the geometric

relationships are established relative to the virtual camera of both the reference and target

vehicles by denoting their feature point positions in Euclidean coordinates. The time varying

position of a feature point on the reference vehicle expressed in V is given in Equation 7-17.

Likewise, the time varying position of a feature point on the target vehicle expressed in V is given

in Equation 7-18.

rlVF,n = mvF +tRFVSF (7-17)

rlvr,n = myr +tRTVSy (7-18)

The components of these Euclidean coordinates are defined in Equations 7-19 and 7-20 and are

relative to the virtual camera frame.

rlVF,n (t) t yt t)(-9

vrn()a x t r()L t (7-20)


10.1 Example 1: Feature Point Generation

A simulation of vision-based feedback is presented to demonstrate the implementation,

and resulting information, associated with sensor models and aircraft dynamics. This simulation

utilizes a nonlinear model of the flight dynamics of an F-16 [110]. A baseline controller is

implemented that allows the vehicle to follow waypoints based entirely on feedback from inertial


Images are obtained from a set of cameras mounted on the aircraft. These cameras include

a stationary camera mounted at the nose and pointing along the nose, a translating camera under

the centerline that moves from the right wing to the left wing, and a pitching camera mounted

under the center of gravity. The parameters for these cameras are given in Table 10-1 in values

relative to the aircraft frame and functions of time given as t in seconds.

Table 10-1. States of the cameras
position (ft) orientation (deg)
camera xe ye ze Wcc V
1 24 0 0 0 90 0
2 -10 15-3t 0 0 45 0
3 0 0 3 0 45-9t 0

The camera parameters are chosen as similar to an existing camera that has been flight

tested [111]. The focal length is normalized so f = 1. Also, the field of view for this model

correlates to angles of y;; = 32 deg and y, = 28 deg. The resulting limits on image coordinates are

given in Table 10-2.

Table 10-2. Limits on image coordinates
coordinate minimum maximum
pu -0.62 0.62
v -0.53 0.53

A virtual environment is established with some characteristics similar to an urban

environment. This environment includes several buildings along with a moving car and a

To my lovely wife, Liza P. Causey, that has supported me every step of the way. Her love and

understanding through the years have brought my passion for life beyond boundaries.

A commonly used algorithm that employed these equations with slight variations is the

Harris comer detector [106]. This method can be extended to edge detection by considering

the structure of the singular values of G. An example of this algorithm is the Canny edge

detector [107].

3.4 Feature Point Tracking

Feature tracking or feature correspondence is the next step in the general state estimation

problem. The correspondence problem is described as the association of a feature point between

two or more images. In other words, the solution to this problem determines that a feature point

in two or more images corresponds to the same physical point in 3D space. The most common

approach to discerning how points are moving between images is the use of intensity or color

matching. This brightness matching is typically performed over a small window, W(x), centered

around the point of interest, as opposed to only matching a single brightness value, which could

have numerous false solutions. The vector of brightness values over a small window set, 1,

contained in the image is shown in Equation 3-21.

1(x) = {I(i)| IE W (x)} (3-21)

This brightness vector can be compared across images, li and I2, and optimized to find

the minimum error. If a feature point of interest is located at xl = [pt, v1] in image 1, li then a

simple translational model of the same scene can be used as an image matching constraint. This

relationship is shown in Equation 3-22,

11 (xl) = I2(h(x1)) +t n(h(x1)) (3-22)

where h(x) defines a general motion transformation to the proceeding image and n(xl) is additive

noise caused by ambiguities such as variations in lighting, reflections, and view point.

Therefore, the correspondence solution is cast as a minimization problem that computes the

best intensity match over a small window by minimizing the intensity error. An equation for the

translation estimate can then be found from this minimization process through Equation 3-23,

noise and unknown camera parameters so, in practice, an averaging process is often used to

estimate the feature coordinates.

There are two fundamental issues regarding the obtained solution. First, by relying on

the solution provided by the eight-point algorithm, then the translation is only determined up

to a scaling factor. The SFM solution will therefore be corrupted from this scale factor unless

an alternative method is used to obtain this scaling. Second, the uncertainty due to intrinsic

parameters, feature detection, feature tracking, along with the uncertainty in the solution of the

eight-point algorithm contributes to large variations in the SFM solution. The solution obtained

from Equation 3-64 is very sensitive to these uncertainties. Chapter 4 will discuss a method to

obtain uncertainty bounds on the SFM estimates based on the sources described.

[51] Silveira, G. F., Carvalho, J. R. H., Madirid, M. K., Rives, P., and Bueno, S. S., "A Fast
Vision-Based Road Following Strategy Applied to the Control of Aerial Robots," IEEE
Proceedings of XIV Brazilian Symposium on Computer Graphics and Image Processing,
1530-1834/01, Florianopolis, Brazil, October 2001, pp. 226-231.

[52] Soatto, S., and Perona, P., "Dynamic Visual Motion Estimation from Subspace
Constraints," IEEE, 0-8186-6950-0/94, 1994, pp. 333-337.

[53] Soatto, S., Frezza, R., and Perona P., "Motion Estimation via Dynamic Vision," IEEE
Transactions on Automatic Control, Vol. 41, No. 3, March 1996, pp. 393-413.

[54] Soatto, S., and Perona, P., "Recursive 3-D Motion Estimation Using Subspace
Constraints," international Journal of Computer Vision, Vol. 22, No. 3, 1997, pp. 235-259.

[55] Soatto, S., and Perona, P., "Reducing 'Structure from Motion'," IEEE, 1063-6919/96,
1996, pp. 825-832.

[56] Soatto, S., and Perona, P., "Visual Motion Estimation from Point Features: Unified View,"
IEEE international C~rr rtf rrtic on Image Processing, Vol. 3, October 1995, pp. 21-24.

[57] Soatto, S., and Perona, P., "Reducing 'Structure from Motion': A General Framework for
Dynamic Vision Part 1: Modeling," IEEE Transactions on Pattern Alrab\ \i\ and 1Machine
Intelligence, Vol. 20, No. 9, September 1998, pp.933-942.

[58] Soatto, S., and Perona, P., "Reducing 'Structure from Motion': A General Framework
for Dynamic Vision Part 2: Implementation and Experimental Assessment," IEEE
Transactions on Pattern Ata rl~li\ and 1Machine intelligence, Vol. 20, No. 9, September
1998, pp. 943-960.

[59] Erol, A., Bebis, G., Nicolescu, M., Boyle, R. D., and Twombly, X., "A Review on
Vision-Based Full DOF Hand Motion Estimation," IEEE Computer Society International
~r~rTforeticc on Computer Vision and Pattern Recognition, San Diego, CA, June 2005.

[60] Huang, T. S., and Netravali, A. N., "Motion and Structure from Feature Correspondences:
A Review," Proceedings of the IEEE, Vol. 82, No. 2, February 1994, pp. 252-268.

[61] Stewart, C. V., "Robust Parameter Estimation in Computer Vision," Society of Industrial
and Aplied Mathelr manil, \, Vol. 41, No. 3, 1999, pp. 513-537.

[62] Weng, J., Huang, T. S., and Ahuja, N., "Motion and Structure from Two Perspective
Views: Algorithms, Error Analysis, and Error Estimation," IEEE Transactions on Pattern
Analysis and 1Machine intelligence, Vol.11, No. 5, May 1989, pp. 451-476.

[63] Jianchao, Y., "A New Method for Passive Location Estimation from Image Sequence
Using Adaptive Extended Kalman Filter," International Confrrticcr, on Signal Processing,
Beijing, China, October 1998, pp. 1002-1005.

[64] Qian, G., Kale, G., and Chellappa, R., "Robust Estimation of Motion and Structure using a
Discrete H-infinity Filter," IEEE 0- 7803-6297-7/00, 2000, pp. 616-619.

Finally, the Markov transition probability function is given explicitly in Equation 8-16

as three-dimensional Gaussian density function and is uniquely determined by the mean and

1 1 I(ai (t) p (ai (t))) 2
P (ai (t))= ep(-6
AGn (ai (t))1 e 2 G2 a t (-6
The probability function is then extended over the entire time interval [t, t +t k] to estimate

the prediction probability. Mathematically, this extension is expressed in Equation 8-17.

P (ai (t +t j)) = P (ai (t +t j 1)) = .. = P (ai (t)) (8-17)

Employing Equations 8-16 and 8-17, the predictive probability for object i at time t +t k

is given as Equation 8-18. This framework enables the flexibility of computing the predicted

estimates at any desired time in the future with the notion that further out in time the probability

Probe (ai (t + k)) = n P (ai (t. +j) (8-18)
j= 0
A similar process is considered for computing the Markov transition functions for both

velocity and position. First, the mean and variance vectors for velocity and position are defined in

Equations 8-19 and 8-20 for the entire time interval.

[p (vi (t + j)) G2 (Vi (t + j))] j ,1 ,k(8-19)

[p(pi (t+- j)) ,G2 (pi(t + j))] j=01.k(8-20)

The initial mean and variance expressions for the velocity are given in Equation 8-22

and 8-23.

pu (vi (t)) = pu (vi (t 1) +t ai (t 1)) (8-21)

= vi (t 1) +t p (ai (t 1))

G2 (Vi (t)) = G2 (Vi (t 1) +t ai (t 1)) (8-22)

The simulation was then played in a virtual environment to enhance the graphics and

illustrate the application of this algorithm. To add the vehicles within the virtual environment the

velocities of each vehicle had to be scaled down to practical values that fit the scene. Snapshots

are shown in Figure 10-22 of the camera view depicting the vehicles and the surrounding scene.

The red vehicle was designated as the reference whereas the grey vehicle was the target vehicle.

The next step in this process is to implement an actual feature tracking algorithm on the synthetic

images that follows the vehicles. This modification alone will degrade the homography results

immensely due to the troublesome characteristics of a feature point tracker.

Figure 10-22. Virtual environment

10.4 Example 4: Closed-loop Aerial Refueling of a UAV

A closed-loop simulation was executed in Matlab to replicate an autonomous aerial

refueling task. As Chapter 1 described the motivation and the benefits of AAR, this section will

demonstrate it by combining the control design given in Chapter 9 with the homography result

in Chapter 7 to form a closed-loop visual servo control system. The vehicles involved in this

simulation includes a Receiver UAV instrumented with a single camera, a tanker aircraft also

are slight differences in direction due to the translating camera. Likewise, the optic flow observed

by camera 3 is different due to the camera's orientation.

-0 -4 02 0 02 04 06 -06 -04 -02 0 02 04 06 -06 -04 -02 0 0 4 0

Figure 10-3. Optic flow Measurements at t = 2 sec for A) camera 1, B) camera 2, and C) camera

A summary of the resulting image plane quantities, position and velocity, is given in

Table 10-5 for the feature points of interest as listed in Table 10-3. The table is organized by the

time at which the image was taken, which camera took the image, and which feature point is

observed. This type of data enables autonomous vehicles to gain awareness of their surroundings

for more advanced applications involving guidance, navigation and control.

Table 10-5. Image coordinates of feature points
Time (s) Camera Feature Point pu v p 9
2 1 1 0.157 0.162 0.610 0.044
2 1 3 0.051 0.267 0.563 -0.012
2 2 2 -0.308 0.075 0.464 -0.254
2 3 2 0.011 0.077 0.583 -0.235
4 2 2 -0.279 -0.243 -0.823 0.479
4 3 2 0.365 -0.248 -0.701 0.603
6 1 3 0.265 -0.084 0.267 -0.015

10.2 Example 2: Feature Point Uncertainty

10.2.1 Scenario

Feature point uncertainty is demonstrated in this section by extending the previous example.

This simulation will examine the uncertainty effects on vision processing algorithms using

simulated feature points and perturbed camera intrinsic parameters.

begin to falter due to the drastic changes in lighting conditions and the large view point change.

Therefore, a more general motion transformation is used such as an affine transformation.

Normalized cross-correlation techniques are also used for large baseline configurations to handle

considerable changes in lighting conditions.

An extremely important concern is the accuracy of these algorithms and how variations in

feature point tracking can effects the final state estimation. These concerns will be addressed in

detail in Chapter 4.

3.5 Optic Flow

The next metric of interest in the image plane is an expression for the velocity of a feature

point. This expression is found simply by taking the time derivative of the feature point position

defined in Equations 3-7 and 3-8. The velocity expressions, shown in Equations 3-32 and 3-33,

describe the movement of feature points in the image plane and is commonly referred to in

literature as the optic flow.

Likewise, the feature point velocity with radial distortion can be computed by differentiating

Equations 3-15 and 3-16 while assuming c = 0 is as follows

ild = ii(1+- dr2) +t 2pdri (3-34)

itd = 9j(1+- dr2) +t 2vdrt (3-35)

pu v
r = p- + 9 (3-36)

3.6 Two-View Image Geometry

The two-view image geometry relates the measured image coordinates to the 3D scene.

The camera configuration could be either two images taken over time of the same scene, as


7.1 Introduction

Autonomous vehicles have gained significant roles and assisted the military on the

battlefield over the last decade by performing missions such as reconnaissance, surveillance,

and target tracking with the aid of humans. These vehicles are now being considered for more

complex missions that involve increased autonomy and decision making to operate in cluttered

environments with less human interaction. One critical component that autonomous vehicles

need for a successful mission is the ability to estimate the location and movement of other objects

or vehicles within the scene. This capability, from a controls standpoint, enables autonomous

vehicles to navigate in complex surroundings for tracking or avoidance purposes.

Target state estimation is an attractive capability for many autonomous systems over a

broad range of applications and is the focus of this dissertation. In particular, unmanned aerial

vehicles (UAV) have shown a great need for this technology. With UAV becoming more prevalent

in the aerospace community, researchers are striving to extend their capabilities while making

them more reliable. The key applications of interest for future UAV regarding target estimation

pertain to both civilian and military tasks. These tasks range from police car pursuits and border

patrol to locating and pursuing enemy vehicles during lethal engagement. A major limitation

to small UAV are their range, payload constraints and fuel capacity. These limitations generate

the need for autonomous aerial refueling (AAR) to extend the vehicle's operational area. Target

state estimation facilitates a portion of the AAR problem by estimating the receptacle's current

position and orientation during approach. Therefore, the purpose of this paper is to demonstrate

a method that estimates the motion of a target using an on-board monocular camera system to

address these applications.

Most techniques for vision-based feedback share some commonality; namely, a sequence of

image processing and vision processing are performed on an image or a set of images to extract

information which is then analyzed to make a decision. The basic unit of information from an

stabilization design also included a roll to elevator connect to help counteract the altitude loss

during a tumn.

The outer-loop is completed by simply closing the loop around the roll tracker using a

proportional gain to follow to desired heading. In addition, command limits of +600 were

placed on roll to regulate aggressive turns and a yaw damper was also implemented that included

a aileron-rudder interconnect which helps a tumn in a number of ways. The aileron-rudder

interconnect helps to raise the nose up during a turn. Meanwhile, the yaw damper is employed to

damp oscillations from the Dutch-roll mode during a heading maneuver. The design of the yaw

damper is provided in Stevens et al. [110]. Consequently, the tumn smoother and contains less


Tracking heading is not sufficient to track the lateral position with the level of accuracy

needed for refueling task. The final loop was added to account for any lateral deviation

accumulated over time due to the delay in heading from position. This delay is mainly due to the

time delay associated with sending a roll command and producing a heading change. Therefore,

this loop was added to generate more roll for compensation. The loop commanded a change in

aileron based of the error in lateral position. This deviation, referred to as Ay, was computed

based on two successive target locations provided by the estimator. The current and previous

(x, y) positions of the target were used to compute a line in space to provide a reference of the it's

motion. The perpendicular distance from the vehicle's position to this line was considered the

magnitude of the lateral command. In addition, the sign of the command was needed to assign

the correct direction. This direction was determined from the relative y position, expressed in the

body-fixed frame, that was found during estimation. Once the lateral deviation was determined,

that signal was passed through a PI structure, as shown in Figure 9-2. The gains corresponding to

the proportional, kyp, and integrator, kyi, were then summed and added to compute the final roll

command. The complete expression for the roll command is shown in Equation 9-1.

#cmd = kW (Vcmd W) +t ky p~y + ~Yi (9-1)

(R, T)

Figure 3-4. Geometry of the epipolar constraint

The expressions in Equation 3-37 and Equation 3-38 reflect that the scalar triple product

of three coplanar vectors is zero, which forms a plane in space. These relationships can be

expanded using linear algebra [102, 103] to generate a standard form of the epipolar geometry

as in Equation 3-39. This new form indicates a relationship between the rotation and translation,

written as the essential matrix denoted as Q, to the intrinsic parameters of the camera and

associated feature points. In this case, the equation is derived for a single feature point that is

correlated between the frames,

[#2 V2 fe 91 v1 fT = 0 (3-39)

where Q = [T] xR and [T] x is defined as the skew-symmetric form of the translation T.

The geometric relationship formed by this triangular plane is also seen in the epipolar lines

of each image. The 3D plane formed through this triangle constrains a feature point in one image

to lie on the epipolar line in the other image. These constraints can be mathematically expressed

function given in Equation 3-27.

El(u) = [VI (i, t)u(x) +tIt (1, t)]2 (3-27)

The minimum of this function is obtained by setting VE1 = 0 to obtain Equation 3-28,

LI,2 LCly + I<^ = 3-

I C,tv LIS ut LIvT, O(-8

or, rewritten in matrix form results in the following

Gu -tb= (3-29)

where G(x) was derived in Equation 3-19.

The final solution for the pixel velocity is found through a least-squares estimate given in

Equation 3-30. These image velocities are also referred to as the optic flow. Once the optic flow

is computed for a feature point then the image displacement for feature tracking is trivial to find.

u = G b (3-30)

On the other hand, the method using SSD, shown in Equation 3-24, attempts to estimate

the Ax while not requiring the computation of image gradients. This approach also employs

the translational model over a windowed region. The method considers the possible range that

window could move, dpu and dv, in the time, dt. This consistency constraint then leads to a

problem of minimizing the error over the possible windows within the described range. This error

function is described mathematically in Equation 3-31.

E2 (dpdv) = [I(p + dp, lv + dv, t +dt) I0pv, t)]2 (3-31)
W( c,v)

The solutions obtained are the displacement components, dpu and dv, of the specified

window that correlates to the translation of the center pixel. This techniques is the foundation

for the Lucas-Kanade tracker [17]. For large baseline tracking simple translational models

section examines the motion detection problem through the residual optical flow to further

classify static objects from dynamic objects in the field of view.

FOE2 -'-


10.5 0 -0.5 -1

Figure 6-3. Residual optic flow for dynamic environments

6.2 Classification

The classification scheme proposed in this dissertation is an iterative approach to computing

the FOE of the static environment using the residual optical flow given in Equation 6-8. An

approximation for the potential location of the FOE is found by extending the translational

optical-flow vectors to form the epipolar lines, as illustrated in Figure 6-3, and obtaining all

possible points of intersection. As mentioned previously, the intersection points obtained will

constitute a number of potential FOEs; however, only one will describe the static background

while the rest are due to moving objects. The approach considered for this classification that

essentially groups the intersection data together through a distance criterion is an iterative

least-squares solution for the potential FOEs.

The iteration procedure tests all intersection points as additional features are introduced

to the system of equations each of which involves 2 unknown image plane coordinates of the

FOE (Pfoe;, VSoe;). The process starts by considering 2 feature points and their FOE intersection

/IT~n XV
e eFeature
Fly~ syPoin E lve~ sPoint

sF FauePoint sF) Fetur n



Figure 7-2. Moving target vector description relative to A) camera I and B) virtual camera V

distortion, discrete mapping into pixels, and field of view constraints which are further also

specified in Chapter 3. Each extension to the model adds another parameter to know for the

estimation problem and each can introduce uncertainty and large errors in the estimation result.

Therefore, this chapter will only consider the field of view constraint and leave the nonlinear

terms and the effects on estimation for future work. Recall the field of view constraints given in

Chapter 3. These constraints can be represented as lower and upper bounds in the image plane

and are dependent on the half angles (yh, t) which are unique to each camera. Mathematically,

these bounds are shown in Equations 7-1 and 7-2 for the horizontal and vertical directions.

[pip] = [-f tanh, f tanyh (7-1)

[v, V] = [- f tany,, f tany,] (7-2)

7.2.2 Homography Estimation

The implicit relationship between camera and environment is known as the epipolar

constraint or, alternatively, the homography constraint. This constraint notes position vectors that

describe a feature point, rln, at two instances in time are coplanar with the camera's translation

initially travel north until the target vehicles makes a left turn and heads west and is subsequently

followed by the pursuit vehicles.

2 100'""


00 20 40 60 <* 0 -00 20 40 60
Time (sec) Time (sec) Time (sec)

Figure 10-13. Position states of the UAV with on-board camera: A) North, B) East, and C) Down

60 10 5

50 0

0 20 40 60 -10 20 40 60 -0020 40 60
Time (sec) Time (sec) Time (sec)

Figure 10-14. Attitude states of the UAV with on-board camera: A) Roll, B) Pitch, and C) Yaw

1~ 5 800

0 5 -15000

00 20 40 60 <* 0 00 20 40 60
Time (sec) Time (sec) Time (sec)

Figure 10-15. Position states of the reference vehicle (pursuit vehicle): A) North, B) East, and C)

10.3.2 Open-loop Results

The homography was computed for this simulation to find the relative rotation and

translation between the ground vehicles. These results are then used to find the relative


and Kanade [115] proposed an algorithm that reconstructs 3D motion of a moving object using

a factorization-based algorithm with the assumption that the object moves linearly with constant

speeds. A nonlinear filtering method was used to solve the process model which involved both

the kinematics and the image sequences of the target [118] This technique requires knowledge

of the height above the target which was done by assuming the target traveled on the ground

plane. This assumption allowed other sensors, such as GPS, to provide this information. The

previous work of Mehta et al. [77] showed that a moving monocular camera system could

estimate the Euclidean homographies for a moving target in reference to a known stationary


The contribution of this chapter is to cast the formulation shown in Mehta el al. to a

more general problem where both target and reference vehicles have general motion and are

not restricted to planar translations. This proposed approach incorporates a known reference

motion into the homography estimation through a transformation. Estimates of the relative

motion between the target and reference vehicle are computed and related back through known

transformations to the UAV. Relating this information with known measurements from GPS

and IMU, the reconstruction of the target's motion can be achieved regardless of its dynamics;

however, the target must remain in the image at all times. Although the formulation can be

generalized for n cameras with independent position, orientation, translations, and rotation this

chapter describes the derivation of a single camera setup. Meanwhile, cues on both the target

and reference objects are achieved through LED lights or markers placed in a known geometric

pattern of the same size. These markers facilitate the feature detection and tracking process by

placing known features that stand out from the surroundings while the geometry and size of the

pattern allows for the computation of the unknown scale factor that is customary to epiploar and

homography based approaches.

This chapter builds on the theory developed in Chapters 3 and 5 while relying on the moving

object detection algorithm to isolate moving objects within an image. Recall the flow of the

overall block diagram shown in Figure 1-6. The process started by computing features in the

provide poor estimation if the motions of moving objects are not accounted for in the registration

process or if the image contains stationary objects close to the camera that result in high parallax.

A technique presented by Irani et al. [35] proposed a unified method to detect moving

objects. This proposed method handles various levels of parallax in the image through a

segmentation process that is performed in layers. The first layer extracts the background objects

which are far away from the camera and have low parallax through a general transformation

involving camera rotation, translation, and zoom through image differencing. The next layer

contains the object with high parallax consisting of both objects close to the camera and objects

that are moving independently of the camera. The parallax is then computed for the remaining

pixels and compared to one pixel. This process separates the objects within the image based

on their computed parallax. The selection may involve choosing a point on a known stationary

object that contains high parallax so any object not obeying this parallax is classified as a moving

object in the scene.

Optic flow techniques are also used to estimate moving target locations once ego-motion

has been estimated. A method that computes the normal image flow has been shown to obtain

motion detection [36]. Coordinate transformations are sometimes used to facilitate this approach

to detecting motion. For instance, a method using complex log mapping was shown to transform

the radial motions into horizontal lines upon which vertical motion indicate independent

motion [37]. Alternatively, spherical mapping was used geometrically to classify moving objects

by segmenting motions which do not radiate from the focus of expansion (FOE) [29].

2.2 State Estimation Using Vision Information

The types of state estimation that can be obtained from an on-board vision system

are (i) localization which estimates the camera motion between image frames from known

stationary feature points, (ii) mapping which estimates the location of 3D feature points using

reconstruction and structure from motion, and (iii) targpet-motion which estimates 3D feature

points that have independent motion. The work related to these topics are described in this


[91] Papaikolopoulos, N. P., Nelson, B. J., and Khosla, P. K., "Six Degree-of-Freedom
Hand/Eye Visual Tracking with Uncertain Parameters," IEEE Transactions on Robotics
and Automation, Vol. 11, No. 5, October 1995, pp. 725-732.

[92] Sznaier, M., and Camps, O. I, "Control issues in Active Vision: Open Problems and Some
Answers," IEEE C~rr rikrrtec on Decision and Control, Tampa, FL, December 1998, pp.

[93] Frezza, R., Picci, G., and Soatto, S., "Non-holonomic Model-based Predictive Output
Tracking of an Unknown Three-dimensional Trajectory," IEEE ~r rTkratec, on Decision
and Control, Tampa, FL, December 1998, pp. 3731-3735.

[94] Papaikolopoulos, N. P., Khosla, P. K., and Kanade, T., "Visual Tracking of a Moving
Target by a Camera Mounted on a Robot: A Combination of Control and Vision," IEEE
Transactions on Robotics and Automation Vol. 9, No. 1, February 1993, pp. 14-35.

[95] Papaikolopoulos, N. P., and Khosla, P. K., "Adaptive Robotic Visual Tracking: Theory and
Experiments," IEEE Transactions on Automatic Control, Vol.38, No. 3, March 1993, pp.

[96] Zanne, P., Morel, G., and Plestan, F., Robust Vision Based 3D Trajectory Tracking using
Sliding Mode Control," IEEE International C~rr rikrate on Robotics and Automation, San
Francisco, CA, April 2000, pp. 2088-2093.

[97] Zergeroglu, E., Dawson, D. M., de Queiroz, M. S., and Behal, A., "Vision-Based
Nonlinear Tracking Controllers with Uncertain Robot-Camera Parameters," IEEE/ASIME
International C~rrik reticc on Advanced 1Mechanics, Atlanta, GA, September 1999, pp.

[98] Valasek, J., Kimmett, J., Hughes, D., Gunnam, K., and Junkins, J. L., "Vision Based
Sensor and Navigation System for Autonomous Aerial Refueling," A1AA 's 1st Technical
C~r~rikicticc and Workshop on Unmanned Aerospace Vehicles, Portsmouth, Virginia, May

[99] Pollini, L., Campa, G., Giulietti, F., and Innocenti, M., "Virtual Simulation Set-Up for
UAVs Aerial Refueling," AIAA 1Modeling and Simulation Technologies C~rrakican c. Austin,
TX, August 2003.

[100] No, T. S., and Cochan, J. E., "Dynamics and COntrol of a Teathered Flight Vehicle,"
Journal of Guidance, Control, and Dynamics, Vol. 18,No. 1, January 1995, pp. 66-72.

[101] Forsyth, D. A., and Ponce, J., "Computer Vision : A Modern Approach, Prentice-Hall
Publishers, Upper Saddle River, NJ, 2003.

[102] Ma, Y., Soatto, S., Kosecka, and Sastry, S. S., "An Invitation to 3-D Vision: From Images
to Geometric Models," Springer-Verlag Publishing New York, NY, 2004.

[103] Faugeras, O., "Three-Dimensional Computer Vision," The MIT Press, Cambridge
Massachusetts, 2001.

known [94]. Adaptive solutions presented in [91, 95-97] have shown control solutions for

target tracking with uncertain camera parameters while estimating depth information.

The homing control problem has numerous applications toward autonomous systems such

as autonomous aerial refueling, spacecraft docking, missile guidance, and object retrieval

using a robtotic manipulator. Kimmett et al. [15, 98] developed a candidate autonomous

probe-and-drogue aerial refueling controller that uses a command generator tracker (CGT) to

track time-varying motions of a non-stationary drogue. The CGT is an explicit model following

control technique and was demonstrated in simulation for a moving drogue with known dynamics

subject to light turbulence. Tandale et al. [16] extended the work of Kimmett and Valasek by

developing a reference observer based tracking controller (ROTC) which does not require a

drogue model or presumed knowledge of the drogue position. This system consist of a reference

trajectory generation module that sends commands to an observer that estimates the desired

states and control for the plant. The input to this controller is the relative position between the

receiver aircraft and the drogue measured by the vision system. A similar vision approach to

aerial refueling is also presented in [99], where models of the tanker and drogue are used in

conjunction with an inferred camera. The drogue model used in this paper was taken from [100]

that uses a multi-segment approach to deriving the dynamics of the hose. Meanwhile, Houshangi

et al. [80] considered grasping a moving target by adaptively controlling a robot manipulator

using vision interaction. The adaptive control scheme was used to account for modeling errors in

the manipulator. In addition, this paper considered unknown target dynamics. An auto-regressive

model approach was used to predict the target's position based on passed visual information

and an estimated target velocity. Experimental test cases are documented that show tracking


9.2 Controller Development

The control architecture chosen for this mission consisted of a Proportional, Integral and

Derivative (PID) framework for waypoint tracking given in Stevens and Lewis [110]. The

standard design approach was used by considering the longitudinal and lateral states separately as

in typical waypoint control schemes. This approach separated the control into three segments: 1)

Altitude control, 2) Heading Control and 3) Depth Control.

9.2.1 Altitude Control

The first stage considered in the control design to home on a target is the altitude tracking.

This stage considers the longitudinal states of the aircraft using the elevator as the control

effector. The homography generates the altitude command necessary to track and dock with the

refueling receptacle. The architecture for the altitude tracking system is shown in Figure 9-1. The

first portion of this system is described as the inner-loop where pitch and pitch rate are used in

feedback to stabilize and track a pitch command. Meanwhile, the second portion is referred to as

the outer-loop which generates pitch commands for the inner-loop based on the current altitude

error. The inner-loop design enables the tracking of a pitch command through proportional

Figure 9-1. Altitude hold block diagram

control. This pitch command in turn will affect altitude through the changes in forces on the

horizontal tail from the elevator position. The two signals used for this inner-loop are pitch and

pitch rate. The pitch rate feedback helps with short period damping and allows for rate variations

in the transient response. A lead compensator was designed in Stevens et al. [110] to raise the

loop gain and to achieve good gain and phase margins for the pitch command to pitch transfer


The outer-loop design involved closing the loop in altitude. The altitude error signal is

generated by the difference in current altitude and the commanded altitude computed by the

estimation algorithm. The compensator designed for the inner-loop pitch is augmented to

maintain the high loop gain and is defined as G, in Figure 9-1. This structure will provide good

disturbance rejection during turbulent conditions. In addition, bounds were placed on the pitch

command to alleviate any aggressive maneuvers during the refueling process.

9.2.2 Heading Control

The next stage in the control design consist of the turn or heading coordination. This

aspect involves the lateral directional states of the aircraft. The control surfaces that effect these

states are ailerons and rudder. Similar to the altitude controller, the homography estimates a

heading command that steers the aircraft in the desired direction toward the target of interest. The

control architecture that accomplishes this objective is depicted in Figure 9-2. The inner-loop

Figure 9-2. Heading hold block diagram

component of Figure 9-2 deals with roll tracking. The feedback signals include both roll and

roll rate through proportional control to command a change in aileron position. The inner-loop















.. 64

.... . .23

..... . 13

.... . 14

.... . .25

.. .126

..... . 19

.... . 31

... .. . 33

. .. . 10

. .. .. .. . 10

Solutions for homography decomposition ......

States of the cameras ......

Limits on image coordinates ......

States of the feature points ......

Aircraft states ......

Image coordinates of feature points .......

Effects of camera perturbations on optic flow ......

Effects of camera perturbations on epipolar geometry .....

Effects of camera perturbations on structure from motion ...

Maximum variations in position due to parametric uncertainty

Maximum variations in attitude due to parametric uncertainty .

through Equation 5-37

Xi(t) = (X (t), U(t), a(t), t) (5-35)

X (0) = Ko (5-36)

Y (t) = g (X (t), a(t), rl, t) (5-37)

where U(t) is defined as a set of control inputs to the aircraft and u(t) is a vector containing

the camera parameters aT = {ul, a2, ak T for k number of cameras. These equations that

utilized feature position will be referred to as the Control Theoretic Form of the governing

camera-aircraft equations.

Alternatively, if the image plane velocities are employed instead of the image plane

positions, as seen in Equation 5-37, then a different set of equations can be obtained which will

be referred to as the Optic Flow Form of the governing aircraft-camera equations of motion. This

system is given in Equation 5-40, which uses the optic flow expression given in Equations 3-34
and 3-35 as the observations.

Xi(t) = (X (t), U(t), a(t), t) (5-38)

X(0) = Ko (5-39)

J"(t) = 3M(X (t), a(t), rl, t) (5-40)

The two system equations just described both have applications to missions involving

unmanned aerial vehicles. The Control Theoretic Form primarily applies to missions involving

target tracking and surveillance such as aerial refueling and automated visual landing.

Meanwhile, the Optic Flow Form is useful for guidance and navigation through unknown

environments. The information provided by optic flow reveals magnitude and direction of each

feature point in the image which gives a sense of objects in close proximity. Incorporating

this information, along with some logic, a control system can be designed to avoid unforeseen

obstacles throughout the desired path.

x01 t 6x10


010 20 30 40 oO 10 20 30 40
Time (sec) Time (sec)

Figure 10-31. Norm error for target state estimates A) translation and B) rotation

---Target 2300--are

8/ 1 -10i

0 10 20 30 40 -50 10 20 30 40 250 10 20 30 40
Time (sec) Time (sec) Time (sec)

Figure 10-32. Closed-loop target position tracking: A) North, B) East, and C) Altitude

The components of the position error between the receiver and drogue are shown in

Figure 10-33 to illustrate the performance of the tracking controller. These plots depict the initial

offset error decaying over time which indicates the receiver's relatives distance is decreasing. The

altitude showed a quick climb response where as the response in axial position was a slow steady

approach which was desired to limit large changes in altitude and angle of attack. The lateral

position is stable for the time period but contains oscillations due the roll to heading lag.

The orientation angles shown in Figure 10-34 indicate the Euler angles for for the

body-fixed transformations corresponding to the body-fixed frame of the receiver and the

body-fixed frame of the drogue. Recall, the only signal being tracked in the control design was

heading. This selection allowed the aircraft to steer and maintain a flight trajectory similar to the

drogue without aligning roll and pitch. The receiver should fly close to a trim condition rather

then matching the full orientation of the drogue, as illustrated in Figure 10-34 for pitch angle.

Full Text








ThisworkwassupportedjointlybyNASAunderNND04GR13HwithSteveJacobsonandJoePahleasprojectmanagersalongwiththeAirForceResearchLaboratoryandtheAirForceOfceofScienticResearchunderF49620-03-1-0381withJohnnyEvers,NealGlassman,SharonHeise,andRobertSierakowskiasprojectmonitors.Additionally,IthankDr.RickLindforhisremarkableguidanceandinspirationthatwilltrulylastalifetime.Finally,IthankmyparentsSandraandJamesCauseyformakingthisjourneypossiblebyprovidingmetheguidanceanddisciplineneededtobesuccessful. 4


page ACKNOWLEDGMENTS .................................... 4 LISTOFTABLES ....................................... 8 LISTOFFIGURES ....................................... 9 LISTOFTERMS ........................................ 12 ABSTRACT ........................................... 19 CHAPTER 1INTRODUCTION .................................... 21 1.1Motivation ...................................... 21 1.2ProblemStatement ................................. 27 1.3PotentialMissions .................................. 27 1.4SystemArchitecture ................................. 30 1.5Contributions .................................... 33 2LITERATUREREVIEW ................................. 36 2.1DetectionofMovingObjects ............................ 36 2.2StateEstimationUsingVisionInformation ..................... 38 2.2.1Localization ................................. 39 2.2.2Mapping ................................... 39 2.2.3Target-MotionEstimation .......................... 40 2.3ModelingObjectMotion .............................. 41 2.4UncertaintyinVisionAlgorithms .......................... 42 2.5ControlUsingVisualFeedbackinDynamicEnvironments ............ 43 3IMAGEPROCESSINGANDCOMPUTERVISION .................. 45 3.1CameraGeometry .................................. 45 3.2CameraModel .................................... 47 3.2.1IdealPerspective .............................. 47 3.2.2IntrinsicParameters ............................. 48 3.2.3ExtrinsicParameters ............................. 49 3.2.4RadialDistortion .............................. 50 3.3FeaturePointDetection ............................... 51 3.4FeaturePointTracking ............................... 53 3.5OpticFlow ..................................... 56 5


............................. 56 3.6.1EpipolarConstraint ............................. 57 3.6.2Eight-PointAlgorithm ............................ 59 3.6.3PlanarHomography ............................. 61 3.6.4StructurefromMotion ............................ 65 4EFFECTSONSTATEESTIMATIONFROMVISIONUNCERTAINTY ........ 67 4.1FeaturePoints .................................... 67 4.2OpticalFlow ..................................... 70 4.3EpipolarGeometry ................................. 71 4.4Homography .................................... 73 4.5StructureFromMotion ............................... 75 5SYSTEMDYNAMICS .................................. 77 5.1DyanmicStates ................................... 77 5.1.1Aircraft ................................... 77 5.1.2Camera ................................... 79 5.2SystemGeometry .................................. 81 5.3NonlinearAircraftEquations ............................ 83 5.4Aircraft-CameraSystem .............................. 84 5.4.1FeaturePointPosition ............................ 85 5.4.2FeaturePointVelocity ............................ 85 5.5SystemFormulation ................................. 86 5.6Simulating ...................................... 89 6DISCERNINGMOVINGTARGETFROMSTATIONARYTARGETS ........ 90 6.1CameraMotionCompensation ........................... 90 6.2Classication .................................... 95 7HOMOGRAPHYAPPROACHTOMOVINGTARGETS ................ 98 7.1Introduction ..................................... 98 7.2StateEstimation ................................... 101 7.2.1SystemDescription ............................. 101 7.2.2HomographyEstimation .......................... 103 8MODELINGTARGETMOTION ............................ 111 8.1Introduction ..................................... 111 8.2DynamicModelingofanObject .......................... 111 8.2.1MotionModels ............................... 112 8.2.2StochasticPrediction ............................ 113 9CONTROLDESIGN ................................... 117 9.1ControlObjectives ................................. 117 6


............................... 118 9.2.1AltitudeControl ............................... 118 9.2.2HeadingControl ............................... 119 9.2.3DepthControl ................................ 121 10SIMULATIONS ...................................... 123 10.1Example1:FeaturePointGeneration ........................ 123 10.2Example2:FeaturePointUncertainty ....................... 126 10.2.1Scenario ................................... 126 10.2.2OpticFlow .................................. 128 10.2.3TheEpipolarConstraint ........................... 130 10.2.4StructureFromMotion ........................... 132 10.3Example3:Open-loopGroundVehicleEstimation ................ 133 10.3.1SystemModel ................................ 134 10.3.2Open-loopResults .............................. 135 10.4Example4:Closed-loopAerialRefuelingofaUAV ................ 138 10.4.1SystemModel ................................ 139 10.4.2ControlTuning ............................... 140 10.4.3Closed-loopResults ............................. 144 10.4.4UncertaintyAnalysis ............................ 148 11CONCLUSION ...................................... 151 REFERENCES ......................................... 154 BIOGRAPHICALSKETCH .................................. 164 7


Table page 3-1Solutionsforhomographydecomposition ....................... 64 10-1Statesofthecameras .................................. 123 10-2Limitsonimagecoordinates .............................. 123 10-3Statesofthefeaturepoints ............................... 124 10-4Aircraftstates ...................................... 125 10-5Imagecoordinatesoffeaturepoints ........................... 126 10-6Effectsofcameraperturbationsonopticow ...................... 129 10-7Effectsofcameraperturbationsonepipolargeometry ................. 131 10-8Effectsofcameraperturbationsonstructurefrommotion ............... 133 10-9Maximumvariationsinpositionduetoparametricuncertainty ............ 150 10-10Maximumvariationsinattitudeduetoparametricuncertainty ............. 150 8


Figure page 1-1TheUAVeet ...................................... 23 1-2AeroVironment'sMAV:TheBlackWidow ....................... 23 1-3TheUFMAVeet .................................... 24 1-4Refuelingapproachusingtheprobe-droguemethod .................. 28 1-5TrackingapursuitvehicleusingavisionequippedUAV ................ 30 1-6Closed-loopblockdiagramwithvisualstateestimation ................ 31 3-1Mappingfromenvironmenttoimageplane ....................... 46 3-2Imageplaneeldofview(topview) .......................... 46 3-3Radialdistortioneffects ................................. 51 3-4Geometryoftheepipolarconstraint ........................... 58 3-5Geometryoftheplanarhomography .......................... 62 4-1Featurepointdependenceonfocallength ........................ 68 4-2Featurepointdependenceonradialdistortion ..................... 68 5-1Body-xedcoordinateframe .............................. 78 5-2Camera-xedcoordinateframe ............................. 80 5-3Scenarioforvision-basedfeedback ........................... 81 6-1Epipolarlinesacrosstwoimageframes ......................... 91 6-2FOEconstraintontranslationalopticowforstaticfeaturepoints ........... 94 6-3Residualopticowfordynamicenvironments ..................... 95 7-1Systemvectordescription ................................ 102 7-2Movingtargetvectordescription ............................ 103 9-1Altitudeholdblockdiagram ............................... 118 9-2Headingholdblockdiagram .............................. 119 10-1Virtualenvironmentforexample1 ........................... 124 10-2Featurepointmeasurementsforexample1 ....................... 125 9


........................ 126 10-4Virtualenvironmentforexample2 ........................... 127 10-5Featurepointsacrosstwoimageframes ........................ 128 10-6Uncertaintyinfeaturepoint ............................... 128 10-7Uncertaintyresultsinopticow ............................ 129 10-8Nominalepipolarlinesbetweentwoimageframes ................... 130 10-9Uncertaintyresultsforepipolargeometry ........................ 131 10-10Nominalestimationusingstructurefrommotion .................... 132 10-11Uncertaintyresultsforstructurefrommotion ...................... 133 10-12Vehicletrajectoriesforexample3 ............................ 134 10-13PositionstatesoftheUAVwithon-boardcamera ................... 135 10-14AttitudestatesoftheUAVwithon-boardcamera ................... 135 10-15Positionstatesofthereferencevehicle ......................... 135 10-16Attitudestatesofthereferencevehicle ......................... 136 10-17Positionstatesofthetargetvehicle ........................... 136 10-18Attitudestatesofthetargetvehicle ........................... 136 10-19Normerror ........................................ 137 10-20Relativepositionstates ................................. 137 10-21Relativeattitudestates .................................. 137 10-22Virtualenvironment ................................... 138 10-23Inner-looppitchtopitchcommandBodeplot ..................... 141 10-24Pitchanglestepresponse ................................ 141 10-25Altitudestepresponse .................................. 142 10-26Inner-looprolltorollcommandBodeplot ....................... 143 10-27Rollanglestepresponse ................................. 144 10-28Headingresponse .................................... 144 10-29Open-loopestimationoftarget'sinertialposition .................... 145 10


.................... 145 10-31Normerrorfortargetstateestimates .......................... 146 10-32Closed-looptargetpositiontracking .......................... 146 10-33Positiontrackingerror .................................. 147 10-34Targetattitudetracking ................................. 147 10-35Trackingerrorinheadingangle ............................. 147 10-36Target'sinertialpositionwithuncertaintybounds ................... 149 10-37Target'sinertialattitudewithuncertaintybounds .................... 149 11














1 ].Thisincreaseincapabilityforsuchcomplextasksrequirestechnologyformoreadvancedsystemstofurtherenhancethesituationalawareness.Overthepastseveralyears,theinterestanddemandforautonomoussystemshasgrownconsiderably,especiallyfromtheArmedForces.Thisinteresthasleveragedfundingopportunitiestoadvancethetechnologyintoastateofrealizablesystems.Sometechnicalinnovationsthathaveemergedfromtheseefforts,fromahardwarestandpoint,consistmainlyofincreasinglycapablemicroprocessorsinthesensors,controls,andmissionmanagementcomputers.TheDefenseAdvancedResearchProjectsAgency(DARPA)hasfundedseveralprojectspertainingtotheadvancementofelectronicdevicesthroughsizereduction,improvedspeedandperformance.Fromthesedevelopments,thecapabilityofautonomoussystemhasbeendemonstratedonvehicleswithstrictweightandpayloadrequirements.Inessence,thecurrenttechnologyhasmaturedtoapointwhereautonomoussystemsarephysicallyachievableforcomplexmissionsbutnotyetalgorithmicallycapable.TheaerospacecommunityhasemployedmanyoftheresearchdevelopedforautonomoussystemsandappliedittoUnmannedAerialVehicles(UAV).Manyofthesevehiclesarecurrently 21


1 ].FuturemissionsenvisionUAVtoconductmorecomplextasksuchasterrainmapping,surveillanceofpossiblethreats,maritimepatrol,bombdamageassessment,andeventuallyoffensivestrike.Thesemissionscanspanovervarioustypesofenvironmentsand,therefore,requireawiderangeofvehicledesignsandcomplexcontrolstoaccommodatetheassociatedtasks.TherequirementsanddesignofUAVareconsideredtoenableaparticularmissioncapability.Eachmissionscenarioisthedrivingforceoftheserequirementsandaredictatedbyrange,speed,maneuverability,andoperationalenvironment.CurrentUAVrangeinsizefromlessthan1poundtoover40,000pounds.SomepopularUAVthatareoperational,intestingphase,andintheconceptphasearedepictedinFigure 1-1 toillustratethevariousdesigns.ThetwoUAVontheleft,GlobalHawkandPredator,arecurrentlyinoperation.GlobalHawkisemployedasahighaltitude,longendurancereconnaissancevehiclewhereasthePredatorisusedforsurveillancemissionsatloweraltitudes.Meanwhile,theremainingtwopicturespresentJ-UCAS,whichisajointcollaborationforboththeAirForceandNavy.ThisUAVisdescribedasamediumaltitudeyerwithincreasedmaneuverabilityoverGlobalHawkandthePredatorandisconsideredforvariousmissions,someofwhichhavealreadybeendemonstratedinight,suchasweapondeliveryandcoordinatedight.Theadvancementsinsensorsandcomputingtechnology,mentionedearlier,hasfacilitatedtheminiaturizationoftheseUAV,whicharereferredtoasMicroAirVehicles(MAV).Thescaleofthesesmallvehiclesrangesfromafewfeetinwingspandowntoafewinches.DARPAhasalsofundedtherstsuccessfulMAVprojectthroughAeroVironment,asshowninFigure 1-2 ,wherebasicautonomywasrstdemonstratedatthisscale[ 2 ].Thesesmallscalesallowhighlyagilevehiclesthatcanmaneuverinandaroundobstaclessuchasbuildingsandtrees.ThiscapabilityenablesUAVtooperateinurbanenvironments,belowrooftoplevels,toprovide 22


TheUAVeet thenecessaryinformationwhichcannotbeobtainedathigheraltitudes.ResearchersarecurrentlypursuingMAVtechnologytoaccomplishtheverysamemissionsstatedearlierfortheuniqueapplicationofoperatingatlowaltitudesinclutteredenvironments.Assensorandcontroltechnologiesevolve,theseMAVcanbeequippedwiththelatesthardwaretoperformadvancedsurveillanceoperationswherethedetection,tracking,andclassicationofthreatsaremonitoredautonomouslyonline.Althoughasinglemircoairvehiclecanprovidedistinctinformation,targetsmaybedifculttomonitorduetobothightpathandsensoreldofviewconstraints.ThislimitationhasmotivatedtheideaofacorporativenetworkoraswarmofMAVcommunicatingandworkingtogethertoaccomplishacommontask. Figure1-2. AeroVironment'sMAV:TheBlackWidow 23


3 4 ].Meanwhile,Stanfordhasexaminedmotionplanningstrategiesthatoptimizeighttrajectoriestomaintainsensorintegrityforimprovedstateestimation[ 5 ].TheworkatGeorgiaTechandBYUhasconsideredcorporativecontrolofMAVforautonomousformationying[ 6 ]andconsensusworkfordistributedtaskassignment[ 7 ].Alternatively,visionbasedcontrolhasalsobeenthetopicofinterestatbothGeorgiaTechandUF.ControlschemesusingvisionhavebeendemonstratedonplatformssuchasahelicopteratGeorgiaTech[ 8 ],whileUFimplementedaMAVthatintegratedvisionbasedstabilizationintoanavigationarchitecture[ 9 10 ].TheUniversityofFloridahasalsoconsideredMAVdesignsthatimprovetheperformanceandagilityofthesevehiclesthroughmorphingtechnology[ 11 13 ].FabricationfacilitiesatUFhaveenabledrapidconstructionofdesignprototypesusefulforbothmorphingandcontroltesting.TheeetofMAVproducedbyUFareillustratedinFigure 1-3 wherethewingspanofthesevehiclesrangefrom24indownto4in. Figure1-3. TheUFMAVeet ThereareanumberofcurrentdifcultiesassociatedwithMAVduetotheirsize.Forexample,characterizingtheirdynamicsunderightconditionsatsuchlowReynoldsnumbersisanextremelychallengingtask.Theconsequenceofincreasedagilityatthisscalealsogivesrisetoerraticbehaviorandaseveresensitivitytowindgustandotherdisturbances.Waszaketal.[ 14 ]performedwindtunnelexperimentson6inchMAVandobtainedtherequiredstabilityderivativesforlinearandnonlinearsimulations.AnothercriticalchallengetowardMAV 24




3 .Thisdissertationwillfocusonthemonocularcameracongurationtoaddressthestateestimationproblemregardingmovingtargets.TheadvantageofthesetechniquesbecomesmoreapparenttoUAVwhenappliedtoguidance,navigation,andcontrol.Bymountingacameraonavehicle,stateestimationofthevehicleandobjectsintheenvironmentcanbeachievedinsomeinstancesthroughvisionprocessing.Oncestateestimatesareknown,theycanthenbeusedinfeedback.Controltechniquescanthenbeutilizedforcomplexmissionsthatrequirenavigation,pathplanning,avoidance,tracking,homing,etc.Thisgeneralframeworkofvisionprocessingandcontrolhasbeensuccessfullyappliedtovarioussystemsandvehiclesincludingroboticmanipulators,groundvehicles,underwatervehicles,andaerialvehiclesbuttherestillexistssomecriticallimitations.Theproblematicissueswithusingvisionforstateestimationinvolvescameranonlinearities,cameracalibration,sensitivitytonoise,largecomputationaltime,limitedeldofview,andsolvingthecorrespondenceproblem.Aparticularsetoftheseimageprocessingissueswillbeaddresseddirectlyinthisdissertationtofacilitatethecontrolofautonomoussystemsincomplexsurroundings. 26


1. segmentingmovingtargetsfromstationarytargetswithinthescene 2. classifyingmovingtargetsintodeterministicandstochasticmotions 3. couplingthevehicledynamicsintothesensorobservations(i.e.images) 4. formulatingthehomographyequationsbetweenamovingcameraandtheviewabletargets 5. propagatingtheeffectsofuncertaintythroughthestateestimationequations 6. establishingcondenceboundsontargetstateestimationThedesignandimplementationofavision-basedcontrollerisalsopresentedinthisdissertationtoverifyandvalidatemanyoftheconceptspertainingtotrackingofmovingtargets. 27


15 ].Thedrogueisdesignedinanaerodynamicshapethatpermitstheextensionfromthetankerwithoutinstability.Theprobe-and-droguemethodisconsideredthepreferredmethodforAAR,mainlyduetothehighpilotworkloadincontrollingtheboom[ 16 ].Figure 1-4 illustratestheviewobservedbyreceiveraircraftduringtherefuelingprocesswherefeaturepointshavebeenplacedonthedrogue. Figure1-4. Refuelingapproachusingtheprobe-droguemethod VisioncanbeusedtofacilitatetheAARproblembyaugmentingtraditionalaircraftsensorssuchasglobalpositioningsystem(GPS)andinertialmeasurementunit(IMU).GighprecisionGPS/IMUsensorscanproviderelativeinformationbetweenthetankerandthereceiverthenvisioncanbeusedtoproviderelativeinformationonthedrogue.Theadvantagetovisioninthiscaseisitspassivenaturewhicheliminatessensoremissionsduringrefuelingoverenemyair 28


1-5 illustratesinasimulatedenvironmentthisscenariowhereaUAVobservesthe 29


Figure1-5. TrackingapursuitvehicleusingavisionequippedUAV 1-6 ,wherecommandsaresenttoavehiclebasedonthemotionsobservedintheimages.ThevehicleconsideredinthisdissertationispredominatelyassumedanautonomousUAV,butisgeneralizedforanydynamicalsystemwithpositionandorientationstates.TheblockspertainingtothisdissertationarehighlightedinFigure 1-6 intheimageprocessingblockandconsistsofthemovingobjectdetection,stateestimationofamovingobject,andclassifyingdeterministicversusstochasticmotion.Abriefdiscussionofeachtopicisdescribedinthissection,whilethedetailsarecoveredintheirrespectivechapters.Distinguishingmovingobjectsfromstationaryobjectswithamovingcameraisachallengingtaskinvisionprocessingandistherststepinthestateestimationprocesswhenconsideringadynamicscene.Thisinformationisextremelyimportantforguidance,navigation,andcontrolofautonomoussystemsbecauseitidentiesobjectsthatpotentiallycouldbeinapathforcollision.Forastationarycamera,movingobjectsinthescenecanbeextractedusing 30


Closed-loopblockdiagramwithvisualstateestimation simpleimagedifferencing,wherethestationarybackgroundissegmentedout;however,thisapproachdoesnotapplytomovingcameras.Inthecaseofamovingcamera,thebackgroundisnolongerstationaryanditbeginstochangeovertimeasthevehicleprogressesthroughtheenvironment.Therefore,theimagestakenbyamovingcameracontainthemotionduetothecamera,commonlycalledego-motion,andthemotionoftheobject.Techniquesthatinvolvecameramotioncompensationorimageregistrationhavebeenproposedtoworkwellwhenthereexistsnostationaryobjectsclosetothecamerawhichcausehighparallax.Thisdissertationwillestablishatechniquetoclassifyobjectsintheeldofviewasmovingorstationarywhileaccountingforstationaryobjectswithhighparallax.Therefore,withaseriesofobservationsofaparticularscene,onecandeterminewhichobjectsaremovingintheenvironment.Knowingwhichobjectsaremovingintheimagedictatesthetypeofimageprocessingrequiredtoaccuratelyestimatetheobject'sstates.Infact,theestimationproblembecomesinfeasibleforamonocularsystemwhenboththecameraandtheobjectaremoving.Thisunattainablesolutioniscausebyanumberoffactorsincluding1)inabilitytodecouplethemotionfromthecameraandtargetand2)failuretotriangulatethedepthestimateoftheobject.Forthisconguration,relativeinformationcanbeobtainedandfusedwithadditionalinformationforstateestimation.First,decouplingthemotionrequiresknowninformationregardingmotionofthecameraorthemotionoftheobject,whichcouldbeobtainedthroughothersensorssuch 31


5 ].Furthermore,theaccuracyofthestateestimatesbecomespoorforsmallbaselinecongurations,whichoccursforMAVusingstereovision.Theseissuesregardingtargetstateestimationwillbeconsideredinthisdissertationtoshowboththecapabilitiesandlimitationstowardautonomouscontrolandnavigation.Anotherimportanttaskinvolvedwithtargetestimationistodetermineapattern(ifany)intheobject'smotionbasedonthetimehistory.Theobjectscanthenbeclassiedintodeterministicandstochasticmotionsaccordingtopastbehavior.Withthisinformation,predictionmodelscanbemadebasedonpreviousimagestoestimatethepositionofanobjectatalatertimewithsomelevelofcondence.Thepredictedestimatescanthenbeusedinfeedbackfortrackingordockingpurposes.Forstochasticlyclassiedobjects,furtherconcernsregardingdockingorAARareimposedonthecontrolproblem.Theprimarytaskofstateestimation,forboththevehicleandobjectsintheenvironment,reliesonaccurateknowledgeoftheimagemeasurementsandtheassociatedcamera.Suchknowledgeisdifculttoobtainduetouncertaintiesinthesemeasurementsandtheinternalcomponentsofthecameraitself.Forinstance,theimagemeasurementscontainuncertaintiesassociatedwiththedetectionofobjectsintheimage,inadditiontonoisecorruption.Thesedrawbackshavepromptedmanyrobustalgorithmstoincreasetheaccuracyoffeaturedetectionwhilehandlingnoiseduringtheestimationprocess.Alternatively,manytechniqueshavebeenusedtoaccuratelyestimatetheinternalparametersofthecamerathroughcalibration.Theparametersthatdescribetheinternalcomponentsofthecameraarereferredtoasintrinsicparametersandtypicallyconsistoffocallength,radialdistortion,skewfactor,pixelsize,andopticalcenter.Thiscalibrationprocesscanbecomecumbersomeforalargenumberofcameras 32








1-6 illustratesthecomponentsofinterestdescribedinthisdissertationforstateestimationandtrackingcontrolwithrespecttoamovingobjectwhichinvolvesobjectmotiondetection,objectstateestimation,andobjectmotionmodelingandprediction.Theliteraturereviewofthesetopicsisgiveninthissection. 17 18 ]hasservedasafoundationformanyalgorithms.Thistechniquereliesonasmoothnessconstraintimposedontheopticowthatmaintainsaconstantintensityacrosssmallbase-linemotionofthecamera.Manytechniqueshavebuiltuponthisalgorithmtoincreaserobustnesstonoiseandoutliers.Oncefeaturetrackinghasbeenobtained,thenextprocessinvolvessegmentingtheimageformovingobjects.Theneedforsuchaclassicationisduethefactthatstandardimage 36


19 21 ].TechniquesformorerealisticapplicationsinvolveKalmanltering[ 22 ]toaccountforlightingconditionsandbackgroundmodelingtechniquesusingstatisticalapproaches,suchasexpectationmaximizationandmixtureofGaussian,toaccountforothervariationsinreal-timeapplications[ 23 28 ].Althoughthesetechniquesworkwellforstationarycameras,theyareinsufcientforthecaseofmovingcamerasduetothemotionofthestationarybackground.Motiondetectionusingamovingcamera,asinthecaseofacameramountedtoavehicle,becomessignicantlymoredifcultbecausethemotionviewedintheimagecouldresultfromanumberofsources.Forinstance,acameramovingthroughascenewillviewmotionsintheimagecausedbycamerainducedmotion,referredtoasegomotion,changesincameraintrinsicparameterssuchaszoom,andindependentlymovingobjects.Therearetwoclassesofproblemsconsideredinliteratureforaddressingthistopic.Therstconsidersthescenariowherethe3Dcameramotionisknownapriorithencompensationcanbemadetoaccountforthismotiontodeterminestationaryobjectsthroughanappropriatetransformation[ 29 30 ].Thesecondclassofproblemsdoesnotrequireknowledgeofthecameramotionandconsistsofatwostageapproachtothemotiondetection.Therststageinvolvescameramotioncompensationwhilethelaststageemploysimagedifferencingontheregisteredimage[ 31 ]toretrievenon-staticobjects.ThetransformationusedtoaccountforcameramotioniscommonlysolvedbyassumingthemajorityofimageconsistsofadominantbackgroundthatisstationaryinEuclideanspace[ 32 33 ].Thissolutionisobtainedthroughaleast-squaresminimizationprocess[ 32 ]orwiththeuseofmorphologicallters[ 34 ].Thetransformationsobtainedfromthesetechniquestypically 37


35 ]proposedauniedmethodtodetectmovingobjects.Thisproposedmethodhandlesvariouslevelsofparallaxintheimagethroughasegmentationprocessthatisperformedinlayers.Therstlayerextractsthebackgroundobjectswhicharefarawayfromthecameraandhavelowparallaxthroughageneraltransformationinvolvingcamerarotation,translation,andzoomthroughimagedifferencing.Thenextlayercontainstheobjectwithhighparallaxconsistingofbothobjectsclosetothecameraandobjectsthataremovingindependentlyofthecamera.Theparallaxisthencomputedfortheremainingpixelsandcomparedtoonepixel.Thisprocessseparatestheobjectswithintheimagebasedontheircomputedparallax.Theselectionmayinvolvechoosingapointonaknownstationaryobjectthatcontainshighparallaxsoanyobjectnotobeyingthisparallaxisclassiedasamovingobjectinthescene.Opticowtechniquesarealsousedtoestimatemovingtargetlocationsonceego-motionhasbeenestimated.Amethodthatcomputesthenormalimageowhasbeenshowntoobtainmotiondetection[ 36 ].Coordinatetransformationsaresometimesusedtofacilitatethisapproachtodetectingmotion.Forinstance,amethodusingcomplexlogmappingwasshowntotransformtheradialmotionsintohorizontallinesuponwhichverticalmotionindicateindependentmotion[ 37 ].Alternatively,sphericalmappingwasusedgeometricallytoclassifymovingobjectsbysegmentingmotionswhichdonotradiatefromthefocusofexpansion(FOE)[ 29 ]. 38


38 39 ]usedthecoplanarityconstraintalsoknownastheepipolarconstraint.Meanwhile,thesubspaceconstrainthasalsobeenemployedtolocalizecameramotion[ 40 ].Thesetechniqueshavebeenappliedtonumeroustypesofautonomoussystems.Themobileroboticcommunityhasappliedthesetechniquesforthedevelopmentofnavigationinvariousscenarios[ 41 45 ].TheapplicationshavealsoextendedintotheresearchofUAVforaircraftstateestimation.GurlandRotstein[ 46 ]wasthersttoextendthisapplicationintheframeworkofanonlinearaircraftmodel.Thisapproachusedopticalowinconjunctionwiththesubspaceconstrainttoestimatetheangularratesoftheaircraftandwasextendedin[ 47 ].Webbetal.[ 48 49 ]employedtheepipolarconstrainttotheaircraftdynamicstoobtainvehiclestates.ThefoundationforbothoftheseapproachesisaKalmanlterinconjunctionwithageometricconstrainttoestimatethecameramotion.SomeapplicationsforaircraftstateestimationhaveinvolvedmissionsforautonomousUAVsuchasautonomousnightlanding[ 50 ]androadfollowing[ 51 ]. 52 58 ].TheseapproachesemploythesubspaceconstrainttoreconstructfeaturepointpositionthroughanextendedKalmanlter.Severalsurveypapershavebeenpublisheddescribingthecurrentalgorithmswhilecomparingtheperformanceandrobustness[ 59 62 ].RobustandadaptivetechniqueshavebeenproposedthatuseanadaptiveextendedKalmanltertoaccountformodeluncertainties[ 63 ].Inaddition,Qianetal.[ 64 ]designedarecursiveHltertoestimatestructurefrommotioninthepresenceofmeasurementandmodeluncertaintieswhile 39


65 ]investigatedtheoptimalapproachestotargetstateestimationanddescribedtheeffectsoflinearsolutionsonvariousnoisedistributions. 66 ].ThismethodextendedthepreviousworkofBroidaetal.[ 67 ]thatonlyconsideredafeaturepointapproach.Forthecaseofmovingmonocularcameraconguration,theproblembecomesextremelydifcultduetotheadditionalmotionofthecamera.Oneapproachusedinliteraturerelevanttomonocularcamerasystemsisbearings-only-tracking.Inthisapproach,thereareseveralassumptionsmade:(i)thevehiclehasknowledgeofitsposition,(ii)anadditionalrangesensor,suchassonarorlaserrangender,isusedtoprovideabearingmeasurement,and(iii)animagemeasurementistakenforanestimateoflateralposition.Theinitialresearchhasinvolvedtheestimationprocessanddesignwithimprovementstotheperformance[ 68 72 ].ThisapproachwasimplementbyFlew[ 5 ]toestimatethemotionoftargetwithinacomputedcovariance.Guanghuietal.[ 73 ]providedamethodforestimatingthemotionofapointtargetfromknowncameramotion.Theroboticcommunityhasexaminedthetarget-motionestimationproblemfromavisualservocontrolframework.Trackingrelativemotionofamovingtargethasbeenshownusinghomography-basedmethods.Thesemethodshavebeendemonstratedtocontrolanautonomousgroundvehicletoadesiredposedenedbyagoalimage,wherethecamerawasmountedonthegroundvehicle[ 74 ].Chenetal.[ 75 76 ]regulatedagroundvehicletoadesiredposeusingastationaryoverheadcamera.Mehtaetal.[ 77 ]extendedthisconceptforamovingcamera,whereacamerawasmountedtoanUAVandagroundvehiclewascontrolledtoadesiredpose. 40


15 ]appliedavisionnavigationalgorithmcalledVisNAVthatwasdevelopedbyJunkinsetal.[ 78 ]toestimatethecurrentrelativepositionandorientationofthetargetdroguethroughaGaussianleast-squaresdifferentialcorrectionalgorithm.Thisalgorithmhasalsobeenappliedtospacecraftformationying[ 79 ]. 80 ]demonstratedthecontrolrequiredtograbanunknownmovingobjectwithroboticmanipulatorusinganauto-regressive(AR)model.Thismodelpredictsafuturepositionofthetargetbasedonvelocityestimatescomputedfromimagesequences.Foraerialvehicles,detectingotheraircraftintheskyiscriticalforcollisionavoidance.NASAhasconsideredvisioninthisscenariotoaidpilotsindetectingaircraftonacrossingtrajectory.AtechniquecombiningimageandnavigationdataestablishedapredictionmethodthroughaKalmanlterapproachtoestimatethepositionandvelocityofthetargetaircraftaheadintime[ 34 ].Similarly,theAARproblemrequiressomeformofmodelpredictionwhendockingtoamovingdrogue.Kimmettetal.[ 15 ]utilizedadiscretelinearmodelforthepredictionofthedrogue.Thepredictedstatesusedforcontrolwerecomputedusingthediscretemodel,thecurrentstates,andlightturbulenceasinputtothedroguedynamics.Successfuldockingwassimulatedforonlylightturbulenceandwithlowfrequencydynamicsimposedonthedrogue.NASAisextremelyinterestedinAARproblemandcurrentlyhasaprojectonthistopic.FlighttestshavebeenconductedbyNASAinanattempttomodelthedroguedynamics[ 81 ].Inthisstudy,theaerodynamiceffectsfromboththereceiveraircraftandthetankeraircraftwereexaminedonthe 41


82 ].Robustnesswasalsoanalyzedusingaleast-squaresolutiontoobtainanexpressionfortheerrorintermsofthemotionvariables[ 83 ].Theuncertaintyinvision-basedfeedbackisoftenchosenasvariationswithinfeaturepoints;however,uncertaintyinthecameramodelmayactuallybeanunderlyingsourceofthosevariations.Essentially,theuncertaintymaybeassociatedwiththeimageprocessingtoextractfeaturepointsorwiththecameraparametersthatgeneratedtheimage.Thepropercharacterizationofcamerauncertaintymaybecriticaltodeterminearealisticleveloffeaturepointuncertainty.Theanalysisofcamerauncertaintyistypicallyaddressedinaprobabilisticmanner.Alineartechniquewaspresentedthatpropagatesthecovariancematrixofthecameraparametersthroughthemotionequationstoobtainthecovarianceofthedesiredcamerastates[ 84 ].Ananalysiswasalsoconductedfortheepipolarconstraintbasedontheknowncovarianceinthecameraparameterstocomputethemotionuncertainty[ 85 ].AsequentialMonteCarlotechniquedemonstratedbyQianetal.[ 86 ]proposedanewstructurefrommotionalgorithmbasedonrandomsamplingtoestimatetheposteriordistributionsofmotionandstructureestimation.Theexperimentalresultsinthispaperrevealedsignicantchallengestowardsolvingforthestructureinthepresenceoferrorsincalibration,featurepointtracking,featureocclusion,andstructureambiguities. 42


87 ]intrafcsituations,lowaltitudeightofarotorcraft[ 88 ],avoidingobstaclesintheightpathofanaircraft[ 34 ],andnavigatingunderwatervehicles[ 89 ].Opticalowtechniqueshavealsobeenutilizedasatoolforavoidancebysteeringawayfromareaswithhighopticowwhichindicateregionsofcloseobstacles[ 90 ].Targettrackingisanotherdesiredcapabilityforautonomoussystems.Inparticular,themilitaryisinterestedinthistopicforsurveillancemissionsbothintheairandontheground.Thecommonapproachestotargettrackingoccurinbothfeaturepointandopticalowtechniques.Thefeaturepointmethodtypicallyconstrainsthetargetmotionintheimagetoadesiredlocationbycontrollingthecameramotion[ 91 92 ].Meanwhile,Frezzaetal.[ 93 ]imposedanonholonomicconstraintonthecameramotionandusedapredictiveoutput-feedbackcontrolstrategybasedontherecursivetrackingofthetargetwithfeasiblesystemtrajectories.Alternatively,opticalowbasedtechniqueshavebeenpresentedforrobotichand-in-eyecongurationtotracktargetsofunknown2Dvelocitieswherethedepthinformationis 43


94 ].Adaptivesolutionspresentedin[ 91 95 97 ]haveshowncontrolsolutionsfortargettrackingwithuncertaincameraparameterswhileestimatingdepthinformation.Thehomingcontrolproblemhasnumerousapplicationstowardautonomoussystemssuchasautonomousaerialrefueling,spacecraftdocking,missileguidance,andobjectretrievalusingarobtoticmanipulator.Kimmettetal.[ 15 98 ]developedacandidateautonomousprobe-and-drogueaerialrefuelingcontrollerthatusesacommandgeneratortracker(CGT)totracktime-varyingmotionsofanon-stationarydrogue.TheCGTisanexplicitmodelfollowingcontroltechniqueandwasdemonstratedinsimulationforamovingdroguewithknowndynamicssubjecttolightturbulence.Tandaleetal.[ 16 ]extendedtheworkofKimmettandValasekbydevelopingareferenceobserverbasedtrackingcontroller(ROTC)whichdoesnotrequireadroguemodelorpresumedknowledgeofthedrogueposition.Thissystemconsistofareferencetrajectorygenerationmodulethatsendscommandstoanobserverthatestimatesthedesiredstatesandcontrolfortheplant.Theinputtothiscontrolleristherelativepositionbetweenthereceiveraircraftandthedroguemeasuredbythevisionsystem.Asimilarvisionapproachtoaerialrefuelingisalsopresentedin[ 99 ],wheremodelsofthetankeranddrogueareusedinconjunctionwithaninferredcamera.Thedroguemodelusedinthispaperwastakenfrom[ 100 ]thatusesamulti-segmentapproachtoderivingthedynamicsofthehose.Meanwhile,Houshangietal.[ 80 ]consideredgraspingamovingtargetbyadaptivelycontrollingarobotmanipulatorusingvisioninteraction.Theadaptivecontrolschemewasusedtoaccountformodelingerrorsinthemanipulator.Inaddition,thispaperconsideredunknowntargetdynamics.Anauto-regressivemodelapproachwasusedtopredictthetarget'spositionbasedonpassedvisualinformationandanestimatedtargetvelocity.Experimentaltestcasesaredocumentedthatshowtrackingconvergence. 44


3-1 .Thevector,h,representsthevectorbetweenthecameraandafeaturepointintheenvironmentrelativetoadenedcamera-xedcoordinatesystem,asdenedbyI.ThisvectoranditscomponentsarerepresentedinEquation 3 45


Mappingfromenvironmenttoimageplane Amajorconstraintplacedonthissensoristhecamera'seldofview(FOV).HeretheFOVcanbedescribedasthe3Dregionforwhichfeaturepointsarevisibletothecamera;hence,featuresoutsidetheFOVwillnotappearintheimage.Thethreephysicalparametersthatdenethisconstraintaretheeldofdepth,thehorizontalangleandtheverticalangle.AtopviewillustrationoftheFOVcanbeseeninFigure 3-2 ,wherethehorizontalFOVisdenedbythehalfangle,gh,andthedistancetotheimageplaneisoflengthf.Likewise,asimilarplotcanbeshowntoillustratetheverticalangle,whichcanbedenedasgv. Imageplaneeldofview(topview) 46


3 ,whererh;visdenedasthelargestspatialextensioninthehorizontalandverticaldirections. 3 forthehorizontalcomponent. 3 fortheverticalangle. 3.2.1IdealPerspectiveAgeometricrelationshipbetweenthecamerapropertiesandafeaturepointisrequiredtodeterminetheimageplanecoordinates.Thisrelationshipismadebyrstseparatingthecomponentsofhthatareparalleltotheimageplaneintotwodirections.Theimageplanecoordinatesarethencomputedfromatangentrelationshipofsimilartrianglesbetweentheverticalandhorizontaldirectionsandthedepthwithascalefactoroffocallength.Thisrelationshipestablishesthestandard2Dimageplanecoordinatesreferredtoasthepin-holecameramodel[ 101 102 ].Equations 3 and 3 representageneralpin-holeprojectionmodel 47


3 and 3 reducetotheverycommonpin-holecameramodelandisrepresentedbyEquations 3 and 3 3 and 3 canbeexpressedinhomogeneouscoordinatesandisshowninEquation 3 3 .First,theimageplaneisdiscretizedintoasetofpixels,correspondingtotheresolutionofthecamera.Thisdiscretizationisbasedonscalefactorsthatrelatereal-worldlengthmeasuresintopixelunitsforboththehorizontalandverticaldirections.Thesescalingtermsaredenedassandsnwhichhaveunitsofpixelsperlength,wherethelengthcouldbeinfeetormeters.Ingeneral,thesetermsaredifferentbutwhenthepixelsaresquarethens=sn.Second,theoriginoftheimageplaneistranslatedfromthecenterofthe 48


3 ,wherepixelmapping,origintranslation,andskewnessareallconsidered. 3 isrewrittentoEquation 3 3 toobtainageneralequationthatmapsfeaturepointsintheinertialframetocoordinatesintheimageplaneforacalibratedcamera. 49


3 ,requiresaninniteseriesoftermstoapproximatethevalue. 3 and 3 ,mapsanundistortedimage,(0;n0),whichisnotmeasurableonaphysicalcamera,intoadistortedimage,(0d;n0d),whichisobservable[ 104 ].Thisdistortionmodelonlyconsidersthersttermintheinniteseriestodescriberadialdistortionandexcludestangentialdistortion.Thisapproximationindistortionhasbeenusedtogenerateanaccuratedescriptionofrealcameraswithoutadditionalterms[ 105 ], 3-1 ,attemptstomodelthecurvatureofthelensduringtheimageplanemapping.Thisdistortionintheimageplanevariesinanonlinearfashionbasedonposition.Thiseffectdemonstratesanaxisymmetricmappingthatincreasesradiallyfromtheimagecenter.AnexamplecanbeseeninFigure 3-3B and 3-3C whichillustrateshowradialdistortionchangesfeaturepointlocationsofaxedpatternintheimagebycomparingittoatypicalpin-holemodelshowninFigure 3-3A .Noticethedistortedimagesseemtotakeonaconvexorconcaveshapedependingonthesignofthedistortion. 50


B CFigure3-3. RadialDistortionEffectsforA)f=0:5d=0,B)f=0:5d=0:0005,andC)f=0:5d=+0:0005 3 .Assuch,theseparametersaretermedtheintrinsicparametersandarefoundthroughcalibration.Afeaturepointmustbeanalyzedwithrespecttotheseintrinsicparameterstoensureproperstateestimation.Theradialdistancefromafeaturepointtothecenteroftheimageisdependentonboththerelativepositionsofcameraandfeaturealongwiththefocallength.Thisradialdistanceisalsorelatedviaanonlinearrelationshiptotheradialdistortion.Clearlyanyanalysisofthefeaturepointswillrequireestimationofthecameraparameters.Chapter 4 willdiscussatechniquethatconsidersboundeduncertaintytowardtheintrinsicparametersandestablishesaboundedconditiononthefeaturepointpositions. 51


3 and 3 [ 102 103 ].Theimagecoordinates(;n)intheseexpressionsarecomputedusingeitherEqaution 3 orEquation 3 dependingonthecameramodel. 3 [ 102 103 ].Thepixelvalueswithinthesearchwindowaredenedasx. 3 .IfEquation 3 issatisedthenthisisavalidfeaturepointbasedontheuserscriterion[ 102 103 ].Thisselectionisafunctionofboththewindowsize,W,andthethreshold,t. 52


106 ].ThismethodcanbeextendedtoedgedetectionbyconsideringthestructureofthesingularvaluesofG.AnexampleofthisalgorithmistheCannyedgedetector[ 107 ]. 3 3 3 53


3 .Oneimportantlimitationofthiscriterionoccurswhenthewindowinbothimagescontainsrelativelyconstantintensityvalues.Thisresultsintheapertureproblemwhereanumberofsolutionsforhareobtained.Therefore,duringthefeatureselectionprocessit'sbenecialtochoosefeaturesthatcontainuniqueinformationinthiswindow. 3 forsmallbaselinetracking:(1)usingthebrightnessconsistencyconstraintand(2)applyingthesumofsquareddifferences(SSD)approach.Eachofthesetechniquesemploysatranslationalmodeltodescribetheimagemotion.Therefore,ifoneassumesasimpletranslationalmodelthenthegeneraltransformationisshowninEquation 3 3 intoEquation 3 whileinitiallyneglectingthenoiseterm.ApplyingtheTaylorseriesexpansiontothisexpressionaboutthepointofinterest,x,whileretainingonlythersttermintheseriesresultsinEquation 3 dt+I 3 inmatrixformresultsinEquation 3 dt;dn 3 constitutes1equationwith2unknownvelocities;therefore,anotherconstraintisneededtosolvethisproblem.Auniquesolutionforthevelocitiescanbedeterminedbyenforcinganadditionalconstraintontheproblem,whichentailsrestrainingregionstoalocalwindowthatmovesatconstantvelocity.Upontheseassumptiononecanminimizetheerror 54


3 3 3 .Thenalsolutionforthepixelvelocityisfoundthroughaleast-squaresestimategiveninEquation 3 .Theseimagevelocitiesarealsoreferredtoastheopticow.Oncetheopticowiscomputedforafeaturepointthentheimagedisplacementforfeaturetrackingistrivialtond. 3 ,attemptstoestimatetheDxwhilenotrequiringthecomputationofimagegradients.Thisapproachalsoemploysthetranslationalmodeloverawindowedregion.Themethodconsidersthepossiblerangethatwindowcouldmove,danddn,inthetime,dt.Thisconsistencyconstraintthenleadstoaproblemofminimizingtheerroroverthepossiblewindowswithinthedescribedrange.ThiserrorfunctionisdescribedmathematicallyinEquation 3 17 ].Forlargebaselinetrackingsimpletranslationalmodels 55


4 3 and 3 .Thevelocityexpressions,showninEquations 3 and 3 ,describethemovementoffeaturepointsintheimageplaneandiscommonlyreferredtoinliteratureastheopticow. 3 and 3 whileassumingc=0isasfollows 56


3-4 whereh1andh2denotethepositionvectorsofthefeaturepoint,P,inthecamerareferenceframes.Also,thevaluesofx1andx2representthepositionvectorsprojectedontothefocalplanewhileTindicatesthetranslationvectoroftheoriginofthecameraframes.AgeometricrelationshipbetweenthevectorsinFigure 3-4 isexpressedbyintroducingRasarotationmatrix.Thisrotationmatrixincludestheroll,pitchandyawanglesthattransformthecameraframesbetweenmeasurements.TheresultingepipolarconstraintisexpressedinEquation 3 3 ,assumesapin-holecamerawhichiscolinearwithitsprojectionintothefocalplane. 57


Geometryoftheepipolarconstraint TheexpressionsinEquation 3 andEquation 3 reectthatthescalartripleproductofthreecoplanarvectorsiszero,whichformsaplaneinspace.Theserelationshipscanbeexpandedusinglinearalgebra[ 102 103 ]togenerateastandardformoftheepipolargeometryasinEquation 3 .Thisnewformindicatesarelationshipbetweentherotationandtranslation,writtenastheessentialmatrixdenotedasQ,totheintrinsicparametersofthecameraandassociatedfeaturepoints.Inthiscase,theequationisderivedforasinglefeaturepointthatiscorrelatedbetweentheframes, 58


3 withl1andl2representingtheepipolarlinesinimage1andimage2beingproportionaltotheessentialmatrix,respectfully. 3 and 3 arerewrittenintermsofthefundamentalmatrix,F,andareshowninEquations 3 and 3 3 3 whichsolvesfortheentriesoftheessentialmatrix.ThisalgorithmwasdevelopedbyLonguet-Higgins[ 39 ]andisdescribedinthissection.TheexpressioninEquation 3 canactuallybeexpressedasinEquation 3 usingadditionalargumentsfromlinearalgebra[ 102 103 ].Thevector,q2R9,containsthestackedcolumnsoftheessentialmatrixQ. 3 ,foreachfeaturepointwheretheentriesoftheessemtialmatrixarestackedinthevectorq.Asetofrowvectorsarestackedtoformamatrix,C,ofnmatchedfeaturepointsand 59


3 .ThematricxC,showninEquation 3 ,isan9matrixofstackedfeaturepointsmatchedbetweentwoviews. 3 existsusingalinearleast-squaresapproachonlyifthenumberofmatchedfeaturesineachframeisatleast8suchthatrank(C)=8.Additionally,morefeaturepointswillobviouslygeneratemoreconstraintsand,presumably,increaseaccuracyofthesolutionduetotheresidualsoftheleast-squares.Inpractice,theleast-squaressolutiontoEquation 3 willnotexistduetonoise;therefore,aminimizationisusedtondanestimateoftheessentialmatrix,asshowninEquation 3 3 3 ,wherethetranslationTisfounduptoascalingfactor.Thesefoursolutions,whichconsistofallpossiblecombinationsofRandT,arecheckedtoverifywhichcombinationgeneratesapositivedepth 60


102 103 ].Whenthissituationoccursonemustusetheplanarhomographyapproach,whichisthetopicofthenextsection. 102 103 ].Figure 3-5 depictsthegeometryinvolvedwithplanarhomography.Thefundamentalrelationshipexpressingapointfeaturein3DspaceacrossasetofimagesisgiventhrougharigidbodytransformationshowninEquation 3 61


Geometryoftheplanarhomography Ifanassumptionismadethatthefeaturepointsarecontainedonthesameplane,thenanewconstraintinvolvingthenormalvectorcanbeestablished.DenoteN=[n1;n2;n3]Tasthenormalvectoroftheplanecontainingthefeaturepointsrelativetocameraframe1.ThentheprojectionontotheunitnormalisshowninEquation 3 ,whereDistheprojecteddistancetotheplane. 3 intoEquation 3 resultsinEquation 3 62


3 canbeextendedtoimagecoordinatesthroughEquation 3 3 withtheskewsymmetricmatrixbx2resultsintheplanarhomographyconstraintshowninEquation 3 3 canberewrittentoEquation 3 3 requiresatleastfourfeaturepointcorrespondences.TheseadditionalconstraintscanbestackedtoformanewconstraintmatrixY,asshowninEquation 3 3 intermsofthenewconstraintmatrixresultsinEquation 3 102 103 ],showninEquation 3 fortheunknownscalerl. 63


3 3 ,thatarepreservedinthehomographymappingandwillfacilitateinthedecompositionprocess. 3 willestablishahomographysolutionexpressedintermsoftheseknownvariables. ThefoursolutionsareshowninTable 3-1 intermsofthematricesgiveninEquations 3 3 andthecolumnsofthematrixV.Noticethetranslationcomponentisestimateduptoa1 Table3-1. Solutionsforhomographydecomposition Solution1 64


3-4 andassumesthattherotation,R,andtranslation,T,betweenframesisknown.Giventhat,thecoordinatesofh1andh2canbecomputed.Recall,thefundamentalrelationshiprepeatedhereinEquation 3 3 andEquation 3 .Theserelationshipsallowsomecomponentsofhxandhytobewrittenintermsofandnwhichareknownfromtheimages.Thus,theonlyunknownsarethedepthcomponents,h1;zandh2;z,foreachimage.TheresultingsystemcanbecastasEquation 3 andsolvedusingaleast-squaresapproach. 3 usingz=[h2;z;h1;z]asthedesiredvectorofdepths. 3 obtainsthedepthestimatesofafeaturepointrelativetobothcameraframes.Thisinformationalongwiththeimageplanecoordinatescanbeusedtocompute(h1;x;h1;y)and(h2;x;h2;y)bysubstitutingthesevaluesbackintoEquations 3 and 3 .Theresultingcomponentsofh1canthenbeconvertedtothecoordinateframeofthesecondimageanditshouldexactlymatchh2.Thesevalueswillnevermatchperfectlydueto 65


3 isverysensitivetotheseuncertainties.Chapter 4 willdiscussamethodtoobtainuncertaintyboundsontheSFMestimatesbasedonthesourcesdescribed. 66


3 .Oncefeaturepointsarelocatedandtrackedacrossimages,anumberstateestimationalgorithms,suchasopticow,epipolarconstraint,andstructurefrommotion,canbeemployed.Althoughcameracalibrationtechniqueshaveproventoprovideaccurateestimatesoftheintrinsicparameters,theprocesscanbecumbersomeandtimeconsumingwhenusingalargequantityoflowqualitycameras.Thischapterdescribesquantitativelytheeffectsonfeaturepointpositionduetouncertaintiesinthecameraintrinsicparametersandhowthesevariationsarepropagatedthroughthestateestimationalgorithms.Thisdeterministicapproachtouncertaintyisanefcientmethodthatdeterminesalevelofboundedvariationsonstateestimatesandcanbeusedforcameracharacterization.Inotherwords,themaximumallowablestatevariationinthesystemwillthendeterminetheaccuracyrequiredinthecameracalibrationstep. 3-1 .TheresultingvaluesarerepeatedinEquations 4 and 4 asafunctionoffocallength,f,andradialdistortion,d,intermsofthecomponentsofh. 67


4-1 ,isdependentonboththerelativepositionsofcameraandthefeature.Thisradialdistance,asshowninFigure 4-2 ,isalsorelatedviaanonlinearrelationshiptotheradialdistortion.Theanalysisofthefeaturepointswillrequireestimationofthecameraparameters. BFigure4-1. FeaturePointDependenceonFocalLengthforA)f=0:5andB)f=0:25 BFigure4-2. FeaturePointDependenceonRadialDistortionforA)d=0:0001andB)d=0:0005 4 ,showstherangeofvaluesthatmustbeconsideredforanominalestimate, 68


4 presentstherangeofvaluesforradialdistortion. 4 andEquation 4 aresubstitutedintothecameramodelofEquation 4 andEquation 4 .TheresultingexpressionsforfeaturepointsarepresentedinEquations 4 and 4 4 andEquation 4 donotdependonuncertaintysotheseportionsrepresentnominalvalues,oandno,whicharethecorrectlocationsoffeaturepoints.Thesecondtermswhichincludedfandddtermsaretheuncertainty,danddn,ineachfeaturepointwhichareboundedinnormbyDandDn.Assuch,thefeaturepointsmaybewrittenasinEquation 4 andEquation 4 toreecttheuncertainty. 69


4 andEquation 4 3 andEquation 3 .Inpractice,thevelocitiesarecomputedbysubtractinglocationsofafeaturepointacrossapairofimagestakenatdifferenttimes.Suchanapproachassumesthatafeaturepointcanbetrackedandcorrelatedbetweentheseframes.TheopticowisthengivenasJusingEquation 4 forafeaturepointat1andn1inoneframeand2andn2inanotherframe. 70


4 andEquation 4 ,canbesubstitutedintoEquation 4 tointroduceuncertainty.TheresultingexpressioninEquation 4 separatestheknownfromunknownelements. 4 wheretheuncertaintyisboundedbyDJ2R. 4 .Theactualboundsonthefeaturepoints,asnotedinEquation 4 andEquation 4 ,variesdependingonthelocationofeachfeaturepointsoboundsofD1andD2aregivenforeachverticalcomponentandDn1andDn2aregivenforeachhorizontalcomponent.Assuch,theboundonvariationisnotedinEquation 4 asspecictotheh1andh2usedtogatherfeaturepointsineachimage. 3 ,requiresapin-holecamerawhoseintrinsicparametersareexactlyknown.Suchasituationisobviouslynotrealisticsotheeffectofuncertaintycanbedetermined.Anon-idealcamerawilllosethe 71


4 andEquation 4 ,whichareactuallycausedbyuncertaintyinthecameraparametersasnotedinEquation 4 andEquation 4 .TheconstraintmatrixfromEquation 3 canthenbewrittenasanominalcomponent,Co,plussomeuncertainty,dC,asinEquation 4 4 andEquation 4 .TheithrowofthismatrixcanthenbewrittenasEquation 4 3 ,whenincludingtheuncertaintymatrixinEquation 4 ,willexist;however,thatsolutionwilldifferfromthetruesolutionorthenominalsolution.Essentially,thesolutioncanbeexpressedasthenominalsolution,qo,andanuncertainty,dq,asinEquation 4 .Thisperturbedsystemcannowbesolvedusingalinearleast-squaresapproachfortheentriesoftheessentialmatrix. 4 hasvariationwhichwillbenormboundedbyDqasinEquation 4 whichindicatestheworse-casevariationimposedontheentriesofq. 72


4 .ThisboundusestherelationshipbetweenuncertaintiesinEquation 4 throughtheconstraintinEquation 4 .Also,thesizeofthisuncertaintydependsonthelocationofeachfeaturepointsotheboundsisnotedasspecictotheh1andh2obtainedfromFigure 3-4 4 ,canthenbeuseddirectlytocomputethevariationinstateestimates.Theentriesofqarerstarrangedbackintomatrixformtoconstructthenewessentialmatrixthatincludesparametervariations.ThisnewessentialmatrixisthendecomposedusingSVDtechniquesdescribedinSection 3.6.1 3 .SubstitutingEquation 4 andEquation 4 intoEquation 3 resultswithavariationinthesystemmatrixY.Likewise,thenewsystemmatrixwithuncertainintrinsicparamterscanbewrittenasanominalmatrix,Y0plussomevariation,dY,asshowninEquation 4 4 andEquation 4 .correspondingly, 73


4 3 ,whenincludingtheuncertainmatrixinEquation 4 ,willexist;however,thatsolutionwilldifferfromthetruesolution.Essentially,thesolutioncanbeexpressedasthenominalsolution,ho,andanuncertainty,dh,asinEquation 4 4 hasvariationwhichwillbenormboundedbyDhasinEquation 4 4 .ThisboundusestherelationshipbetweenuncertaintiesinEquation 4 throughtheconstraintinEquation 4 .Also,thesizeofthisuncertaintydependsonthelocationofeachfeaturepointsotheboundsisnotedasspecictotheh1andh2obtainedfromFigure 3-4 4 ,canthenbeuseddirectlytocomputethevariationinstateestimates.Theentriesofharerstarrangedbackintomatrixformtoconstructthenewhomographymatrixthatincludesparameter 74


3.6.3 3 forthestructurefrommotionrelationship.Assuch,thematrixshouldbewrittenintermsofanominalvalue,Ao,andanuncertainperturbation,dA,asinEq. 4 4 andEquation 4 intoEquation 3 .TheperturbationisthenwrittenasEquation 4 3 whenconsideringEquation 4 willobviouslyresultinadepthestimatethatdiffersfromthecorrectvalue.Denezoastheactualdepthsthatwouldbecomputedusingtheknownparametersofthenominalcameraanddzasthecorrespondingerrorintheactualsolution.Theleast-squaresproblemcanthenbewrittenasEquation 4 andsolvedusingapseudo-inverseapproach. 4 .Thisrangeofsolutionswillliewithintheboundedrangedeterminedfromtheworst-casebound. 75


4 .Thisboundnotesthattheboundonvariationsinfeaturepoints,andultimatelytheboundonsolutionstostructurefrommotion,dependsonthelocationofthosefeaturepoints. 76


3 toderivethesystemequations. 5-1 alongwiththerespectiveorigins.Thebody-xedcoordinatesystemhastheoriginlocatedatthecenterofgravityoftheaircraft.Theaxesareorientedsuchthatb1alignsoutthenoseandb2alignsouttherightwingwithb3pointedoutthebottom.Themovementoftheaircraft,whichincludesaccelerating,willobviouslyaffectthecoordinatesystem;consequently,thebody-xedcoordinatesystemisnotaninertialreferenceframe. 77


Body-xedcoordinateframe Theorientationanglesoftheaircraftareofparticularinterestformodelingavision-basedsensor.Therollangle,f,describesrotationaboutb1,thepitchangle,q,describesrotationaboutb2andtheyawangle,y,describesrotationaboutb3.ThetransformationfromavectorrepresentedintheEarth-xedcoordinatesystemtothebody-xedcoordinatesystemisrequiredtorelateon-boardmeasurementstoinertialmeasurements.Thistransformation,giveninEquation 5 ,usesREBwhichareEulerrotationsofroll,pitchandyaw[ 29 108 ], 5 .Theorderofthismatrixmultiplicationneedstobemaintainforcorrectcomputation. 78


5 5 torepresenttheserates. 5-2 ,usethetraditionalchoiceofi3aligningthroughthecenterofviewofthecamera.Theremainingaxesareusuallychosenwithi2alignedrightoftheviewandi1alignedoutthetopalthoughsomevariationinthesechoicesisallowedaslongastheresultingaxesretaintheright-handedproperties.Thedirectionofthecamerabasisvectorsaredenedthroughthecamera'sorientationrelativetothebody-xedframe.Thisframeworkisnotedasthecamera-xedcoordinatesystembecausetheoriginisalwayslocatedataxedpointonthecameraandmovesinthesamemotionasthecamera.Thecameraisallowedtomovealongtheaircraftthroughadynamicmountingwhichadmitsbothrotationandtranslation.Thisfunctionalityenablesthetrackingoffeatureswhilethevehiclemovesthroughanenvironment.Theoriginofthecamera-xedcoordinatesystemisattachedtothismovingcamera;consequently,thecamera-xedframeisnotaninertialreference.A6degree-of-freedommodelofthecameraisassumedwhichadmitsafullrangeofmotion.Figure 5-2 alsoillustratesthecamera'ssensingconewhichdescribesboththeimageplaneandtheeldofviewconstraint. 79


Camera-xedcoordinateframe Similartothebody-xedcoordinateframe,atransformationcanbedenedforthemappingbetweenthebody-xedframe,Bandthecameraframe,IasseeninEquation 5 5 ,similartothebody-xedrotationmatrix.Theorientationanglesofthecameraarerequiredtodeterminetheimagingusedforvision-basedfeedback.Therollangle,fc,describesrotationabouti3,thepitchangle,qc,describesrotationabouti2andtheyawangle,yc,describesrotationabouti1. 5 willtransformavectorinbody-xedcoordinatestocamera-xedcoordinates.Thistransformationisrequiredtorelatecamerameasurementstoon-boardvehiclemeasurementsfrominertialsensors.Thematrixagaindependsontheangular 80


5 torepresenttheseangles. 5-3 ,thusrelatesthecameraandtheaircrafttothefeaturepointalongwithsomeinertialorigin. Figure5-3. Scenarioforvision-basedfeedback 81


5 andEquation 5 aretypicallyrepresentedintheinertialreferenceframerelativetotheEarth-axisorigin. 5 ,istypicallygivenwithrespecttothebody-axisorigin.Thischoiceofcoordinatesystemsreectsthatthecameraisintrinsicallyaffectedbyanyaircraftmotion. 3 todescribetherelativepositionbetweenthecameraandthefeaturepoint.Recall,thisvectorwasgiveninthecamera-xedcoordinatesystemtonotetheresultingimageisdirectlyrelatedtopropertiesrelativetothecamera.TherepresentationofhisrepeatedhereinEquation 5 forcompleteness. 5 isused.Thisexpressionincorporatesthetranslationsinvolvedwiththeoriginsofeachcoordinateframethroughaseriesofsingle-axisrotationsuntilthecorrectframeisreached. 82


108 110 ]andarerepeatedinEquation 5 to 5 foroverallcompleteness.Fxmgsinq=m(u+qwrv) 5 .Theaircraftstatesofinterestforthecameramotionsystemconsistofthepositionandvelocityoftheaircraft'scenterofmass,TEBandvb,theangularvelocity,w,andtheorientation 83


5 .AsstatedinEquation 5 ,theaircraft'svelocityisexpressedinthebody-xedcoordinateframe.Eachoftheseparameterswillappearexplicitlyintheaircraft-cameraequations. 84


5-3 forafeaturepointrelativetotheinertialframe.Therefore,thevectorsumcanbeusedtosolvefortherelativepositionbetweenthecameraanda3Dfeaturepoint.AftermakingthepropercoordinatetransformationsbyusingEquations 5 and 5 ,thisrelativepositioncanbeexpressedincameraframe,I,asshowninEquation 5 5 intoEquations 3 and 3 animagecanbeconstructedasafunctionofaircraftstates.Themajorassumptionoftheseequationsispriorknowledgeofthefeaturepointlocationrelativetotheinertialframe,whichmaybeprovidedbyGPSmaps.Furthermore,theimageresultsobtainedcanalsobepassedthroughEquations 3 and 3 toaddtheeffectsofradialdistortion.Thedistortedimagewillprovideamoreaccuratedescriptionofanimageseenbyaphysicalcamera,assumingtheintrinsicparametersofthecameraareknown. 5 withrespecttotheinertialframe,asshowninEquation 5 dt(h)=Ed dt(x)Ed dt(TEB)Ed dt(TBI)(5) 85


5 cannowberewrittentoEquation 5 foranon-stationaryfeaturepoint. dt(h)=xTEBBd dt(TBI)wTBIEwIh(5)ThisequationcanbereducedfurtherifthecamerasareconstrainedtohavenotranslationrelativetotheaircraftsoBd dt(TEI)=0.Alternatively,thistermisretainedinthederivationtoallowthisdegreeoffreedominthecamerasetup.Theangularvelocity,EwI,canbefurtherdecomposedusingtheAdditionTheorem.ThenalstepimplementsEquations 5 and 5 totransformeachtermintothecameraframe.Aftersomemanipulation,theexpressionforthevelocityofafeaturepointrelativetothecameraframeresultsinEquation 5 5 and 5 intoEquations 3 and 3 .Thisresultwillprovideadescriptionoftheopticalowforeachfeaturepointformedbyeitherthecameratravelingthroughtheenvironmentorthemotionofthefeaturepointsthemselves.ToincorporateradialdistortioneffectsintotheopticowcomputationrequirestheadditionalsubstitutionintoEquations 3 and 3 86


5 5 3 3 3 ,and 3 .Arrangingtheparametersforthekthcameraintoasinglevector,asshowninEquation 5 ,resultsthenintheformulationofagenericaircraft-camerasystemwithkcamerasallhavingindependentmotionthattracknfeaturepointsisobtained. 5 .ThisvectorcanbeextendedtoincludeothercamerafeaturessuchasCCDarraymisalignment,skewness,etc.ThefocalplanepositionscanthenbeassembledintoavectorofobservationsasshowninEquation 5 ,wherennumberoffeaturepointsareobtained.Likewise,thestatesoftheaircraftcanbecollectedandrepresentedasastatevectorasshowninEquation 5 .Inaddition,theinitialstatesofthevehiclearedenedasX0. 5 and 5 .TheobservationsusedinthisdissertationconsistofmeasureableimagesshowninEquations 3 and 3 whichcapturenonlinearitiessuchasradialdistortion.Thissystem,whichmeasuresimageplaneposition,isdescribedmathematically 87


5 5 ,thenadifferentsetofequationscanbeobtainedwhichwillbereferredtoastheOpticFlowFormofthegoverningaircraft-cameraequationsofmotion.ThissystemisgiveninEquation 5 ,whichusestheopticowexpressiongiveninEquations 3 and 3 astheobservations.X(t)=F(X(t);U(t);a(t);t) 88




3 andhowthesefeaturesrelatetothestatesofanaircraftinChapter 5 ,theeffectsofindependentlymovingobjectsneedtobehandledinadifferentmanner.Forcasesinvolvingastationarycamera,suchasinsurveillanceapplications,simplelteringandimagedifferencingtechniquesareemployedtodetermineindependentlymovingobjects.Althoughthesetechniquesworkwellforstationarycameras,adirectimplementationtomovingcameraswillnotsufce.Foramovingcamera,theapparentimagemotioniscausedbyanumberofsources,suchascamerainducedmotion(i.e.ego-motion)andthemotionduetoindependentlymovingobjects.Acommonapproachtodetectingmovingobjectconsidersatwostageprocessthatincludes(i)acompensationroutinetoaccountforcameramotionand(ii)aclassicationschemetodetectindependentlymovingobjects. 3.6 .Thesecondapproachusesthesmoothnessconstraintinattempttominimizethesumofsquaredifferences(SSD)overeitheraselectnumberoffeaturesortheentireoweld.Thisapproachassumesthestationary 90


3.6 .Theepipolarconstraintcanbeusedtorelatefeaturepointsacrossimagesequencesthrougharigidbodytransformation.TheepipolarlinesofastaticenvironmentarecomputedusingEquation 3 orEquation 3 dependingiftheessentialmatrixorthefundamentalmatrixisrequired.AnillustrationofthecomputedepipolarlinesisdepictedinFigure 6-1 forastaticenvironmentobservedbyamovingcamera.Noticeforthisstaticcase,thefeaturepointsinthesecondimage(therightimagecontainingtheoverlaidepipolarlines)areshowntoliedirectlyontheepipolarlines. BFigure6-1. EpipolarLinesAcrossTwoImageFrames:A)InitialFeaturePointsandB)FinalFeaturePointswithOverlayedEpipolarLines Oncecameramotionestimationhasbeenfound,theepipolarlinescanbeusedasanindicationofmovingobjectsintheimage.Forinstance,thefeaturepointscorrespondingtothestationarybackgroundwilllieontheepipolarlineswhilethefeaturepointscorrespondingtomovingobjectswillviolatethisconstraint.Similarly,thecomputationofopticalowcanalsobeusedfordetectingindependentlymovingobjects.Incomputingtheopticalow,themotioninducedbythecameraalongwithmovingobjectsisfusedtogetherinthemeasuredimage.Recall,theopticowexpressions 91


3 and 3 orEquations 3 and 3 forradialdistortion.Decomposingtheopticalowintoitscomponentsofcamerarotation(r;nr)andtranslation(t;nt)andindependentlymovingobjects(i;ni)facilitatesthedetectionproblem.Therefore,thecomponentsoftheopticalowcanbewrittenasinEquation 6 5 :thetranslationalvelocity[u;v;w]Tandtheangularvelocity[p;q;r]Tofthecamera.TheresultingexpressionsareshowninEquations 6 and 6 andappliesonlytofeaturesstationaryintheenvironment.ThedetailsdescribingthesubstitutionofthecameramotionstatesaredescribedinChapter 5 hz1 111 ]thattherotationalstates[p;q;r]Tcanbeestimatedaccuratelyforastaticenvironmentthroughanonlinearminimizationprocedurefornfeatureswheren6.Theapproachusedavector-valuedoweldJ(x)andisgiveninEquation 6 92


6 iscomposedofunknownvehiclestatesanddepthparameters. 6 thatminimizesthemagnitudeofthecostfunction. 2kJ(x)k2(6)Thesameapproachistakenherewithcaution.Recallthatthemeasuredopticalowalsocontainsmotionduetoindependentlymovingobjectsinadditiontotheinducedopticalowcausedbythecameramotion.Ingeneral,thesevariationsinthemeasuredopticalowwillintroduceerrorintothe[p;q;r]Testimates.Ifsomeassumptionsaremaderegardingtherelativeopticalowbetweenthestaticenvironmentandmovingobjects,thenerrorsinthestateestimatescanhaveminimaleffect.Forinstance,ifthestaticportionofthesceneisassumedtobethedominantmotionintheopticalowthentheestimateswillcontainminimalerrors.Employingthisassumption,estimatesfortheangularvelocities[p;q;r]Tofthecamera/vehicleareobtained.SubstitutingtheseestimatesintoEquation 6 resultsinestimatesfortherotationalportionoftheopticalow,asshowninEquation 6 6 .TheresidualopticalowRes;nRescontainsonlythecomponentsofthecameratranslationandindependentlymovingobjects.Fromthisexpression,constraintscanbeemployedonthecamerainducedmotiontodetectindependentlymovingobjects. 93


6-2 .Consequently,featurepointsthatviolatethisconditioncanbeclassiedasindependentlymovingobjects.Thischaracteristicobservedfromstaticfeatureswillbethebasisfortheclassicationscheme. Figure6-2. FOEconstraintontranslationalopticowforstaticfeaturepoints TheresidualopticalowmaycontainindependentlymovingobjectswithintheenvironmentthatradiatefromtheirownFOE.AnexampleofasimplescenarioisillustratedinFigure 6-3 forasinglemovingobjectontheleftandasimulationwithsyntheticdataoftwomovingvehiclesontheright.NoticethetwoprobableFOEsinpictureontheleft,onepertainingtothestaticenvironmentandtheotherdescribingthemovingobject.Inaddition,theepipolarlinesofthetwodistinctFOEsintersectatdiscretepointsintheimage.Thesepropertiesofmovingobjectsarealsoveriedinthesyntheticdatashownintheplotontheright.Thus,aclassicationschememustbedesignedtohandlethesescenariostodetectindependentlymovingobjects.Thenext 94


Residualopticowfordynamicenvironments 6 .AnapproximationforthepotentiallocationoftheFOEisfoundbyextendingthetranslationaloptical-owvectorstoformtheepipolarlines,asillustratedinFigure 6-3 ,andobtainingallpossiblepointsofintersection.Asmentionedpreviously,theintersectionpointsobtainedwillconstituteanumberofpotentialFOEs;however,onlyonewilldescribethestaticbackgroundwhiletherestareduetomovingobjects.Theapproachconsideredforthisclassicationthatessentiallygroupstheintersectiondatatogetherthroughadistancecriterionisaniterativeleast-squaressolutionforthepotentialFOEs.Theiterationproceduretestsallintersectionpointsasadditionalfeaturesareintroducedtothesystemofequationseachofwhichinvolves2unknownimageplanecoordinatesoftheFOEfoei;nfoei.Theprocessstartsbyconsidering2featurepointsandtheirFOEintersection 95


6 fortheFOEcoordinatesfoe1;nfoe1(fortherstiterationaleast-squaressolutionisnotnecessarybecausetwolinesintersectatasinglepoint). 2kM264n375bk2(6) where 6 fortheithiteration.Mathematically,theclassicationschemefortheithiterationisgiveninEquations 6 and 6 foeifoei12+nfoeinfoei12(6) 96






3 .Amongthetechniquesthatutilizefeaturepoints,theapproachrelatedtothispaperinvolvesepipolargeometry[ 39 112 ].Thepurposeofthistechniqueistoestimaterelativemotionbasedonasetofpixellocations.Thisrelativemotioncandescribeeithermotionofthecamerabetweentwoimagesortherelativedistanceoftwoobjectsofthesamesizefromasingleimage.The3DscenereconstructionofamovingtargetcanbedeterminedfromtheepipolargeometrythroughthehomographyapproachdescribedinChapter 3 .Forthecasedescribedinthischapter,amovingcameraattachedtoavehicleobservesaknownmovingreferenceobjectalongwithanunknownmovingtargetobject.Thegoalistoemployahomographyvision-basedapproachtoestimatetherelativeposeandtranslationbetweenthetwoobjects.Therefore,acombinationofvisionandtraditionalsensorssuchasaglobalpositioningsystem(GPS)andaninertialmeasurementunit(IMU)arerequiredtofacilitatethisproblemforasinglecameraconguration.ForexampleintheAARcase,GPSandIMUmeasurementsareavailableforboththereceiverandtankeraircrafts.Ingeneral,asinglemovingcameraaloneisunabletoreconstructthe3Dscenecontainingmovingobjects.Thisrestrictionisduetothelossoftheepipolarconstraint,wheretheplaneformedbythepositionvectorsrelativetotwocamerapositionsintimetoapointofinterestandthetranslationvectorisnolongervalid.Techniqueshavebeenformulatedtoreconstructmovingobjectsviewedbyamovingcamerawithvariousconstraints[ 35 113 116 ].Forinstance,ahomographybasedmethodthatsegmentsbackgroundfrommovingobjectsandreconstructsthetarget'smotionhasbeenachieved[ 117 ].Theirreconstructionisdonebycomputingavirtualcamerawhichxesthetarget'spositionintheimageanddecomposesthehomographysolutionintomotionofthecameraandmotioncausedbythetarget.Thisdecompositionisdoneusingaplanartranslationconstraintwhichrestrictsthetarget'smotiontoagroundplane.Similarly,Han 99

PAGE 100

115 ]proposedanalgorithmthatreconstructs3Dmotionofamovingobjectusingafactorization-basedalgorithmwiththeassumptionthattheobjectmoveslinearlywithconstantspeeds.Anonlinearlteringmethodwasusedtosolvetheprocessmodelwhichinvolvedboththekinematicsandtheimagesequencesofthetarget[ 118 ].Thistechniquerequiresknowledgeoftheheightabovethetargetwhichwasdonebyassumingthetargettraveledonthegroundplane.Thisassumptionallowedothersensors,suchasGPS,toprovidethisinformation.ThepreviousworkofMehtaetal.[ 77 ]showedthatamovingmonocularcamerasystemcouldestimatetheEuclideanhomographiesforamovingtargetinreferencetoaknownstationaryobject.ThecontributionofthischapteristocasttheformulationshowninMehtaelal.toamoregeneralproblemwherebothtargetandreferencevehicleshavegeneralmotionandarenotrestrictedtoplanartranslations.Thisproposedapproachincorporatesaknownreferencemotionintothehomographyestimationthroughatransformation.EstimatesoftherelativemotionbetweenthetargetandreferencevehiclearecomputedandrelatedbackthroughknowntransformationstotheUAV.RelatingthisinformationwithknownmeasurementsfromGPSandIMU,thereconstructionofthetarget'smotioncanbeachievedregardlessofitsdynamics;however,thetargetmustremainintheimageatalltimes.Althoughtheformulationcanbegeneralizedforncameraswithindependentposition,orientation,translations,androtationthischapterdescribesthederivationofasinglecamerasetup.Meanwhile,cuesonboththetargetandreferenceobjectsareachievedthroughLEDlightsormarkersplacedinaknowngeometricpatternofthesamesize.Thesemarkersfacilitatethefeaturedetectionandtrackingprocessbyplacingknownfeaturesthatstandoutfromthesurroundingswhilethegeometryandsizeofthepatternallowsforthecomputationoftheunknownscalefactorthatiscustomarytoepiploarandhomographybasedapproaches.ThischapterbuildsonthetheorydevelopedinChapters 3 and 5 whilerelyingonthemovingobjectdetectionalgorithmtoisolatemovingobjectswithinanimage.RecalltheowoftheoverallblockdiagramshowninFigure 1-6 .Theprocessstartedbycomputingfeaturesinthe 100

PAGE 101

6 .Oncemovingobjectsintheimagearedetected,thehomographyestimationalgorithmproposedinthischapterisimplementedfortargetstateestimation. 7.2.1SystemDescriptionThesystemdescribedinthispaperconsistsofthreeindependentlymovingvehiclesorobjectscontaining6-DOFmotion.TodescribethemotionofthesevehiclesaEuclideanspaceisdenedwithveorthonormalcoordinateframes.TherstframeisanEarth-xedinertialframe,denotedasE,whichrepresentstheglobalcoordinateframe.Theremainingfourcoordinateframesaremovingframesattachedtothevehicles.Therstvehiclecontainstwocoordinateframes,denotedasBandI,torepresentthevehicle'sbodyframeandcameraframe,asdescribedinChapter 5 inFigure 5-1 .Thisvehicleisreferredtoasthechasevehicleandisinstrumentedwithanon-boardcameraandGPS/IMUsensorsforpositionandorientation.Thesecondvehicle,denotedasF,isconsideredareferencevehiclethatalsocontainsGPS/IMUsensorsandprovidesitsstatestothechasevehiclethroughacommunicationlink.Lastly,thethirdvehicle,denotedasT,isthetargetvehicleofinterestinwhichunknownstateinformationistobeestimated.Inaddition,actitiouscoordinateframewillbeusedtofacilitatetheestimationprocessandisdenedasthevirtualcoordinatesystem,V.Thecoordinatesofthissystemarerelatedthroughtransformationscontainingbothrotationalandtranslationalcomponents.TherotationalcomponentisestablishedusingasequenceofEulerrotationsintermsoftheorientationanglestomaponeframeintoanother.LettherelativerotationmatricesREB,RBI,REF,REV,RIV,RFV,RTVandRETdenotetherotationfromEtoB,BtoI,EtoF,EtoV,ItoV,FtoV,TtoV,andEtoT.Secondly,thetranslationsaredenedasTEB,xF,xV,xT,xF;n,xT;n,TBI,xIV,mIF,mIT,hF;n,hT;n,mVF,mVT,hVF;n,andhVT;nwhichdenotetherespectivetranslationsfromEtoB,EtoF,EtoV,EtoT,EtothenthfeaturepointonthereferencevehicleandtargetvehiclesallexpressedinE,BtoIexpressedinB,ItoV,ItoF,ItoT,ItothenthfeaturepointonthereferenceandtargetvehiclesexpressedinI,VtoF,V

PAGE 102

7-1 foracameraonboardaUAVwhilethevectorsrelatingthefeaturepointstoboththerealandvirtualcamerasaredepictedinFigure 7-2 .TheestimatedquantitiescomputedfromthevisionalgorithmaredenedasRTBandxTBwhicharetherelativerotationandtranslationfromTtoBexpressedinB. Figure7-1. Systemvectordescription Thecameraismodeledthroughatransformationthatmaps3-dimensionalfeaturepointsontoa2-dimensionalimageplaneasdescribedinChapter 3 .Thistransformationisageometricrelationshipbetweenthecamerapropertiesandthepositionofafeaturepoint.Theimageplanecoordinatesarecomputedbasedonatangentrelationshipfromthecomponentsofhn.ThecamerarelationshipusedinthischapterisreferredtoasthecontinuouspinholecameramodelandisgiveninEquations 3 and 3 forazerolensoffset,wherefisthefocallengthofthecameraandhx;n,hy;n,hz;narethe(x;y;z)componentsofthenthfeaturepoint.Thispinholemodelisacontinuousmappingthatcanbefurtherextendedtocharacterizepropertiesofaphysicalcamera.Somecommonadditionstothismodelincludeskewness,radial 102

PAGE 103

BFigure7-2. MovingtargetvectordescriptionreltivetoA)cameraIandB)virtualcameraV 3 .Eachextensiontothemodeladdsanotherparametertoknowfortheestimationproblemandeachcanintroduceuncertaintyandlargeerrorsintheestimationresult.Therefore,thischapterwillonlyconsidertheeldofviewconstraintandleavethenonlineartermsandtheeffectsonestimationforfuturework.RecalltheeldofviewconstraintsgiveninChapter 3 .Theseconstraintscanberepresentedaslowerandupperboundsintheimageplaneandaredependentonthehalfangles(gh;gv)whichareuniquetoeachcamera.Mathematically,theseboundsareshowninEquations 7 and 7 forthehorizontalandverticaldirections. 103

PAGE 104

102 ].Thesameconstraintholdsfortheimagecoordinatesaswellbutalsointroducesanunknownscalefactor.Employingthisconstraint,estimatesofrelativemotioncanbeacquiredforbothcamera-in-handandxedcameracongurations.Thisdissertationdealswiththecamera-in-handcongurationwhileassumingaperfectfeaturepointdetectionandtrackingalgorithm.Thisassumptionenablestheperformanceofthevisionbasedstateestimationtobetestedbeforeintroducingmeasurementerrorsandnoise.Thehomographyconstraintrequiresafewassumptionsbasedonthequantityandthestructureofthefeaturepoints.Thealgorithmrstrequiresaminimumoffourplanarfeaturepointstodescribeeachvehicle.Thisrequirementenablesauniquesolutiontothehomographyequationbasedonthenumberofunknownquantities.ThereferencevehiclewillhaveaminimumoffourpixelvaluesineachimagewhichwillbedenedaspF;n=[F;n;nF;n]8nfeaturepoints.Likewise,thetargetvehiclewillhavefourpixelvaluesandwillbedenedaspT;n=[T;n;nT;n]8nfeaturepoints.Thisarrayoffeaturepointpositionsarecomputedat30Hzwhichistypicalforstandardcamerasandtheframecountisdenotedbyi.Thenalrequirementisaknowndistanceforboththereferenceandtargetvehicle.OnedistancerepresentsthepositionvectortoafeatureonthereferencevehicleinEuclideanspacerelativetothelocalframeFandtheseconddistancerepresentsthepositionvectortoafeatureonthetargetvehicleinEuclideanspacerelativetothelocalframeT.Inaddition,thelengthofthesevectorsalsomustbeequalwhichallowstheunknownscalefactortobedetermined.ThevectordescribingthereferencefeaturepointwillbedenotedassFexpressedinF,whilethevectordescribingthetargetfeaturepointisreferredtoassTexpressedinT.ThesefeaturepointpositionvectorsarealsoillustratedinFigure 7-2 .Thefeaturepointsarerstrepresentedbypositionvectorsrelativetothecameraframe,I.TheexpressionsforboththereferenceandtargetfeaturepointsaregiveninEquations 7 and 7 .ThesevectorcomponentsarethenusedtocomputetheimagecoordinatesgiveninEquations 7 and 7 .ThecomputationinEquation 7 requiresinformationregardingthetargetwhichisdonesolelytoproduceimagemeasurementsthatnormallywouldbeobtainedfromthesensor.Remainingcomputations,regardingthehomography,willonlyusesensor 104

PAGE 105

7 and 7 arethetruerotationsmatricesfromFtoBandTtoB,repectfully,andareshowninEquations 7 and 7 77 ].Inthiscase,boththereferenceandtargetvehiclesareinmotionandarebeingviewedbyamovingcamera.Therefore,thenextstepistotransformthecameratoavirtualcongurationthatobservesthereferencevehiclemotionlessintheimageovertwoframes.Inotherwords,thisapproachcomputesaEuclideantransformationthatmapsthecamera'sstatesati1toavirtualcamerathatmaintainstherelativepositionandorientationbetweenframestoxthefeaturepointsofthereferencevehicle.Thistransformationisdonebymakinguseofthepreviousimageframeandstateinformationati1fromboththecameraandthereferencevehicle.Afterthevirtualcameraisestablishedthehomographyequationscanbeemployedforstateestimation.TocomputethelocationandposeofthevirtualcameraatitherelativepositionandorientationfromItoFati1isrequired.ThisrelativemotioniscomputedthroughknownmeasurementsfromGPS/IMUandtheexpressionsareshowninEquations 7 and 7 fortranslationandrotationati1,respectfully. 105

PAGE 106

7 and 7 forthecurrentframei. 3 and 3 .TheexpressionsforthenewvectorshVF;nandhVT;nintermsofthevirtualcameraaregiveninEquations 7 and 7 forthereferenceandtargetvehicles. 7 and 7 areonewaytocomputeimagecoordinatesforthevirtualcamera,butthereareunknowntermsinEquation 7 thataren'tmeasurableorcomputeableinthiscase.Therefore,analternativemethodmustbeusedtocomputeimagevaluesofthetargetinthevirtualcamera.Usingthepositionandorientationofthevirtualcamera,asgiveninEquations 7 and 7 ,therelativemotioniscomputedfromcameraItocameraVwhileusingepipolargeometrytocomputethenewpixellocations.ThisrelativecameramotionisgiveninEquations 7 and 7 wherethetranslationisexpressedinI. 106

PAGE 107

7 7 whichreliesontherelativemotionremainingconstanttomaintainthereferencestationaryintheimage. 7 .Likewise,thetimevaryingpositionofafeaturepointonthetargetvehicleexpressedinVisgiveninEquation 7 7 and 7 andarerelativetothevirtualcameraframe. 107

PAGE 108

7 7 and 7 whichdescribetherelativemotionbetweenthereferenceandtargetobjects. 7 andthereferencevehiclelocationisknownthroughGPSalongwiththefeaturepointlocations;therefore,aprojecteddistancecanbecomputedthatscalesthedepthofthescene.Tocomputethisdistancethenormalvector,n,thatdenestheplanewhichthereferencefeaturepointslieisrequiredandcanbecomputedfromknowninformation.Ultimately,theprojectivedistancecanbeobtainedandisdenedinEquation 7 throughtheuseofthereferenceposition. 7 intoEquation 7 resultsinanintermediateexpressionfortheEuclideanhomographyandisshowninEquation 7 DnThVF;n(7)Tofacilitatethesubsequentdevelopment,thenormalizedEuclideancoordinatesareusedanddenedinEquations 7 and 7 108

PAGE 109

7 7 ,and 7 thenormalizedEuclideanhomographyisestablishedwhichrelatesthetranslationandrotationbetweencoordinateframesFandT.ThishomographyexpressionisshowninEquation 7 intermsofthenormalizedEuclideancoordinates. {z }hVF;naH(7)InEquation 7 7 D(7)TheEuclideanhomographycannowbeexpressedintermsofimagecoordinatesorpixelvaluesthroughtheidealpin-holecameramodelgiveninEquations 3 and 3 .Thisexpressingisdonebyrstrewritingthecameramodelintomatrixformwhichisreferredtoasthecameracalibrationmatrix,K.SubstitutingthecameramappingintoEquation 7 andusingthecameracalibrationmatrix,K,thehomographyintermsofpixelcoordinatesisobtainedandgiveninEquation 7 .ThisnalexpressionrelatestherotationandtranslationofthetwovehiclesFandTintermsoftheirimagescoordinates.Therefore,toobtainasolutionfromthishomographyexpressionbothvehiclesneedtobeviewableintheimageframe. {z }pVF;nG(7)ThematrixG(t)isdenotedasaprojectivehomographyinEquation 7 whichareasetofequationsthatcanbesolveduptoascalefactorusingalinearleastsquaresapproach.OncethecomponentsofhomographymatrixareestimatedthematrixneedstobedecomposedintotranslationalandrotationalcomponentstoobtainxhandR.Thisdecompositionisaccomplishedusingtechniquessuchassingularvaluedecompositionandgeneratesfourpossiblesolutions[ 119 120 ].Todetermineauniquesolutionsomephysicalcharacteristicsoftheproblem 109

PAGE 110

7 7 toobtainx.Secondly,xisthendividedbyatoscalethedepthratioresultinginthenalxexpressedinI.ThisresultinconjunctionwithRisthenusedinEquation 7 tosolveformVT.ThenextstepistocomputetherelativetranslationfromItoVwhichisgiveninEquation 7 7 7 and 7 representtherelativemotionbetweenthecameravehicleandthetargetvehicle.Thisinformationisvaluableforthecontroltasksdescribedearlierinvolvingbothtrackingandhomingapplications.Thenextsectionwillimplementthisalgorithminsimulationtoverifythestateestimatorforthenoisefreecase. 110

PAGE 111


PAGE 112

121 ]thatemploysaHiddenMarkovModeltopredictthemotionofmovingobjects.ThebenetsinusingaHiddenMarkovModelincludeatimedependenceframeworkincorporatedintotheprobabilisticmodelaswellastheabilitytohandlestochasticprocesses.TheunderliningconceptofaHiddenMarkovModeldescribestheprobabilityofaprocesstosequentiallygofromonestatetoanother.Thissequentialpropertyprovidesthenecessaryframeworkfortimedependencemodeling,whichisanattractiveapproachfortheapplicationsconsidered,wherethetimehistorydataisacriticalpieceofinformationincludedinthemodeling. 8 .Therefore,thevelocityandpositionareupdatedthroughEquations 8 and 8 .Althoughthismodelislimited,itdescribesafoundationformodelingtargetmotionandcoversthebasicmodelconstantvelocity. 112

PAGE 113

8 andischaracterizedbyarandomvector,w(t)andisscaledbyaconstant,r.ThevelocitycorrespondingtothisaccelerationisdescribedinEquation 8 .Thismodelattemptstocapturethestochasticbehaviorsbyutilizingaprobabilisticdistributionfunction. 8 canbemodiedtoincorporatesomedependenceonthepreviousaccelerationvalue.ThisdependenceisachievedbyweightingthepreviousaccelerationinthemodelandisshowninEquation 8 .ThebenettothistypeofmodelasopposetoEquation 8 requiressomeknowledgeofthetarget;namely,thatthetargetcannotachievelargeabruptchanginginacceleration.TheresultingvelocityexpressionforthismodelisgiveninEquation 8 8 fortheithtargetandNimageframes.ThevelocityproleiscomputedusingabackwardsdifferencemethodandisgiveninEquation 8 8 ,isobtainedfromthevelocityprolegiveninEquation 8 .Thesamebackwardsdifferencemethodisusedtocomputethis 113

PAGE 114

8 .Thisaccelerationtimehistoryiscomputedimplicitlythroughthepositionestimatesobtainedfromthehomographyalgorithm 8 and 8 providetheinitialmotionstatedescriptionthatpropagatestheMarkovtransitionprobabilityfunction.TheformoftheMarkovtransitionprobabilityfunctionisassumedtobeaGaussiandensityfunctionthatonlyrequirestwoparametersforitsrepresentation.TheparametersneededforthisfunctionincludethemeanandvariancevectorsfortheaccelerationprolegiveninEquation 8 .Note,duringthischapter(x)isthemeanoperatorandnottheverticalcomponentintheimageplane.Likewise,s2(x)isreferredtoasthevarianceoperator. 8 ,wheretheargumentsconsistofthemeanandvariancepertainingtotheestimatedacceleration. 8 and 8 forthetransitionfunction.Thefunctionsfandfsarechosenbasedonthedesiredweightingofthetimehistoryandcansimplybeaweightedlinearcombinationofthearguments.Theseinitialstatisticalparametersareusedinthepredictionstepandupdatedonceanewmeasurementisobtained.

PAGE 115

8 asthree-dimensionalGaussiandensityfunctionandisuniquelydeterminedbythemeanandvariance. 2(ai(t)(ai(t)))2 8 8 and 8 ,thepredictiveprobabilityforobjectiattimet+kisgivenasEquation 8 .Thisframeworkenablestheexibilityofcomputingthepredictedestimatesatanydesiredtimeinthefuturewiththenotionthatfurtheroutintimetheprobabilitydiminishes. 8 and 8 fortheentiretimeinterval. 8 and 8

PAGE 116

8 and 8 2ai(t1) 2(ai(t1)) 2ai(t1) 2s2(ai(t1))Lastly,theprobabilityfunctionsforvelocityandpositionareusedtocomputethepredictiveprobabilitiesforobjectithataregiveninEquations 8 and 8 forvelocityandposition,respectfully. 8 istheprobabilitythattargetiislocatedinpositionp(x;y;z).Sothetheoverallprocessisaniterativemethodthatusesthemotionmodels,giveninSection 8.2.1 ,toprovideguessesforpositionandvelocityinattempttomaximizetheprobabilityfunctionsgiveninEquations 8 and 8 .ThepositionthatmaximizesEquation 8 isthemostlikelylocationofthetargetatt+kwithaknownprobability. 116

PAGE 117

7 .Effectively,thesequantitiesaretheerrorsignalsusedforcontroltotrackthemovingcameratowardadesiredlocationbasedonthemotionofthetarget.TheframeworkpresentedherewilluseaircraftandUAVnavigationschemesfortheaerialmissionsdescribedinChapter 1 .Therefore,thecontroldesigndescribedinthischapterfocusesonthehomingmissiontofacilitatetheAARproblem,whichinvolvestrackingthepositionstatescomputedfromthehomography.Varioustypesofguidancecontrollerscanbeimplementedforthesetypesoftaskoncetherelativepositionandorientationareknown.Dependingonthecontrolobjectivesandhowfastthedynamicsofthemovingtargetare,lowpasslteringoralowgaincontrollermayberequiredtoavoidhighratecommandstotheaircraft.IntheAARproblem,thesuccessofthedockingcontrollerwilldirectlyrelyonseveralcomponents.TherstcomponentistheaccuracyofestimatedtargetlocationwhichduringAARneedstoprecise.Secondly,thedynamicsofthedroguearestochastic.Thiscausesthemodelingtasktobeimpracticalinreplicatingreallifesothecontrollerislimitedtothemodelsconsideredinthedesign.Inaddition,thedrogue'sdynamicsmaynotbedynamicallyfeasiblefortheaircrafttotrackwhichmayfurtherreduceperformance.Lastly,thecontrollerideallyshouldmakepositionmaneuversinstagesbyconsideringthealtitudeasonestage,thelateralpositionasanotherstage,andthedepthpositionasthenalstage.Incloseproximity,thecontrollershouldimplementonlysmallmaneuverstohelpmaintainthevehiclesintheFOV. 15 ]. 117

PAGE 118

110 ].Thestandarddesignapproachwasusedbyconsideringthelongitudinalandlateralstatesseparatelyasintypicalwaypointcontrolschemes.Thisapproachseparatedthecontrolintothreesegments:1)Altitudecontrol,2)HeadingControland3)DepthControl. 9-1 .Therstportionofthissystemisdescribedastheinner-loopwherepitchandpitchrateareusedinfeedbacktostabilizeandtrackapitchcommand.Meanwhile,thesecondportionisreferredtoastheouter-loopwhichgeneratespitchcommandsfortheinner-loopbasedonthecurrentaltitudeerror.Theinner-loopdesignenablesthetrackingofapitchcommandthroughproportional 6 Altitudeholdblockdiagram control.Thispitchcommandinturnwillaffectaltitudethroughthechangesinforcesonthehorizontaltailfromtheelevatorposition.Thetwosignalsusedforthisinner-looparepitchandpitchrate.Thepitchratefeedbackhelpswithshortperioddampingandallowsforratevariationsinthetransientresponse.AleadcompensatorwasdesignedinStevensetal.[ 110 ]toraisethe 118

PAGE 119

9-1 .Thisstructurewillprovidegooddisturbancerejectionduringturbulentconditions.Inaddition,boundswereplacedonthepitchcommandtoalleviateanyaggressivemaneuversduringtherefuelingprocess. 9-2 .Theinner-loop -+fcmd 6y ?Dy 6f Headingholdblockdiagram componentofFigure 9-2 dealswithrolltracking.Thefeedbacksignalsincludebothrollandrollratethroughproportionalcontroltocommandachangeinaileronposition.Theinner-loop 119

PAGE 120

110 ].Consequently,theturnsmootherandcontainslessoscillations.Trackingheadingisnotsufcienttotrackthelateralpositionwiththelevelofaccuracyneededforrefuelingtask.Thenalloopwasaddedtoaccountforanylateraldeviationaccumulatedovertimeduetothedelayinheadingfromposition.Thisdelayismainlyduetothetimedelayassociatedwithsendingarollcommandandproducingaheadingchange.Therefore,thisloopwasaddedtogeneratemorerollforcompensation.Theloopcommandedachangeinaileronbasedoftheerrorinlateralposition.Thisdeviation,referredtoasDy,wascomputedbasedontwosuccessivetargetlocationsprovidedbytheestimator.Thecurrentandprevious(x;y)positionsofthetargetwereusedtocomputealineinspacetoprovideareferenceoftheit'smotion.Theperpendiculardistancefromthevehicle'spositiontothislinewasconsideredthemagnitudeofthelateralcommand.Inaddition,thesignofthecommandwasneededtoassignthecorrectdirection.Thisdirectionwasdeterminedfromtherelativeyposition,expressedinthebody-xedframe,thatwasfoundduringestimation.Oncethelateraldeviationwasdetermined,thatsignalwaspassedthroughaPIstructure,asshowninFigure 9-2 .Thegainscorrespondingtotheproportional,kyp,andintegrator,kyi,werethensummedandaddedtocomputethenalrollcommand.ThecompleteexpressionfortherollcommandisshowninEquation 9 120

PAGE 121


PAGE 122

8 providesamethodtoestimatetargetsinEuclideanspacewhenfeaturesdoexittheimage.Thismethodworkswellforshortperiodsoftimeafterthetargethasleft;however,thetrustinthepredictedvaluedegradestremendouslyastimeincreases.Consequently,whenafeatureleavestheimagethecontrollercanrelyonthepredictedestimatestosteertheaircraftinitiallybutmayresorttoalternativeapproachesbeyondaspeciedtime.Asalastresort,thecontrollercancommandtheaircrafttoslowdownandregainabroaderperspectiveofthescenetorecapturethetarget. 122

PAGE 123

110 ].Abaselinecontrollerisimplementedthatallowsthevehicletofollowwaypointsbasedentirelyonfeedbackfrominertialsensors.Imagesareobtainedfromasetofcamerasmountedontheaircraft.Thesecamerasincludeastationarycameramountedatthenoseandpointingalongthenose,atranslatingcameraunderthecenterlinethatmovesfromtherightwingtotheleftwing,andapitchingcameramountedunderthecenterofgravity.TheparametersforthesecamerasaregiveninTable 10-1 invaluesrelativetotheaircraftframeandfunctionsoftimegivenastinseconds. Table10-1. Statesofthecameras position(ft) orientation(deg) camera 24 0 0 0 90 0 2 -10 15-3t 0 0 45 0 3 0 0 3 0 45-9t 0 Thecameraparametersarechosenassimilartoanexistingcamerathathasbeenighttested[ 111 ].Thefocallengthisnormalizedsof=1.Also,theeldofviewforthismodelcorrelatestoanglesofgh=32degandgv=28deg.TheresultinglimitsonimagecoordinatesaregiveninTable 10-2 Table10-2. Limitsonimagecoordinates coordinate minimum maximum 0.62 0.53 Avirtualenvironmentisestablishedwithsomecharacteristicssimilartoanurbanenvironment.Thisenvironmentincludesseveralbuildingsalongwithamovingcaranda 123

PAGE 124

10-3 ,isassociatedwitheachfeaturefordirectidenticationinthecameraimages. Table10-3. Statesofthefeaturepoints position(ft) featurepoint north east altitude 1 3500 200 -1500 2 1000+200t 500 -500 3 6000 200cos(2p TheightpaththroughthisenvironmentisshowninFigure 10-1 alongwiththefeatures.TheaircraftinitiallyiesstraightandleveltowardtheNorthbutthenturnssomewhattowardstheEastandbeginstodescendfromadivemaneuver. BFigure10-1. VirtualEnvironmentforExample1:A)3DViewandB)TopView ImagesaretakenatseveralpointsthroughouttheightasindicatedinFigure 10-1 bymarkersalongthetrajectory.ThestatesoftheaircraftattheseinstancesaregiveninTable 10-4 .Theimageplanecoordinates(;n)areplottedinFigure 10-2 forthethreecamerasatt=2sec.ThiscomputationisaccomplishedbyusingEquation 5 inconjunctionwithEquations 3 and 3 whileapplyingtheeldofviewconstraintshowninEquations 3 and 3 .Allthreecamerascontainsomeportionoftheenvironmentalongwithdistinctviewsofthefeaturepointsofinterest.Forexample,camera1containsaforwardlookingviewofastationary 124

PAGE 125

Aircraftstates Time North East Down v w (ft) (ft) (ft) (ft=s) (ft=s) 1196.9 0.44 -2174.8 573.52 56.79 -126.46 4 2112.7 143.04 -1645.4 527.37 -54.94 17.77 6 2989.8 353.63 -1100.7 528.30 4.26 45.57 Time q y q r (deg) (deg) (deg) (deg=s) (deg=s) -13.92 -22.43 -1.81 -13.56 -36.82 1.38 4 -39.21 -37.90 22.79 32.31 28.41 0.04 6 6.98 -14.85 11.93 7.63 -9.34 -1.15 pointonthecornerofabuildingaswellasthemovinghelicopter.Meanwhile,cameras2and3observeatopviewofthemovinggroundvehicletravelingforwarddownaroad.Theseimagemeasurementsprovideasignicantamountofdataandallowformoreadvancedalgorithmsforstateestimationandreconstruction. B CFigure10-2. FeaturepointMmasurementsatt=2secforA)camera1,B)camera2,andC)camera3 Figure 10-3 depictstheopticowcomputedforthesamedatasetshowninFigure 10-2 .Thisimagemeasurementgivesasenseofrelativemotioninmagnitudeanddirectioncausedbycameraandfeaturepointmotion.TheexpressionsrequiredtocomputeopticowconsistedofEqs. 5 5 3 3 3 3 3 and 3 .Inthisexample,theopticowhasmanycomponentscontributingtothenalvalue.Forinstance,theaircraft'svelocityandangularratescontributealargeportionoftheopticowbecauseoftheirlargemagnitudes.Inaddition,thesmallercomponentsinthisexamplearecausedfromvehicleandcameramotionwhicharesmallerinmagnitudebuthaveasignicanteffectondirection.Comparingcameras1and2,there 125

PAGE 126

B CFigure10-3. OpticowMeasurementsatt=2secforA)camera1,B)camera2,andC)camera3 Asummaryoftheresultingimageplanequantities,positionandvelocity,isgiveninTable 10-5 forthefeaturepointsofinterestaslistedinTable 10-3 .Thetableisorganizedbythetimeatwhichtheimagewastaken,whichcameratooktheimage,andwhichfeaturepointisobserved.Thistypeofdataenablesautonomousvehiclestogainawarenessoftheirsurroundingsformoreadvancedapplicationsinvolvingguidance,navigationandcontrol. Table10-5. Imagecoordinatesoffeaturepoints Time(s) Camera FeaturePoint 1 1 0.157 0.162 0.610 0.044 2 1 3 0.051 0.267 0.563 -0.012 2 2 2 -0.308 0.075 0.464 -0.254 2 3 2 0.011 0.077 0.583 -0.235 4 2 2 -0.279 -0.243 -0.823 0.479 4 3 2 0.365 -0.248 -0.701 0.603 6 1 3 0.265 -0.084 0.267 -0.015 10.2.1ScenarioFeaturepointuncertaintyisdemonstratedinthissectionbyextendingthepreviousexample.Thissimulationwillexaminetheuncertaintyeffectsonvisionprocessingalgorithmsusingsimulatedfeaturepointsandperturbedcameraintrinsicparameters. 126

PAGE 127

10-4 alongwithapairofpointsindicatingthelocationsatwhichimageswillbecaptured.Theaircraftisinitiallystraightandlevelthentranslatesforwardwhilerolling4.0degandyawing1.5degatthenallocation. BFigure10-4. Virtualenvironmentofobstacles(solidcircles)andimaginglocations(opencircles)A)3DviewandB)topview Asinglecameraissimulatedatthecenterofgravityoftheaircraftwithlineofsightalignedtothenoseoftheaircraft.Theintrinsicparametersarechosensuchthatfo=1:0anddo=0:0forthenominalvalues.TheimagesforthenominalcameraassociatedwiththescenarioinFigure 10-4 arepresentedinFigure 10-5 toshowthevariationbetweenframes.Thevision-basedfeedbackiscomputedforasetofperturbedcameras.Theseperturbationsrangeasdf2[0:2;0:2]anddd2[0:02;0:02].ObviouslythefeaturepointsinFigure 10-5 willvaryasthecameraparametersareperturbed.Theamountofvariationwilldependonthefeature 127

PAGE 128

BFigure10-5. FeaturepointsforA)initialandB)nalimages point,asnotedinEquations 4 and 4 ,buttheeffectcanbenormalized.Thevariationinfeaturepointgivennominalvaluesofo=no=1isshowninFigure 10-6 forvariationinbothfocallengthandradialdistortion.Thissurfacecanbescaledaccordinglytoconsiderthevariationatotherfeaturepoints.TheperturbedsurfaceshowninFigure 10-6 ispropagatedthroughthreemainimageprocessingtechniquesforanalysis. Figure10-6. Uncertaintyinfeaturepoint 10-6 tofocallengthandradialdistortion.ArepresentativecomparisonofopticowforthenominalcameraandasetofperturbedcamerasisshowninFigure 10-7 128

PAGE 129

B CFigure10-7. Opticalowfornominal(black)andperturbed(red)camerasforA)f=1:1andd=0,B)f=1:0andd=0:01,andC)f=1:1andd=0:01 10-7 indicateseveraleffectsofcameraperturbationsnotedinEquations 4 and 4 .Theperturbationstofocallengthscalethefeaturepointssothemagnitudeofopticowisuniformlyscaled.Theperturbationstoradialdistortionhavelargereffectasthefeaturepointmovesawayfromthecenteroftheimagesotheopticowvectorsarealteredindirection.Thecombinationofperturbationsclearlychangestheopticowinbothmagnitudeanddirectionanddemonstratesthefeedbackvariationsthatcanresultfromcameravariations.Theopticowiscomputedforimagescapturedbyeachoftheperturbedcameras.ThechangeinopticowfortheperturbedcamerasascomparedtothenominalcameraisrepresentedasdJandisboundedinmagnitude,asderivedinEquation 4 ,byDJ.ThegreatestvalueofdJpresentedbythesecameraperturbationsiscomparedtotheupperboundinTable 10-6 .ThesenumbersindicatethevariationsinopticowareindeedboundedbythetheoreticalboundderivedinChapter 4 andindicatethelevelofowvariationsinducedfromthevariationsincameraparameters. Table10-6. Effectsofcameraperturbationsonopticow PerturbationSet Analyzeonlywithdf kDJk kdJk kDJk kdJk kDJk 0.0476 0.0040 0.0040 0.0496 0.0543 0.0476 0.0020 0.0040 0.0252 0.0543 0.0476 0.0020 0.0040 0.0264 0.0543 0.0476 0.0040 0.0040 0.0543 0.0543 129

PAGE 130

10-8 showsthequalityoftheestimation.Essentially,theepipolargeometryrequiresafeaturepointinoneimagetoliealongtheepipolarline.Thisepipolarlineisconstructedbytheintersectionbetweentheplaneformedbytheepipolarconstraintandtheimageplaneatthelastmeasurement.ThedatainFigure 10-8 showthefeaturesinthesecondimagedoindeedlieexactlyontheepipolarlines. BFigure10-8. Epipolarlinesbetweentwoimageframes:A)initialframeandB)nalframewithoverlayedepipolarlinesfornominalcamera Theintroductionofuncertaintyintotheepipolarconstraintwillcausevariationsintheessentialmatrixwhichwillalsopropagatethroughthecomputationoftheepipolarline.Thesevariationsintheepipolarlinearevisualcluesofthequalityoftheestimateintheessentialmatrix.Thesevariationscanoccuraschangesintheslopeandthelocationoftheepipolarline.Figure 10-9 illustratestheepipolarvariationsduetoperturbationsondf=0:1anddd=0:01tothecameraparameters.Thefeaturepointswithuncertaintyandthecorrespondingepipolarlinewasplottedalongwiththenominalcasetoillustratethevariations.Thekeypointinthisguresisthesmallvariationsintheslopeoftheepipolarlinesandthesignicantvariationsinfeature 130

PAGE 131

BFigure10-9. Uncertaintyresultsforepipolargeometry:A)initialframeandB)nalframewithoverlayedepipolarlinesforcameraswithf1:0andd=0:0(black)andf=1:1andd=0:01(red) Theessentialmatrixiscomputedfortheimagestakenusingasetofcameramodels.EachmodelisperturbedfromthenominalconditionusingthevariationsinFigure 10-6 .Thechangeinestimatedstatesbetweennominalandperturbedcamerasisgivenbydqovertheuncertaintyrangeandisbounded,asderivedinEquation 4 ,byDq.ThevalueofdqforaspecicperturbationisshownincomparisontotheupperboundinTable 10-7 whichalsoindicatethevariationinenteriesoftheessentailmatrixwhichpropagatetothecamerastates. Table10-7. Effectsofcameraperturbationsonepipolargeometry PerturbationSet Analyzeonlywithdf kDqk kdqk kDqk kdqk kDqk 293.14 4.45 4.45 288.75 297.34 293.14 2.19 2.19 288.75 297.34 293.14 2.11 2.19 288.75 297.34 293.14 4.15 4.45 288.75 297.34 131

PAGE 132

10-10 toindicatethatallerrorswerelessthan106. Figure10-10. Nominalestimationusingstructurefrommotion Thedepthsarealsoestimatedusingstructurefrommotiontoanalyzeimagesfromtheperturbedcameras.ArepresentativesetoftheseestimatesareshowninFigure 10-11 ashavingclearerrors.Aninterestingfeatureoftheresultsisthedependenceonsignoftheperturbationtofocallength.Essentially,thesolutiontendstoestimateadepthlargerthanactualwhenusingapositiveperturbationandadepthsmallerthanactualwhenusinganegativeperturbation.Sucharelationshipisadirectresultofthescalingeffectthatfocallengthhasonthefeaturepoints.Estimatesarecomputedforeachoftheperturbedcamerasandcomparedtothenominalestimate.Theworst-caseerrorsinestimationarecomparedtothetheoreticalbound,giveninEquation 4 ,totheseerrors.ThesenumbersshowninTable 10-8 indicatethevariationinstructurefrommotiondependsonthesignoftheperturbation.Theapproachisactuallyseentobelesssensitivetopositiveperturbations,whichcausesalargerestimateindepth,thantonegativeperturbations.Also,thetheoreticalboundwasgreaterthan,orequalto,theerrorcausedbyeachcameraperturbation. 132

PAGE 133

B CFigure10-11. Estimationusingstructurefrommotionfornominal(black)andperturbed(red)cameraswithA)f=1:1andd=0,B)f=1:0andd=0:01,andC)f=1:1andd=0:01 Effectsofcameraperturbationsonstructurefrommotion PerturbationSet Analyzeonlywithdf kDzk kdzk kDzk kdzk kDzk 4679.8 75.02 75.02 4903.5 4903.5 4679.8 36.90 75.02 1076.6 4903.5 4679.8 35.73 75.02 498.76 4903.5 4679.8 70.34 75.02 1092.5 4903.5 1 involvingapolicepursuitisdemonstratedthroughthissimulation.Thesetupconsistedofthreevehicles:anUAVyingabovewithamountedcamera,electronicsandcommunication,areferencegroundvehiclewhichisconsideredthepolicepursuitcar,andatargetvehicledescribingthesuspectsvehicle.ThegoalofthismissionisfortheUAVtotrackbothvehiclesintheimage,whilereceivingpositionupdatesfromthereferencevehicle,andestimatethetarget'slocationusingtheproposedestimationalgorithm.ThecamerasetupconsideredinthisproblemconsistofasingledownwardpointingcameraattachedtotheUAVwithxedpositionandorientation.Whileinightthecamerameasuresandtracksfeaturepointsonboththetargetvehicleandthereferencevehicleforuseintheestimationalgorithm.Thissimulationassumesperfectcameracalibration,featurepointextraction,and 133

PAGE 134

7 thegeometryofthefeaturepointsarepredescribedandaknowndistanceisprovidedforeachvehicle.AfurtherdescriptionofthisassumptionisgiveninSection 7.2.2 .Futureworkwillexaminemorerealisticaspectsofthecamerasystemtoreproduceamorepracticalscenarioandtrytoalleviatethelimitationsimposedonthefeaturepoints. 10-12 ,forillustration.Theinitialframeforthissimulationislocatedattheaircraft'spositionwhenthesimulationstarts.Thevelocityofthegroundvehicleswerescaleduptotheaircraft'svelocitywhichresultedinlargedistancesbutalsohelpedtomaintainthevehiclesintheimage. BFigure10-12. Vehicletrajectoriesforexample3:A)3DviewandB)topview ThepositionandorientationstatesofthethreevehiclesareplottedinFigures 10-13 10-18 andallarerepresentedintheinertialframe,E.Thepositionsindicatethatallthreevehicle 134

PAGE 135

B CFigure10-13. PositionstatesoftheUAVwithon-boardcamera:A)North,B)East,andC)Down B CFigure10-14. AttitudestatesoftheUAVwithon-boardcamera:A)Roll,B)Pitch,andC)Yaw B CFigure10-15. Positionstatesofthereferencevehicle(pursuitvehicle):A)North,B)East,andC)Down 135

PAGE 136

B CFigure10-16. Attitudestatesofthereferencevehicle(pursuitvehicle):A)Roll,B)Pitch,andC)Yaw B CFigure10-17. Positionstatesofthetargetvehicle(chasevehicle):A)North,B)East,andC)Down B CFigure10-18. Attitudestatesofthetargetvehicle(chasevehicle):A)Roll,B)Pitch,andC)Yaw motionfromtheUAVtothetargetofinterest.ThenormerrorofthismotionaredepictedinFigure 10-19 .Theseresultsindicatethatwithsyntheticimagesandperfecttrackingofthevehiclesnearlyperfectmotioncanbeextracted.Oncenoiseintheimageortrackingisintroducedtheestimatesofthetargetdeterioratequicklyevenwithminutenoise.Inaddition,imageartifactssuchasinterferenceanddropoutswillalsohaveanadverseaffectonhomographyestimation. 136

PAGE 137

BFigure10-19. NormerrorforA)relativetranslationandB)relativerotation Figures 10-20 and 10-21 showtherelativetranslationandrotationdecomposedintotheirrespectivecomponentsandexpressedinthebodyframe,B.Thesecomponentsrevealtherelativeinformationneededforfeedbacktotrackorhomeinonthetargetofinterest. B CFigure10-20. Relativepositionstates:A)X,B)Y,andC)Z B CFigure10-21. Relativeattitudestates:A)Roll,B)Pitch,andC)Yaw 137

PAGE 138

10-22 ofthecameraviewdepictingthevehiclesandthesurroundingscene.Theredvehiclewasdesignatedasthereferencewhereasthegreyvehiclewasthetargetvehicle.Thenextstepinthisprocessistoimplementanactualfeaturetrackingalgorithmonthesyntheticimagesthatfollowsthevehicles.Thismodicationalonewilldegradethehomographyresultsimmenselyduetothetroublesomecharacteristicsofafeaturepointtracker. Figure10-22. Virtualenvironment 1 describedthemotivationandthebenetsofAAR,thissectionwilldemonstrateitbycombiningthecontroldesigngiveninChapter 9 withthehomographyresultinChapter 7 toformaclosed-loopvisualservocontrolsystem.ThevehiclesinvolvedinthissimulationincludesaReceiverUAVinstrumentedwithasinglecamera,atankeraircraftalso 138

PAGE 139

5 withadditionalstatessuchasV,a,b,theaccelerationterms,Machnumber,anddynamicpressure.Althoughthecontrollerwillnotuseallstates,theassumptionoffullstatefeedbackwasmadetoallowallstatesaccessiblebythecontroller.Thecontrollerusesthesestatesoftheaircraftalongwiththeestimatedresultstocomputeactuatorcommandsaroundthespeciedtrimcondition. 139

PAGE 140

9 isintegratedandtunedforthenonlinearF-16modeltoaccomplishthissimulation.Itwasassumedthatfullstatefeedbackoftheaircraftstatesweremeasurableincludingposition.Theunitsusedinthissimulationaregiveninftanddegwhichmeansthegainsdeterminedinthecontrolloopswerealsofoundbasedontheseunits.First,thepitchtrackingforaltitudecontrollerisconsidered.Theinner-loopgainsforthiscontrolleraregivenaskq=3andkq=2:5.ThebodediagramforpitchcommandtopitchangleisdepictedinFigure 10-23 forthespeciedgains.Thisdiagramrevealsthedamping 140

PAGE 141

Figure10-23. Inner-looppitchtopitchcommandBodeplot ThestepresponseforthepitchcontrollerisgiveninFigure 10-24 andshowsacceptableperformance.Theouter-loopcontrolwillnowbedesignedusingthiscontrollertotrackaltitude. Figure10-24. Pitchanglestepresponse 141

PAGE 142

10 andwasdesignedinStevensetal.[ 110 ].AstepresponseforthiscontrollerisillustratedinFigure 10-25 thatshowsasteadyclimbwithnoovershootandasteady-stateerrorof2ft.ThisresponseisrealisticforanF-16butnotidealforautonomousrefuelingmissionwheretolerancesareonthecmlevel.Thealtitudetransitionisslowduetothecompensatorbutonemayconsidermoreaggressivemaneuversformissionssuchastargettrackingthatmayrequireadditionalagility. Figure10-25. Altitudestepresponse Thenextstagethatwastunedinthecontroldesignwastheheadingcontroller.Theinner-loopgainswerechosentobekf=5:7andkp=1:6fortherolltracker.ThebodediagramforthiscontrollerofrollcommandtorollangleisshowninFigure 10-26 whichshowsattenuationinthelowerfrequencyrange.Thisattenuationremovesanyhighfrequencyresponsefromtheaircraftwhichisdesiredduringarefuelingmission,especiallyincloseproximity.Meanwhile,thecouplingbetweenlateralandlongitudinalstatesduringaturnwascounteracted 142

PAGE 143

Figure10-26. Inner-looprolltorollcommandBodeplot ThestepresponseforthisbankcontrollerisillustratedinFigure 10-27 .Thetrackingperformanceisacceptablebasedonarisetimeof0:25sec,anovershootof6%andlessthana3%steady-stateerror.Theouter-looptuningforheadingcontrollerconsistedofrsttuningthegainonheadingerror.Againofky=1:5waschosenforthismissionwhichdemonstratedacceptableperformance.Figure 10-28 showstheheadingresponseusingthiscontrollerforarightturn.Theresponserevealasteadyrisetime,noovershoot,andasteady-stateerroroflessthan2deg.Finally,thelooppertainingtolateraldeviationwastunedtokyp=0:5andkyi=0:025whichproducedreasonabletrackingandsteadyerrorforlateralposition.Thenalstageofthecontrollerinvolvestheaxialposition.Thisstagewasdesignedtoincreasethrustbasedonavelocitycommandoncethelateralandaltitudestateswerealigned.Aproportionalgainwastunedbasedonvelocityerrortoachieveaslowsteadyapproachspeed 143

PAGE 144

Rollanglestepresponse Figure10-28. Headingresponse tothetarget.Againofkx=3:5wasdeterminedforthisloopwhichgeneratesthedesiredapproach.Lastly,tohelplimitthenumberoftimesthefeaturepointsexittheeldofviewalimitwasimposedonthepitchangle.Thislimitwasenforcedwhentheapproachachieveaspecieddistance.Forthisexample,thedistancewassettowithin75ftintheaxialpositionofthebody-xedframewhichwasdeterminedexperimentallyfromthetarget'ssize. 144

PAGE 145

10-29 forpositionandFigure 10-30 fororientationandrevealedacorrectt.Thisresultdemonstratesthefunctionalityoftheestimatorwithanaccuracyontheorderof109.ThiserrorwasplottedinFigure 10-31 forbothpositionandorientation. B CFigure10-29. Open-loopestimationoftarget'sinertialposition:A)North,B)East,andC)Altitude B CFigure10-30. Open-loopestimationoftarget'sinertialattitude:A)Roll,B)Pitch,andC)Yae Furthermore,theclosed-loopresultsforthissimulationwereplottedinFigures 10-32 and 10-34 forpositionandorientationofboththereceiveraircraftandthetargetdroguerelativetotheearth-xedframe.Thetrackingofthiscontrollershowedreasonableperformanceforthedesiredpositionandheadingsignals.Theremainingorientationangleswerenotconsideredinfeedbackbutestimatedforthepurposeofmakingsurethedrogue'spitchandrollarewithinthedesiredvaluesbeforedocking.AsseeninFigure 10-32 ,thereceiverwasabletotrackthegrossmotionofdroguewhilehavingsomedifcultlytrackingtheprecisemotion. 145

PAGE 146

BFigure10-31. NormerrorfortargetstateestimatesA)translationandB)rotation B CFigure10-32. Closed-looptargetpositiontracking:A)North,B)East,andC)Altitude ThecomponentsofthepositionerrorbetweenthereceiveranddrogueareshowninFigure 10-33 toillustratetheperformanceofthetrackingcontroller.Theseplotsdepicttheinitialoffseterrordecayingovertimewhichindicatesthereceiver'srelativesdistanceisdecreasing.Thealtitudeshowedaquickclimbresponsewhereastheresponseinaxialpositionwasaslowsteadyapproachwhichwasdesiredtolimitlargechangesinaltitudeandangleofattack.Thelateralpositionisstableforthetimeperiodbutcontainsoscillationsduetherolltoheadinglag.TheorientationanglesshowninFigure 10-34 indicatetheEuleranglesforforthebody-xedtransformationscorrespondingtothebody-xedframeofthereceiverandthebody-xedframeofthedrogue.Recall,theonlysignalbeingtrackedinthecontroldesignwasheading.Thisselectionallowedtheaircrafttosteerandmaintainaighttrajectorysimilartothedroguewithoutaligningrollandpitch.Thereceivershouldyclosetoatrimconditionratherthenmatchingthefullorientationofthedrogue,asillustratedinFigure 10-34 forpitchangle. 146

PAGE 147

B CFigure10-33. Positiontrackingerror:A)North,B)East,andC)Altitude TheerrorinheadingisdepictedinFigure 10-35 whichshowsacceptabletrackingperformanceoverthetimeinterval. B CFigure10-34. Targetattitudetracking:A)Roll,B)Pitch,andC)Yaw Figure10-35. Trackingerrorinheadingangle Theresultsshownintheseplotsindicatethatthetrackinginthelateralpositionandaltitudearenearlysufcientfortherefuelingtask.Thesimulationrevealsboundederrorsinthese 147

PAGE 148

8 willhelptoaidthecontroller,oratleasttohelpdeterminearegionofwherethefeaturesmostlikelyhavetraveled. 7 .Toseewhatlevelsofvariationsexistintheseresultsanuncertaintyanalysiswasperformed.Chapter 4 derivedamethodtocomputeworse-caseboundsonstateestimatesfromthehomographyapproachusingvisualinformation.ThetechniquedescribedinChapter 4 wasusedforthisuncertaintyanalysis.ThetargetestimatesforabsolutepositionandorientationalongwithupperandlowerboundswerecomputedforthissimulationandareshowninFigures 10-36 and 10-37 .These 148

PAGE 149

B CFigure10-36. Target'sinertialpositionwithuncertaintybounds:A)North,B)East,andC)Altitude B CFigure10-37. Target'sinertialattitudewithuncertaintybounds:A)Roll,B)Pitch,andC)Yaw Themaximumuncertaintiesintargetpositionrelativetotheearth-xedframearesummarizedinTable 10-9 .Meanwhile,Table 10-10 containsthemaximumuncertaintiesintargetorientation.Thethreelevelsofuncertaintyareincludedinthesetables.Thiscomparisonhelpstoverifythatthemaximumstatevariationcorrespondstothemaximumcameraparameter 149

PAGE 150

Table10-9. Maximumvariationsinpositionduetoparametricuncertainty uncertaintyparameter north(ft) 4.10 20.54 10.53 14.40 15.09 30.82 Table10-10. Maximumvariationsinattitudeduetoparametricuncertainty uncertaintyparameter 0 0 4.48 2.29 7.94 3.48 150

PAGE 151


PAGE 152


PAGE 153

8 intotherefuelingsimulationwillhelpthecontrollerbyprovidingstateestimatewhenthetargetexitstheeldofview. 153

PAGE 154

[1] SecretaryofDefense,UnmannedAircraftSystemsRoadmap2005-2030,website: Grasmeyer,J.M.,andKeennon,M.T.,DevelopmentoftheBlackWidowMicro-AirVehicle,39thAerospaceSciencesMeetingandExhibit,AIAA2001-0127,Reno,NV,January2001. [3] Beard,R.,Kingston,D.,Quigley,M.,Snyder,D.,Christiansen,R.,Johnson,W.,Mclain,T.,andGoodrich,M.,AutonomousVehicleTechnologiesforSmallFixedWingUAVs,AIAAJournalofAerospaceComputing,Information,andCommunication,Vol.2,No.1,January2005,p.92-108. [4] Kingston,D.,Beard,R.,McLain,T.,Larsen,M.,andRen,W.,AutonomousVehicleTechnologiesforSmallFixedWingUAVs,AIAA2ndUnmannedUnlimitedSystems,Technologies,andOperationsAerospace,Land,andSeaConferenceandWorkshopandExhibit,AIAA-2003-6559,SanDiego,CA,September2003. [5] Frew,E.,ObserverTrajectoryGenerationforTarget-MotionEstimationUsingMonocularVision,PhDDissertation,StanfordUniversity,August2003. [6] Sattigeri,R.,Calise,A.J.,SooKim,B.,Volyanskyy,K.,andNakwan,K.,-DOFNonlinearSimulationofVision-basedFormationFlight,AIAAGuidance,NavigationandControlConferenceandExhibit,AIAA-2005-6002,SanFrancisco,CA,August2005. [7] Beard,R.,Mclain,T.,Nelson,D.,andKingston,D.,DecentralizedCooperativeAerialSurveillanceusingFixed-WingMiniatureUAVs,IEEEProceedings:SpecialIssueonMulti-RobotSystems,Vol.94,Issue7,July2006,pp.1306-1324. [8] Wu,A.D.,Johnson,E.N.,andProctor,A.A.,Vision-AidedInertialNavigationforFlightControl,AIAAGuidance,Navigation,andControlConferenceandExhibit,AIAA2005-5998,SanFrancisco,CA,August2005. [9] EttingerS.M.,Nechyba,M.C.,Ifju,P.G.,andWaszak,M.,Vision-GuidedFlightStabilityandControlforMicroAirVehicle,IEEE/RSJinternationalConferenceonIntelligentRobotsandSystem,Vol.3,September/October2002,pp.2134-2140. [10] Kehoe,J.,Causey,R.,Abdulrahim,M.,andLind,R.,WaypointNavigationforaMicroAirVehicleusingVision-BasedAttitudeEstimation,AIAAGuidance,Navigation,andControlConferenceandExhibit,AIAA-2005-6400,SanFrancisco,CA,August2005. [11] Abdulrahim,M.,andLind,R.,ControlandSimulationofaMulti-RoleMorphingMicroAirVehicle,AIAAGuidance,NavigationandControlConferenceandExhibit,AIAA-2005-6481,SanFrancisco,CA,August2005. [12] Abdulrahim,M.,Garcia,G.,Ivey,G.F.,andLind,R.,FlightTestingofaMicroAirVehicleUsingMorphingforAeroservoelasticControl,AIAAStructures,StructuralDynamics,andMaterialsConference,AIAA-2004-1674,PalmSprings,CA,April2004. 154

PAGE 155

Garcia,H.,Abdulrahim,M.,andLind,R.,RollControlforaMicroAirVehicleUsingActiveWingMorphing,AIAAGuidance,NavigationandControlConferenceandExhibit,AIAA-2003-5347,Austin,TX,August2003. [14] Waszak,M.R.,Jenkins,L.N.,andIfju,P.G.,StabilityandControlPropertiesofanAeroelasticFixedWingMicroAirVehicle,AIAAAtmosphericFlightMechanicsConferenceandExhibit,AIAA2001-4005,Montreal,Canada,August2001. [15] Kimmett,J.,Valasek,J.,andJunkinsJ.K.,VisionBasedControllerforAutonomousAerialRefueling,IEEEInternationalConferenceonControlApplications,Glasgow,Scotland,U.K.,September2002,pp.1138-1143. [16] Tandale,M.D.,Bowers,R.,andValasek,J.,RobustTrajectoryTrackingControllerforVisionBasedProbeandDrogueAutonomousAerialRefueling,AIAAGuidance,Navigation,andControlConferenceandExhibit,AIAA2005-5868,SanFrancisco,CA,August2005. [17] Lucas,B.,andKanade,T.,AnIterativeImageRegistrationTechniquewithanApplicationtoStereoVision,ProceedingsoftheDARPAImageUnderstandingWorkshop,1981,pp.121-130. [18] Tomasi,C.,andKanade,T.,DetectionandTrackingofPointFeatures,Tech.ReportCMU-CS-91-132,CarnegieMellonUniversity,April1991. [19] Kanade,T.,Collins,R.,Lipton,A.,Burt,P.,andWixson,L.,AdvancesinCooperativeMulti-SensorVideoSurveillance,ProceedingsofDARPAImageUnderstandingWork-shop,Vol.1,November1998,pp.3-24. [20] Piccardi,M.,BackgroundSubtractionTechniques:AReview,IEEEInternationalConferenceonSystems,ManandCybernetics,TheHague,TheNetherlands,October2004. [21] Schunck,B.G.,MotionSegmentationbyConstraintLineClusttering,IEEEWorkshoponComputerVision:RepresentationandControl,1984,pp.58-62. [22] Ridder,C.,Munkelt,O.,andKirchner,H.,AdaptiveBackgroundEstimationandForegroundDetectionusingKalman-Filtering,InternationalConferenceonRecentAdvancedinMechatronics,Istanbul,Turkey,June1995,pp.193-199. [23] Bailo,G.,Bariani,M.,Ijas,P.,andRaggio,M.,BackgroundEstimationwithGaussianDistributionforImageSegmentation,aFastApproach,IEEEInternationalWorkshoponMeasurementSystemsforHomelandSecurity,ContrabandDetectionandPersonalSafety,Orlando,FL,March2005. [24] Friedman,N.,andRussel,S.,ImageSegmentationinVideoSequences:AProbabilisticApproach,InternationalProceedingsoftheThirteenthConferenceofUncertaintyinArticialIntelligence(UAI),Providence,RI,August1997. 155

PAGE 156

Sheikh,Y.,andShah,M.,BayesianObjectDetectioninDynamicScenes,IEEEComputerSocietyConferenceonComputerVisionandPatternRecognition,SanDiego,CA,June2005. [26] Stauffer,C.,andGrimson,W.E.L.,AdaptiveBackgroundMixtureModelsforReal-TimeTracking,IEEEConferenceonComputerVisionandPatternRecognition,FortCollins,CO,June1999,pp.246-252. [27] Toyama,K.,Krumm,J.,Brumitt,B.,andMeyers,B.,Wallower:PrinciplesandPracticeofBackgroundMaintenance,InternationalConferenceonComputerVision,Corfu,Greece,September1999. [28] Zhou,D.,andZhang,H,ModiedGMMBackgroundModelingandOpticalFlowforDetectionofMovingObjects,IEEEInternationalConferenceonSystem,Man,andCybernetics,BigIsland,Hawaii,October2005. [29] Nelson,R.C.,QualitativeDetectionofmotionbyaMovingObserver,InternationalJournalofComputerVision,Vol.7,No.1,1991,pp.33-46. [30] Thompson,W.B.,andPong,T.G.,DetectingMovingObjects,InternationalJournalofComputerVision,Vol.4,1990,pp.39-57. [31] Odobez,J.M.,andBouthemy,P.,DetectionofMultipleMovingObjectsUsingMultiscaleMRPWithCameraMotionCompensation,IEEEInternationalConferenceonImageProcessing,Austin,TX,November1994,pp.245-249. [32] Irani,M.,Rousso,B.,andPeleg,S.,DetectingandTrackingMultipleMovingObjectsUsingTemporalIntegration,EuropeanConferenceonComputerVision,SantaMargheritaLigure,Italy,May1992pp.282-287. [33] Torr,P.H.S.,andMurray,D.W.,StatisticalDetectionofIndependentMovementfromaMovingCamera,ImageandComputing,Vol.11,No.4,May1993,pp.180-187. [34] Gandhi,T.,Yang,M.T.,Kasturi,R.,Camps,O.,Coraor,L.,andMcCandless,J.,DetectionofObstaclesintheFlightPathofanAircraft,IEEETransactionsonAerospaceandElectronicSystems,Vol.39,No.1,January2003,pp.176-191. [35] Irani,M.,andAnandan,P.,AUniedApproachtoMovingObjectDetectionin2Dand3DScenes,IEEETransactionsonPatternAnalysisandMAchineIntelligence,Vol.20,No.6,June1998. [36] Sharma,R.,andAloimonos,Y.,EarlyDetectionofIndependentMotionfromActiveControlofNormalFlowPatterns,IEEETransactionsonSystems,Man,andCybernetics,Vol.26,No.1,February1996. [37] Frazier,J.,andNevatia,R.,DetectingMovingObjectsfromaMovingPlatform,IEEEInternationalConferenceonRoboticsandAutomation,Nice,France,May1992,pp.1627-1633. 156

PAGE 157

Liu,Y.,Huang,T.S.,andFaugeras,O.D.,DeterminationofCameraLocationfrom2Dto3DlineandPointCorrespondence,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.12,No.1,January1990pp.28-37. [39] Longuet-Higgins,H.C.,AComputerAlgorithmforReconstructingaScenefromTwoProjections,Nature,Vol.293,September1981,pp.133-135. [40] Heeger,D.J.,andJepson,A.D.,SubspaceMethodforRecoveringRigidMotion1:AlgorithmandImplementation,InternationalJournalofComputerVision,Vol.7,No.2,January1992. [41] Gutmann,J.S.,andFox,D.,AnExperimentalComparisonofLocalizationMethodsContinued,IEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,Lausanne,Switzerland,October2002. [42] Martin,M.C.,andMoravec,H.,RobotEvidenceGrids,TechnicalReportCMU-RI-TR-96-06,RoboticsInstitute,CarnegieMellonUniversity,March1996. [43] Olson,C.F.,SelectingLandmarksforLocalizationinNaturalTerrains,AutonomousRobots,Vol.12,2002,pp.201-210. [44] Olson,C.F.,andMatthies,L.H.,MaximumLikelihoodRoverLocalizationbyMatchingRangeMaps,IEEEInternationalConferenceonRoboticsandAutomation,Leuven,Belgium,May1998,pp.272-277. [45] Volpe,R.,Estlin,T.,Laubach,S.,Olson,C.,andBalaram,J.,EnhancedMarsRoverNavigationTechniques,IEEEInternationalConferenceonRoboticsandAutomation,SanFrancisco,CA,April2000,pp.926-931. [46] Gurl,P.,andRotstein,H.,PartialAircraftStateEstimationfromVisualMotionUsingtheSubspaceContraintsApproach,JournalofGuidance,Control,andDynamics,Vol.24,No.5,September-October2001,pp.1016-1028. [47] Markel,M.D.,Lopez,J.,Gebert,G.,andEvers,J.,Vision-AugmentedGNC:PassiveRangingfromImageFlow,AIAAGuidance,Navigation,andControlConferenceandExhibit,AIAA2002-5026,Monterey,CA,August2002. [48] Webb,T.P.,Prazenica,R.J.,Kurdila,A.J.,andLind,R.,Vision-BasedStateEstimationforAutonomousMicroAirVehicles,AIAAGuidance,Navigation,andControlConfer-enceandExhibit,Providence,RI,August2004. [49] Webb,T.P.,Prazenica,R.J.,Kurdila,A.J.,andLind,R.,Vision-BasedStateEstimationforUninhibitedAerialVehicles,AIAAGuidance,Navigation,andControlConferenceandExhibit,AIAA2005-5869,SanFrancisco,CA,August2005. [50] Chatterji,G.B.,Vision-BasedPositionandAttitudeDeterminationforAircraftNightLanding,AIAAGuidance,NavigationandControlConferenceandExhibit,AIAA-96-3821,July1996. 157

PAGE 158

Silveira,G.F.,Carvalho,J.R.H.,Madirid,M.K.,Rives,P.,andBueno,S.S.,AFastVision-BasedRoadFollowingStrategyAppliedtotheControlofAerialRobots,IEEEProceedingsofXIVBrazilianSymposiumonComputerGraphicsandImageProcessing,1530-1834/01,Florianopolis,Brazil,October2001,pp.226-231. [52] Soatto,S.,andPerona,P.,DynamicVisualMotionEstimationfromSubspaceConstraints,IEEE,0-8186-6950-0/94,1994,pp.333-337. [53] Soatto,S.,Frezza,R.,andPeronaP.,MotionEstimationviaDynamicVision,IEEETransactionsonAutomaticControl,Vol.41,No.3,March1996,pp.393-413. [54] Soatto,S.,andPerona,P.,Recursive3-DMotionEstimationUsingSubspaceConstraints,InternationalJournalofComputerVision,Vol.22,No.3,1997,pp.235-259. [55] Soatto,S.,andPerona,P.,Reducing'StructurefromMotion',IEEE,1063-6919/96,1996,pp.825-832. [56] Soatto,S.,andPerona,P.,VisualMotionEstimationfromPointFeatures:UniedView,IEEEInternationalConferenceonImageProcessing,Vol.3,October1995,pp.21-24. [57] Soatto,S.,andPerona,P.,Reducing'StructurefromMotion':AGeneralFrameworkforDynamicVisionPart1:Modeling,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.20,No.9,September1998,pp.933-942. [58] Soatto,S.,andPerona,P.,Reducing'StructurefromMotion':AGeneralFrameworkforDynamicVisionPart2:ImplementationandExperimentalAssessment,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.20,No.9,September1998,pp.943-960. [59] Erol,A.,Bebis,G.,Nicolescu,M.,Boyle,R.D.,andTwombly,X.,AReviewonVision-BasedFullDOFHandMotionEstimation,IEEEComputerSocietyInternationalConferenceonComputerVisionandPatternRecognition,SanDiego,CA,June2005. [60] Huang,T.S.,andNetravali,A.N.,MotionandStructurefromFeatureCorrespondences:AReview,ProceedingsoftheIEEE,Vol.82,No.2,February1994,pp.252-268. [61] Stewart,C.V.,RobustParameterEstimationinComputerVision,SocietyofIndustrialandAppliedMathematics,Vol.41,No.3,1999,pp.513-537. [62] Weng,J.,Huang,T.S.,andAhuja,N.,MotionandStructurefromTwoPerspectiveViews:Algorithms,ErrorAnalysis,andErrorEstimation,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.11,No.5,May1989,pp.451-476. [63] Jianchao,Y.,ANewMethodforPassiveLocationEstimationfromImageSequenceUsingAdaptiveExtendedKalmanFilter,InternationalConferenceonSignalProcessing,Beijing,China,October1998,pp.1002-1005. [64] Qian,G.,Kale,G.,andChellappa,R.,RobustEstimationofMotionandStructureusingaDiscreteH-innityFilter,IEEE0-7803-6297-7/00,2000,pp.616-619. 158

PAGE 159

Weng,J.,Ahuja,N.,andHuang,T.S.,OptimalMotionandStructureEstimation,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.15,No.9,September1993,pp.864-884. [66] Blostein,S.D.,Zhao,L.,Chann,andR.M.,Three-DimensionalTrajectoryEstimationfromImagePositionandVelocity,IEEETransactionsonAerospaceandElectronicSystems,Vol.36,No.4,October2000,pp.1075-1089. [67] Broida,T.J.,Chandrashekhar,S.,andChellappa,R.,Recursive3-DMotionEstimationfromaMonocularImageSequence,IEEETransactionsonAerospaceandElectronicSystems,Vol.26,No.4,1990,pp.639-656. [68] Aidala,V.J.,KalmanFilterBehaviorinBearings-OnlyTrackingApplications,IEEETransactionsonAerospaceandElectronicSystems,Vol.15,No.1,1979,pp.29-39. [69] Bolger,P.L.,TrackingaManeuveringTargetUsingInputEstimation,IEEETransac-tionsonAerospaceandElectronicSystems,Vol.23,No.3,1987,pp.298-310. [70] Gavish,M.,andFogel,E.,EffectofbiasonBearings-OnlyTargetLocation,IEEETransactionsonAerospaceandElectronicSystems,Vol.26,No.1,January1990,pp.22-25. [71] Peach,N.,Bearings-OnlyTrackingUsingaSetofRange-ParameterizedExtendedKalmanFilters,IEEProceedings-ControlTheoryandApplications,Vol.142,No.1,January1995,pp.73-80. [72] Taff,L.G.,TargetLocalizationfromBearings-OnlyObservations,IEEETransactionsonAerospaceandElectronicSystems,Vol.33,No.1,January1997,pp.2-9. [73] Guanghui,O.,Jixiang,S.,Hong,L.,andWenhui,W.,Estimating3DMotionandPositionofaPointTarget,ProceedingsofSPIE,Vol.3173,1997,pp.386-394. [74] Fang,Y.,Dawson,D.M.,Dixon,W.E.,andQueiroz,,.5DVisualServoingofWheeledMobileRobots,IEEEConferenceonDecisionandControl,LasVegas,NV,December2002,pp.2866-2871. [75] Chen,J.,Dixon,W.E.,Dawson,D.M.,andChitrakaranV.K.,VisualServoTrackingControlofaWheeledMobileRobotwithaMonocularFixedCamera,IEEEConferenceonControlApplications,Taipei,Taiwan,September2004,pp.1061-1066. [76] Chen,J.,Dawson,D.M.,Dixon,W.E.,andBehal,A.,AdaptiveHomography-BasedVisualServoTrackingforFixedandCamera-in-HandCongurations,IEEETransactionsonControlSystemsTechnology,Vol.13,No.5,September2005,pp.814-825. [77] Mehta,S.S.,Dixon,W.E.,MacArthur,D.,andCrane,C.D.,VisualServoControlofanUnmannedGroundVehicleviaaMovingAirborneMonocularCamera,IEEEAmericanControlConference,Minneapolis,MN,June2006. 159

PAGE 160

Junkins,J.L.,andHughes,D.,Vision-BasedNavigationforRendezvous,DockingandProximityOperations,AASGuidanceandControlsConference,Breckenridge,CO,February1999. [79] Alonso,R.,Crassidis,J.L.,andJunkins,J.L.,Vision-BasedRelativeNavigationforFormationFlyingofSpacecraft,AIAAGuidance,NavigationandControlConferenceandExhibit,AIAA-2000-4439,Denver,CO,August2000. [80] Houshangi,N.,ControlofaRoboticManipulatortoGraspaMovingTargetusingVision,IEEEInternationalConferenceonRoboticsandAutomation,CH2876-1/90/0000/0604,Cincinnati,Ohio,1990,pp.604-609. [81] Hansen,J.L.,Murry,J.E.,andCampos,N.V.,TheNASADrydenAARProject:AFlightTestApproachtoanAerailRefuelingSystem,AIAAAtmosphericFlightMechanicsConferenceandExhibit,Providence,RhodeIsland,August2004. [82] Chang,P.,andHebert,P.,RobustTrackingandStructurefromMotionthroughSamplingBasedUncertaintyRepresentation,InternationalConferenceonRoboticsandAutoma-tion,WashingtonD.C.,May2002. [83] Oliensis,J.,ExactTwo-ImageStructureFromMotion,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.24,No.12,December2002,pp.1618-1633. [84] Svoboda,T.,andSturm,P.,BadlyCalibratedCamerainEgo-MotionEstimation-PropagationofUncertainty,InternationalConferenceComputerAnalysisofImageandPatterns,Kiel,Germany,September1997,pp.183-190. [85] Zhang,Z.,DeterminetheEpipolarGeometryanditsUncertainty:AReview,Interna-tionalJournalofComputerVision,Vol.27,No.2,1998,pp.161-195. [86] Qian,G.,andChellappa,R.,StructureFromMotionUsingSequentialMonteCarloMethods,InternationalConferenceOnComputerVision,Vancouver,Canada,July2001,pp.614-621. [87] Franke,U.,andHeinrich,S.,FastObstacleDetectionforUrbanTrafcSituations,IEEETransactionsonIntelligentTransportationSystems,Vol.3,No.3,September2002,pp.173-181. [88] Bhanu,B.,Das,S.,Roberts,B.,andDuncan,D.,ASystemforObstacleDetectionDuringRotorcraftLowAltitudeFlight,IEEETransactionsonAerospaceandElectronicSystems,Vol.32,No.3,July1996,pp.875-897. [89] Huster,A.,Fleischer,S.D.,andRock,S.M.,DemonstrationofaVision-BasedDead-ReckoningSystemforNavigationofanUnderwaterVehicle,OCEANS'98ConferenceProceedings,0-7803-5045-6/98,Vol.1,September1998,pp.326-330. [90] Roderick,A.,Kehoe,J.,andLind,R.,Vision-BasedNavigationusingMulti-RateFeedbackfromOpticFlowandSceneReconstruction,AIAAGuidance,Navigation,andControlConferenceandExhibit,SanFrancisco,CA,August2005. 160

PAGE 161

Papaikolopoulos,N.P.,Nelson,B.J.,andKhosla,P.K.,SixDegree-of-FreedomHand/EyeVisualTrackingwithUncertainParameters,IEEETransactionsonRoboticsandAutomation,Vol.11,No.5,October1995,pp.725-732. [92] Sznaier,M.,andCamps,O.I,ControlIssuesinActiveVision:OpenProblemsandSomeAnswers,IEEEConferenceonDecisionandControl,Tampa,FL,December1998,pp.3238-3244. [93] Frezza,R.,Picci,G.,andSoatto,S.,Non-holonomicModel-basedPredictiveOutputTrackingofanUnknownThree-dimensionalTrajectory,IEEEConferenceonDecisionandControl,Tampa,FL,December1998,pp.3731-3735. [94] Papaikolopoulos,N.P.,Khosla,P.K.,andKanade,T.,VisualTrackingofaMovingTargetbyaCameraMountedonaRobot:ACombinationofControlandVision,IEEETransactionsonRoboticsandAutomationVol.9,No.1,February1993,pp.14-35. [95] Papaikolopoulos,N.P.,andKhosla,P.K.,AdaptiveRoboticVisualTracking:TheoryandExperiments,IEEETransactionsonAutomaticControl,Vol.38,No.3,March1993,pp.429-445. [96] Zanne,P.,Morel,G.,andPlestan,F.,RobustVisionBased3DTrajectoryTrackingusingSlidingModeControl,IEEEInternationalConferenceonRoboticsandAutomation,SanFrancisco,CA,April2000,pp.2088-2093. [97] Zergeroglu,E.,Dawson,D.M.,deQueiroz,M.S.,andBehal,A.,Vision-BasedNonlinearTrackingControllerswithUncertainRobot-CameraParameters,IEEE/ASMEInternationalConferenceonAdvancedMechanics,Atlanta,GA,September1999,pp.854-859. [98] Valasek,J.,Kimmett,J.,Hughes,D.,Gunnam,K.,andJunkins,J.L.,VisionBasedSensorandNavigationSystemforAutonomousAerialRefueling,AIAA's1stTechnicalConferenceandWorkshoponUnmannedAerospaceVehicles,Portsmouth,Virginia,May2002. [99] Pollini,L.,Campa,G.,Giulietti,F.,andInnocenti,M.,VirtualSimulationSet-UpforUAVsAerialRefueling,AIAAModelingandSimulationTechnologiesConference,Austin,TX,August2003. [100] No,T.S.,andCochan,J.E.,DynamicsandCOntrolofaTeatheredFlightVehicle,JournalofGuidance,Control,andDynamics,Vol.18,No.1,January1995,pp.66-72. [101] Forsyth,D.A.,andPonce,J.,ComputerVision:AModernApproach,Prentice-HallPublishers,UpperSaddleRiver,NJ,2003. [102] Ma,Y.,Soatto,S.,Kosecka,andSastry,S.S.,AnInvitationto3-DVision:FromImagestoGeometricModels,Springer-VerlagPublishing,NewYork,NY,2004. [103] Faugeras,O.,Three-DimensionalComputerVision,TheMITPress,CambridgeMassachusetts,2001. 161

PAGE 162

Castro,G.J.,Nieto,J.,Gallego,L.M.,Pastor,L.,andCabello,E.,AnEffectiveCameraCalibrationMethod,IEEE0-7803-4484-7/98,1998,pp.171-174. [105] Tsai,R.Y.,AVersatileCameraCalibrationTechniqueforHight-Accuracy3DMachineVisionMetrologyUsingOff-theshelfTVCamerasandLens,IEEEJournalofRoboticsandAutomation,Vol.RA-3,No.4,August1987,pp.323-344. [106] Harris,C.,andStephens,M.,ACombinedCornerandEdgeDetector,ProceedingsoftheAlveyVisionConference,1988,pp.147-151. [107] Canny,J.F.,AComputationalApproachtoEdgeDetection,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.8,No.6,November1986,pp.679-698. [108] Etkin,B.,andReid,L.D.,DynamicsofFlight:StabilityandControl,JohnWiley&Sons,NewYork,1996. [109] Nelson,R.C.,FlightStabilityandAutomaticControl,McGraw-Hill,NewYork,1989. [110] Stevens,B.L.,andLewis,F.L.,AircraftControlandSimulation,JohnWiley&Sons,Inc.,NewYork,1992. [111] Kehoe,J.J.,Causey,R.S.,Arvai,A.,andLind,R.,PartialAircraftStateEstimationfromOpticalFlowusingNon-Model-BasedOptimization,IEEEAmericanControlConference,Minneapolis,MN,June2006. [112] KaminskiJ.Y.,andTeicher,M.,AGeneralFrameworkforTrajectoryTriangulation,JournalofMathematicalImagingandVision,Vol.21,2004,pp.27-41. [113] AvidanS.,andShashuaA.,TrajectoryTriangulation:3DReconstructionofMovingPointsfromaMonocularImageSequence,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.22,No.4,2000,pp.348-357. [114] Fitzgibbon,A.W.,andZisserman,A.,MultibodyStructureandMotion:3DReconstructionofIndependentlyMovingObjects,EuropeanConferenceonComputerVision,Dublin,Ireland,July2000,Vol.1,pp.891-906. [115] Han,M.,andKanade,T.,ReconstructionofaScenewithMultipleLinearlyMovingObjects,InternationalJournalofComputerVision,Vol.59,No.3,2004,pp.285-300. [116] Ozden,K.E.,Cornelis,K.,VanEycken,L.,andVanGool,L.,Reconstructing3DTrajectoriesofIndependentlyMovingObjectsusingGenericConstraints,JournalofComputerVisionandImageUnderstanding,Vol.96,No.3,2004,pp.453-471. [117] YuanC.,andMedioni,G.,DReconstructionofBackgroundandObjectsMovingonGroundPlaneViewedfromaMovingCamera,IEEEConferenceonComputerVisionandPatternRecognition,NewYork,NY,June2006. 162

PAGE 163

Dobrokhodov,V.N.,Kaminer,I.I.,Jones,K.D.,andGhabcheloo,R.,Vision-BasedTrackingandMotionEstimationforMovingTargetsusingSmallUAVs,AmericanControlConference,Minneapolis,MN,June2006. [119] Faugeras,O.,andLustman,F.,MotionandStructureFromMotioninaPiecewisePlanarEnvironment,InternationalJournalofPatternRecognitionandArticialIntelligence,Vol.2,No.3,pp.485-508,1988. [120] Zhang,Z.,andHanson,A.R.,ScaledEuclidean3DReconstructionBasedonExternallyUncalibratedCameras,IEEESymp.onComputerVision,1995,pp.37-42. [121] Zhu,Q.,AStochasticAlgorithmforObstacleMotionPredictioninVisualGuidanceofRobotMotion,IEEEInternationalConferenceonSystemsEngineering,Pittsburgh,PA,August1990. 163

PAGE 164

RyanScottCauseywasborninMiami,Florida,onMay10,1978.Hegrewupinastablefamilywithonebrotherinatypicalsuburbanhome.Duringhisteenageyearsandintoearlyadolescence,Ryanbuiltandmaintainedasmallbusinessprovidinglawncaretothelocalneighborhood.Thetoolsacquiredfromthisworkcarriedoverintohiscollegecareer.AftergraduatingfromMiamiKillianSeniorHighSchoolin1996,RyanattendedMiamiDadeCommunityCollegeforthreeyearsandreceivedanAssociateinArtsdegree.AtransferstudenttotheUniversityofFlorida,Ryanwaspreparedtotacklethestressesofauniversityasidefromthepoorstatisticsontransferstudents.Afewyearslater,hereceivedaBachelorofScienceinAerospaceEngineeringwithhonorsin2002andwasconsideredinthetopthreeofhisclass.RyansoonafterchosetoattendgraduateschoolbackattheUniversityofFloridaunderDr.RickLindintheDynamicsandControlsLaboratory.Duringthesummertime,RyaninternedtwiceatHoneywellSpaceSystemsasaSystemsEngineerinClearwater,FLandonceatTheAirForceResearchLaboratoryinDayton,OH.Vision-basedcontrolofautonomousairvehiclesbecamehisinterestandheisnowpursuingadoctoratedegreeonthistopic.RyanwasawardedaNASAGraduateStudentResearchProgram(GSRP)fellowshipin2004forhisproposedinvestigationonthisresearch. 164