Citation |

- Permanent Link:
- https://ufdc.ufl.edu/UFE0021231/00001
## Material Information- Title:
- Vision-Based Control for Flight Relative to Dynamic Environments
- Creator:
- Causey, Ryan S
- Place of Publication:
- [Gainesville, Fla.]
Florida - Publisher:
- University of Florida
- Publication Date:
- 2007
- Language:
- english
- Physical Description:
- 1 online resource (164 p.)
## Thesis/Dissertation Information- Degree:
- Doctorate ( Ph.D.)
- Degree Grantor:
- University of Florida
- Degree Disciplines:
- Aerospace Engineering
Mechanical and Aerospace Engineering - Committee Chair:
- Lind, Richard C.
- Committee Members:
- Crane, Carl D.
Dixon, Warren E. Slatton, Kenneth C. - Graduation Date:
- 8/11/2007
## Subjects- Subjects / Keywords:
- Aircraft ( jstor )
Cameras ( jstor ) Coordinate systems ( jstor ) Geometric planes ( jstor ) Homography ( jstor ) Moving images ( jstor ) Remotely piloted vehicles ( jstor ) Simulations ( jstor ) State estimation ( jstor ) Velocity ( jstor ) Mechanical and Aerospace Engineering -- Dissertations, Academic -- UF autonomous, camera, estimation, homography, moving, tracking, vision, visual City of Miami ( local ) - Genre:
- bibliography ( marcgt )
theses ( marcgt ) government publication (state, provincial, terriorial, dependent) ( marcgt ) born-digital ( sobekcm ) Electronic Thesis or Dissertation Aerospace Engineering thesis, Ph.D.
## Notes- Abstract:
- The concept of autonomous systems has been considered an enabling technology for a diverse group of military and civilian applications. The current direction for autonomous systems is increased capabilities through more advanced systems that are useful for missions that require autonomous avoidance, navigation, tracking, and docking. To facilitate this level of mission capability, passive sensors, such as cameras, and complex software are added to the vehicle. By incorporating an on-board camera, visual information can be processed to interpret the surroundings. This information allows decision making with increased situational awareness without the cost of a sensor signature, which is critical in military applications. The concepts presented in this dissertation facilitate the issues inherent to vision-based state estimation of moving objects for a monocular camera configuration. The process consists of several stages involving image processing such as detection, estimation, and modeling. The detection algorithm segments the motion field through a least-squares approach and classifies motions not obeying the dominant trend as independently moving objects. An approach to state estimation of moving targets is derived using a homography approach. The algorithm requires knowledge of the camera motion, a reference motion, and additional feature point geometry for both the target and reference objects. The target state estimates are then observed over time to model the dynamics using a probabilistic technique. The effects of uncertainty on state estimation due to camera calibration are considered through a bounded deterministic approach. The system framework focuses on an aircraft platform of which the system dynamics are derived to relate vehicle states to image plane quantities. Control designs using standard guidance and navigation schemes are then applied to the tracking and homing problems using the derived state estimation. Four simulations are implemented in MATLAB that build on the image concepts present in this dissertation. The first two simulations deal with feature point computations and the effects of uncertainty. The third simulation demonstrates the open-loop estimation of a target ground vehicle in pursuit whereas the four implements a homing control design for the Autonomous Aerial Refueling (AAR) using target estimates as feedback. ( en )
- General Note:
- In the series University of Florida Digital Collections.
- General Note:
- Includes vita.
- Bibliography:
- Includes bibliographical references.
- Source of Description:
- Description based on online resource; title from PDF title page.
- Source of Description:
- This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
- Thesis:
- Thesis (Ph.D.)--University of Florida, 2007.
- Local:
- Adviser: Lind, Richard C.
- Statement of Responsibility:
- by Ryan S Causey.
## Record Information- Source Institution:
- UFRGP
- Rights Management:
- Copyright Causey, Ryan S. Permission granted to the University of Florida to digitize, archive and distribute this item for non-profit research and educational purposes. Any reuse of this item in excess of fair use or other copyright exemptions requires permission of the copyright holder.
- Resource Identifier:
- 662596751 ( OCLC )
- Classification:
- LD1780 2007 ( lcc )
## UFDC Membership |

Downloads |

## This item has the following downloads:
causey_r.pdf
causey_r_Page_159.txt causey_r_Page_084.txt causey_r_Page_138.txt causey_r_Page_103.txt causey_r_Page_062.txt causey_r_Page_056.txt causey_r_Page_055.txt causey_r_Page_067.txt causey_r_Page_104.txt causey_r_Page_035.txt causey_r_Page_087.txt causey_r_Page_059.txt causey_r_Page_132.txt causey_r_Page_024.txt causey_r_Page_152.txt causey_r_Page_107.txt causey_r_Page_034.txt causey_r_Page_096.txt causey_r_Page_054.txt causey_r_Page_050.txt causey_r_Page_058.txt causey_r_Page_127.txt causey_r_Page_064.txt causey_r_Page_048.txt causey_r_Page_140.txt causey_r_Page_077.txt causey_r_Page_065.txt causey_r_pdf.txt causey_r_Page_091.txt causey_r_Page_147.txt causey_r_Page_079.txt causey_r_Page_110.txt causey_r_Page_086.txt causey_r_Page_044.txt causey_r_Page_001.txt causey_r_Page_015.txt causey_r_Page_144.txt causey_r_Page_022.txt causey_r_Page_073.txt causey_r_Page_002.txt causey_r_Page_088.txt causey_r_Page_139.txt causey_r_Page_101.txt causey_r_Page_057.txt causey_r_Page_114.txt causey_r_Page_085.txt causey_r_Page_136.txt causey_r_Page_134.txt causey_r_Page_119.txt causey_r_Page_042.txt causey_r_Page_137.txt causey_r_Page_028.txt causey_r_Page_016.txt causey_r_Page_142.txt causey_r_Page_109.txt causey_r_Page_163.txt causey_r_Page_011.txt causey_r_Page_052.txt causey_r_Page_113.txt causey_r_Page_025.txt causey_r_Page_076.txt causey_r_Page_045.txt causey_r_Page_080.txt causey_r_Page_046.txt causey_r_Page_116.txt causey_r_Page_162.txt causey_r_Page_018.txt causey_r_Page_120.txt causey_r_Page_014.txt causey_r_Page_074.txt causey_r_Page_007.txt causey_r_Page_083.txt causey_r_Page_039.txt causey_r_Page_133.txt causey_r_Page_135.txt causey_r_Page_100.txt causey_r_Page_130.txt causey_r_Page_089.txt causey_r_Page_149.txt causey_r_Page_072.txt causey_r_Page_036.txt causey_r_Page_106.txt causey_r_Page_141.txt causey_r_Page_128.txt causey_r_Page_125.txt causey_r_Page_118.txt causey_r_Page_082.txt causey_r_Page_158.txt causey_r_Page_094.txt causey_r_Page_009.txt causey_r_Page_031.txt causey_r_Page_006.txt causey_r_Page_081.txt causey_r_Page_023.txt causey_r_Page_090.txt causey_r_Page_115.txt causey_r_Page_030.txt causey_r_Page_146.txt causey_r_Page_063.txt causey_r_Page_123.txt causey_r_Page_041.txt causey_r_Page_151.txt causey_r_Page_126.txt causey_r_Page_047.txt causey_r_Page_043.txt causey_r_Page_019.txt causey_r_Page_068.txt causey_r_Page_003.txt causey_r_Page_004.txt causey_r_Page_099.txt causey_r_Page_150.txt causey_r_Page_129.txt causey_r_Page_049.txt causey_r_Page_124.txt causey_r_Page_017.txt causey_r_Page_070.txt causey_r_Page_117.txt causey_r_Page_066.txt causey_r_Page_145.txt causey_r_Page_020.txt causey_r_Page_071.txt causey_r_Page_053.txt causey_r_Page_102.txt causey_r_Page_013.txt causey_r_Page_095.txt causey_r_Page_111.txt causey_r_Page_027.txt causey_r_Page_157.txt causey_r_Page_164.txt causey_r_Page_078.txt causey_r_Page_092.txt causey_r_Page_131.txt causey_r_Page_156.txt causey_r_Page_051.txt causey_r_Page_154.txt causey_r_Page_021.txt causey_r_Page_069.txt causey_r_Page_026.txt causey_r_Page_075.txt causey_r_Page_037.txt causey_r_Page_005.txt causey_r_Page_161.txt causey_r_Page_148.txt causey_r_Page_060.txt causey_r_Page_010.txt causey_r_Page_061.txt causey_r_Page_093.txt causey_r_Page_155.txt causey_r_Page_098.txt causey_r_Page_160.txt causey_r_Page_097.txt causey_r_Page_105.txt causey_r_Page_040.txt causey_r_Page_122.txt causey_r_Page_008.txt causey_r_Page_153.txt causey_r_Page_112.txt causey_r_Page_029.txt causey_r_Page_033.txt causey_r_Page_038.txt causey_r_Page_012.txt causey_r_Page_032.txt causey_r_Page_121.txt causey_r_Page_108.txt causey_r_Page_143.txt |

Full Text |

CHAPTER 2 LITERATURE REVIEW The proliferation of autonomous systems is generating a demand for smarter, more complex vehicles. The motivation behind these concept vehicles is to operate in urban environments which requires a number of complex systems. Video cameras have been chosen as sensors to facilitate topics such as obstacle detection and avoidance, target tracking and path planning. These technologies have stemmed from two communities in the literature: (i) image processing and computer vision and (ii) performance and control of autonomous vehicles. This chapter will focus on the research applied to autonomous systems and describe the current state of this research, problems that have been addressed, some difficulties associated with vision, and some areas in need of contribution. In particular, the review will cover the topics of most relevance to this dissertation and highlight the efforts toward autonomous UAV The block diagram shown in Figure 1-6 illustrates the components of interest described in this dissertation for state estimation and tracking control with respect to a moving object which involves object motion detection, object state estimation, and object motion modeling and prediction. The literature review of these topics is given in this section. 2.1 Detection of Moving Objects In order to track and estimate the motion of objects in images using a monocular camera system, a number of steps are required. A common first step in many image processing algorithms is feature point detection and tracking. This step determines features of interest, such as comers, in the image that usually correspond to objects of interest, such as windows, in the environment. The famous feature point tracker proposed by Lucas and Kanade [17, 18] has served as a foundation for many algorithms. This technique relies on a smoothness constraint imposed on the optic flow that maintains a constant intensity across small base-line motion of the camera. Many techniques have built upon this algorithm to increase robustness to noise and outliers. Once feature tracking has been obtained, the next process involves segmenting the image for moving objects. The need for such a classification is due the fact that standard image image where the optical axis penetrates the image plane to the upper left hand corner of the image. This translation is done using the terms o,, and ov, given in units of pixels. The skew factor is another intrinsic parameter which accounts for pixels that are not rectangular and is defined as so. The ideal perspective transformation now takes the general form given in Equation 3-11, where pixel mapping, origin translation, and skewness are all considered. p' fs,, fse o,, 1 00 The perspective transformation obtained in Equation 3-11 is rewritten to Equation 3-12. rlzx' = Knorl (3-12) The 3 x 3 matrix K is called the intrinsic parameter matrix or the calibration matrix while the 3 x 4 constant matrix Ho defines the perspective projection, and finally x' represents the homogeneous image coordinates [p', v', 1]' that contain pixel mapping and skew. 3.2.3 Extrinsic Parameters In order to achieve this transformation to image coordinates, both intrinsic and extrinsic parameters must be known or estimated a priori through calibration. The extrinsic parameters of the camera can be described as the geometric relationship between the camera frame and the inertial frame. This relationship consists of the relative position, T, and orientation, R, of the camera frame to an inertial frame. By defining the position vector of a feature point relative to an inertial as 5 = [t, y,(, ]'l, transformations can map the expression found in Equation 3-12 to obtain a general equation that maps feature points in the inertial frame to coordinates in the image plane for a calibrated camera. p' fs,, fse o,, 1 0 Be v' 0 fsv ov 0 0 R (3-13) 1 0 1 001 x 10 4 -011 200 400 Index (counts) 200 400 Index (counts) Figure 10-19. Norm error for A) relative translation and B) relative rotation Figures 10-20 and 10-21 show the relative translation and rotation decomposed into their respective components and expressed in the body frame, B. These components reveal the relative information needed for feedback to track or home in on the target of interest. -4500 -5000 -5500 -600 Index (counts) Index (counts) C Index (counts) Figure 10-20. Relative position states: A) X, B) Y, and C) Z 60 40 20 Index (counts) B Figure 10-21. Relative attitude states: A) Roll, B) Pitch, and C) Yaw -Jc- 100 00O 00 200 400 600 Index (counts) C Index (counts) A BIOGRAPHICAL SKETCH Ryan Scott Causey was born in Miami, Florida, on May 10, 1978. He grew up in a stable family with one brother in a typical suburban home. During his teenage years and into early adolescence, Ryan built and maintained a small business providing lawn care to the local neighborhood. The tools acquired from this work carried over into his college career. After graduating from Miami Killian Senior High School in 1996, Ryan attended Miami Dade Community College for three years and received an Associate in Arts degree. A transfer student to the University of Florida, Ryan was prepared to tackle the stresses of a university aside from the poor statistics on transfer students. A few years later, he received a Bachelor of Science in Aerospace Engineering with honors in 2002 and was considered in the top three of his class. Ryan soon after chose to attend graduate school back at the University of Florida under Dr. Rick Lind in the Dynamics and Controls Laboratory. During the summertime, Ryan interned twice at Honeywell Space Systems as a Systems Engineer in Clearwater, FL and once at The Air Force Research Laboratory in Dayton, OH. Vision-based control of autonomous air vehicles became his interest and he is now pursuing a doctorate degree on this topic. Ryan was awarded a NASA Graduate Student Research Program (GSRP) fellowship in 2004 for his proposed investigation on this research. fo, and uncertainty bounded by size of Ay E R. A similar expression in Equation 4-4 presents the range of values for radial distortion. f = { fo $ f : If || < Ay} (4-3) d = {do+-t8d 116 8d ~d} (4-4) The variations of feature points due to the camera uncertainties can be directly computed. The uncertain parameters given in Equation 4-3 and Equation 4-4 are substituted into the camera model of Equation 4-1 and Equation 4-2. The resulting expressions for feature points are presented in Equations 4-5 and 4-6. pl = foil~lx~ 1+d fo24_ r 2872 (4-5) +3doffi, +3dlo +doi +fid+3f d + 3 fod8yd vl = fo 1+off + + xE28(46 +3dof rl +3dfo +di +f id +3f8df+3f yd d These~~~~~~ eqain deosrt a opiae eaiosi ewe neranyi etr points and uncrtainty in camra~ ,6 paaees The featur poits3,6, actual vary lieal with unerait in foa eghfracmr ihu ail itrinoeeteicuino u=~~~~~~ {u 8, : ,| CHAPTER 1 INTRODUCTION 1.1 Motivation Autonomous systems are an enabling technology to facilitate the needs of both military and civilian applications. The usefulness of autonomous systems ranges from robotic assembly lines for streamlining an operation to a rover exploring the terrain of a distant planet. The main motivation behind these types of systems is the removal of a human operator which in many cases reduces operational cost, human errors, and, most importantly, human risk. In particular, military missions consistently place soldiers in hazardous environments but in the future could be performed using an autonomous system. The federal sector is considering autonomous vehicles, specifically, to play a more prominent role in several missions such as reconnaissance, surveillance, border patrol, space and planet exploration over the next 30 years [1]. This increase in capability for such complex tasks requires technology for more advanced systems to further enhance the situational awareness. Over the past several years, the interest and demand for autonomous systems has grown considerably, especially from the Armed Forces. This interest has leveraged funding opportunities to advance the technology into a state of realizable systems. Some technical innovations that have emerged from these efforts, from a hardware standpoint, consist mainly of increasingly capable microprocessors in the sensors, controls, and mission management computers. The Defense Advanced Research Projects Agency (DARPA) has funded several projects pertaining to the advancement of electronic devices through size reduction, improved speed and performance. From these developments, the capability of autonomous system has been demonstrated on vehicles with strict weight and payload requirements. In essence, the current technology has matured to a point where autonomous systems are physically achievable for complex missions but not yet algorithmically capable. The aerospace community has employed many of the research developed for autonomous systems and applied it to Unmanned Aerial Vehicles (UAV). Many of these vehicles are currently a single camera setup, known as monocular vision, and a two camera setup, known as stereo vision. For monocular vision, a sequence of images are taken over time whereas stereo vision uses two images taken by different cameras at the same time. Motion estimates, using monocular vision, has been solved for the cases associated with the movement of the camera relative to a stationary objects and the reverse problem involving movement of objects relative to a stationary camera. The process of determining camera motion from stationary objects is commonly referred to as localization. Conversely, determining the motion or position of an object in space from a pair of images is known as structure from motion. For fixed objects, simultaneous localization and mapping (SLAM) can be employed to estimate the camera motion in conjunction with the object's locations. Meanwhile, the use of stereo vision allows one to estimate the motion of objects while the camera is also moving. Solutions to these methods are well established in the computer science community and the mathematical details regarding these techniques are provided in Chapter 3. This dissertation will focus on the monocular camera configuration to address the state estimation problem regarding moving targets. The advantage of these techniques becomes more apparent to UAV when applied to guidance, navigation, and control. By mounting a camera on a vehicle, state estimation of the vehicle and objects in the environment can be achieved in some instances through vision processing. Once state estimates are known, they can then be used in feedback. Control techniques can then be utilized for complex missions that require navigation, path planning, avoidance, tracking, homing, etc. This general framework of vision processing and control has been successfully applied to various systems and vehicles including robotic manipulators, ground vehicles, underwater vehicles, and aerial vehicles but there still exists some critical limitations. The problematic issues with using vision for state estimation involves camera nonlinearities, camera calibration, sensitivity to noise, large computational time, limited field of view, and solving the correspondence problem. A particular set of these image processing issues will be addressed directly in this dissertation to facilitate the control of autonomous systems in complex surroundings. Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy VISION-BASED CONTROL FOR FLIGHT RELATIVE TO DYNAMIC ENVIRONMENTS By Ryan Scott Causey August 2007 Chair: Richard C. Lind Major: Aerospace Engineering The concept of autonomous systems has been considered an enabling technology for a diverse group of military and civilian applications. The current direction for autonomous systems is increased capabilities through more advanced systems that are useful for missions that require autonomous avoidance, navigation, tracking, and docking. To facilitate this level of mission capability, passive sensors, such as cameras, and complex software are added to the vehicle. By incorporating an on-board camera, visual information can be processed to interpret the surroundings. This information allows decision making with increased situational awareness without the cost of a sensor signature, which is critical in military applications. The concepts presented in this dissertation facilitate the issues inherent to vision-based state estimation of moving objects for a monocular camera configuration. The process consists of several stages involving image processing such as detection, estimation, and modeling. The detection algorithm segments the motion field through a least-squares approach and classifies motions not obeying the dominant trend as independently moving objects. An approach to state estimation of moving targets is derived using a homography approach. The algorithm requires knowledge of the camera motion, a reference motion, and additional feature point geometry for both the target and reference objects. The target state estimates are then observed over time to model the dynamics using a probabilistic technique. The effects of uncertainty on state estimation due to camera calibration are considered through a bounded deterministic approach. The system framework focuses on an aircraft platform of which the system dynamics are derived to relate vehicle states operational and have served reconnaissance missions during Operation Iraqi Freedom. The Department of Defense (DoD) has recorded over 10,000 flight hours performed by UAV in support of the war in Iraq since September 2004 and that number is expected to increase [1]. Future missions envision UAV to conduct more complex task such as terrain mapping, surveillance of possible threats, maritime patrol, bomb damage assessment, and eventually offensive strike. These missions can span over various types of environments and, therefore, require a wide range of vehicle designs and complex controls to accommodate the associated tasks. The requirements and design of UAV are considered to enable a particular mission capability. Each mission scenario is the driving force of these requirements and are dictated by range, speed, maneuverability, and operational environment. Current UAV range in size from less than 1 pound to over 40,000 pounds. Some popular UAV that are operational, in testing phase, and in the concept phase are depicted in Figure 1-1 to illustrate the various designs. The two UAV on the left, Global Hawk and Predator, are currently in operation. Global Hawk is employed as a high altitude, long endurance reconnaissance vehicle whereas the Predator is used for surveillance missions at lower altitudes. Meanwhile, the remaining two pictures present J-UCAS, which is a joint collaboration for both the Air Force and Navy. This UAV is described as a medium altitude flyer with increased maneuverability over Global Hawk and the Predator and is considered for various missions, some of which have already been demonstrated in flight, such as weapon delivery and coordinated flight. The advancements in sensors and computing technology, mentioned earlier, has facilitated the miniaturization of these UAV, which are referred to as Micro Air Vehicles (MAV). The scale of these small vehicles ranges from a few feet in wingspan down to a few inches. DARPA has also funded the first successful MAV project through AeroVironment, as shown in Figure 1-2, where basic autonomy was first demonstrated at this scale [2]. These small scales allow highly agile vehicles that can maneuver in and around obstacles such as buildings and trees. This capability enables UAV to operate in urban environments, below rooftop levels, to provide -04, -0 /-0 A B C Figure 10-7. Optical flow for nominal (black) and perturbed (red) cameras for A) f = 1.1 and d = 0, B) f = 1.0 and d = 0.01, and C) f = 1.1 and d = 0.01 The vectors in Figure 10-7 indicate several effects of camera perturbations noted in Equations 4-5 and 4-6. The perturbations to focal length scale the feature points so the magnitude of optic flow is uniformly scaled. The perturbations to radial distortion have larger effect as the feature point moves away from the center of the image so the optic flow vectors are altered in direction. The combination of perturbations clearly changes the optic flow in both magnitude and direction and demonstrates the feedback variations that can result from camera variations. The optic flow is computed for images captured by each of the perturbed cameras. The change in optic flow for the perturbed cameras as compared to the nominal camera is represented as 6Sy and is bounded in magnitude, as derived in Equation 4-14, by Ay. The greatest value of Sy- presented by these camera perturbations is compared to the upper bound in Table 10-6. These numbers indicate the variations in optic flow are indeed bounded by the theoretical bound derived in Chapter 4 and indicate the level of flow variations induced from the variations in camera parameters. Table 10-6. Effects of camera perturbations on optic flow Perturbation Analyze Analyze Analyze Set only with 87 only with 8d with 8f and 6d 87 = -0.2 and 6d = -0.02 0.0476 0.0476 0.0040) 0.0040) 0.0496 0.0543 87 = -0. 1 and 8d -0.01 0.0238 0.0476 0.0020 0.0040 0.0252 0.0543 87 = 0.1 and 8d 0.01 0.0238 0.0476 0.0020 0.0040 0.0264 0.0543 87 = 0.2 and 6d = 0.02 0.0476 0.0476 0.0040 0.0040 0.0543 0.0543 10-3 10-4 10-5 10-6 10-7 10-8 10-9 10-10 10-11 10-12 10-13 10-14 10-15 10-16 10-17 10-18 10-19 10-20 10-21 10-22 10-23 10-24 10-25 10-26 10-27 10-28 Optic flow measurements for example 1 ..... Virtual environment for example 2 ...... Feature points across two image frames ..... Uncertainty in feature point ....... Uncertainty results in optic flow ...... Nominal epipolar lines between two image frames Uncertainty results for epipolar geometry .... Nominal estimation using structure from motion . Uncertainty results for structure from motion .. Vehicle trajectories for example 3 ...... Position states of the UAV with on-board camera Attitude states of the UAV with on-board camera Position states of the reference vehicle ...... Attitude states of the reference vehicle ...... Position states of the target vehicle ....... Attitude states of the target vehicle ....... Norm error ....... Relative position states ...... Relative attitude states ...... Virtual environment ....... Inner-loop pitch to pitch command Bode plot .. Pitch angle step response ...... Altitude step response ....... Inner-loop roll to roll command Bode plot .... Roll angle step response ...... Heading response ....... . .126 ..... .. .. .127 ....... .. .128 ........ .. .128 ........ .. .129 . .. ... .. .130 . ...... .. .131 .... .. .. .132 . ... .. .. .133 ........ .. .134 .... .. .. .135 .... .. .. .135 ....... .. .135 ....... .. .136 ........ .. .136 ........ .. .136 ......... .. .. 137 .......... .. 137 .......... .. 137 ........ .. .138 . ... .. .. .141 ........ .. .141 .......... .. 142 . .... .. .. .143 .......... .. 144 ......... .. .. 144 10-29 Open-loop estimation of target's inertial position checking the condition that NTe3 = n3 > 0, where e3 is in the direction of the optical axis normal to the image plane. 3.6.4 Structure from Motion Structure from motion (SFM) is a technique to estimate the location of environmental features in 3D space. This technique utilizes the epipolar geometry in Figure 3-4 and assumes that the rotation, R, and translation, T, between frames is known. Given that, the coordinates of rll and r12 can be computed. Recall, the fundamental relationship repeated here in Equation 3-62. r12 = Rrll + T (3-62) The location of environmental features is obtained by first noting the relationships between feature points and image coordinates given in Equation 3-7 and Equation 3-8. These relationships allow some components of rlx and rly to be written in terms of pu and v which are known from the images. Thus, the only unknowns are the depth components, rl1,z and rl2,z, fOr each image. The resulting system can be cast as Equation 3-63 and solved using a least-squares approach. 2 (R11 z -+R12" +1IR13) Tx z- (R21+R2 +R23 r2,z =I T (3-63) 1 (R31 9+-tR32" +1IR33)41 Tz This equation can be written in a compact form as shown in Equation 3-64 using z = [r12,z, r1,z] as the desired vector of depths. Az = T (3-64) The least-squares solution to Equation 3-64 obtains the depth estimates of a feature point relative to both camera frames. This information along with the image plane coordinates can be used to compute (Tll,x,Tl1,y) and (r12,x,r12,y) by substituting these values back into Equations 3-7 and 3-8. The resulting components of rll can then be converted to the coordinate frame of the second image and it should exactly match r12. These values will never match perfectly due to profile and is provided in Equation 8-11. This acceleration time history is computed implicitly through the position estimates obtained from the homography algorithm [ai (t -1) ,ai (t -2),...,ai (t N+-2),ai (t N+-1)] (8-10) ai (t j) = vi (t j) vi (t j 1) (8-11) The motion profiles given in Equations 8-8 and 8-10 provide the initial motion state description that propagates the Markov transition probability function. The form of the Markov transition probability function is assumed to be a Gaussian density function that only requires two parameters for its representation. The parameters needed for this function include the mean and variance vectors for the acceleration profile given in Equation 8-12. Note, during this chapter pu (x) is the mean operator and not the vertical component in the image plane. Likewise, 02 (X) iS referred to as the variance operator. [p (ai (t +t j)) G2 (ai (t + j))] j ,1 ,k(8-12) The Markov transition function is defined in Equation 8-13, where the arguments consist of the mean and variance pertaining to the estimated acceleration. P (ai (t + j)) = xn Cp (ai (t + j)) 02 (ai (t + j))) (8-13) The initial mean and variance for acceleration are computed in Equations 8-14 and 8-16 for the transition function. The functions f, and f, are chosen based on the desired weighting of the time history and can simply be a weighted linear combination of the arguments. These initial statistical parameters are used in the prediction step and updated once a new measurement is obtained. p (ai (t)) = fe (ai (t 1) ai (t 2) ,. .ai (t N)) (8-14) a2 Si F)) f (a ir l ) i (t-2))2 (8-15) P (x) R RBI REB REF RET REV RFV Rry Rl T T TBI = TEB = U fl V W X Xo Y lk 3 h Probability density function Essential matrix Relative rotation Rotational transformation from body-fixed to camera-fixed coordinates Rotational transformation from Earth-fixed to body-fixed coordinates Rotational transformation from Earth-fixed to reference coordinates Rotational transformation from Earth-fixed to target coordinates Rotational transformation from Earth-fixed to virtual coordinates Rotational transformation from reference-fixed to virtual coordinates Rotational transformation from target-fixed to virtual coordinates Rotational transformation from camera-fixed to virtual coordinates Relative translation Target-fixed coordinate frame Position of camera along {$1,82, ~3} axes Position of aircraft along {&1l, e^2, 83 } axes Control input vector Classification group of features to an independently moving object Virtual coordinate frame Search window in the image Vector of aircraft states Vector of initial aircraft states Feature point measurements in the image plane Camera parameter of the k camera Horizontal angle for field of view (xc,Yc,Zc) - (Xb, Yb ,Zb) The same model was also used for the tanker or reference vehicle. The tanker was exactly trimmed at the same conditions and airspeed as the receiver aircraft and given a specified trajectory to follow. Initially the tanker's position was offset from the receiver's position at the start of the simulation. The values of this offset are described relative to receiver's coordinate frame and are as follows: 500 ft in front (+tX direction), 20 ft to the side (+tY direction), and 100 ft above (-Z direction). The trajectory generated for the tanker aircraft prior to the simulation was a straight and level flight with a slight drift toward the East direction. This lateral variation was added to the trajectory to incorporate all three dimensions into the motion to test in all directions. On the other hand, the modeling of the drogue is much more difficult to characterize and is of much interest in the research community. The stochastic nature of its motion is what makes the modeling so challenging. The flow field affecting the drogue consist of many nonlinear excitations including turbulence due to wake effects and vortex shedding from the tanker aircraft. For this drogue model an offset trajectory of the tanker's motion was used as the drogue's general motion. The offset of the drogue is initially at 200 ft in front +tX direction), O ft to the side (+tY direction), and 70 ft above (-Z direction) relative to the receiver aircraft. More complicated motions of the drogue were considered during testing but resulted in a diverging trajectory for the receiver. This deviation from the desired path was due high rate commands saturating the actuators. Low passing filtering can be incorporated to alleviate this behavior. 10.4.2 Control Tuning The control architecture described in Chapter 9 is integrated and tuned for the nonlinear F-16 model to accomplish this simulation. It was assumed that full state feedback of the aircraft states were measurable including position. The units used in this simulation are given in ft and deg which means the gains determined in the control loops were also found based on these units. First, the pitch tracking for altitude controller is considered. The inner-loop gains for this controller are given as ke = -3 and kg = -2.5. The bode diagram for pitch command to pitch angle is depicted in Figure 10-23 for the specified gains. This diagram reveals the damping The relationship shown in Equation 3-51 can be extended to image coordinates through Equation 3-53. x2 = Hxl (3-53) A similar approach as used in the eight-point algorithm can be used to solve for the entries of H. Multiplying both sides of Equation 3-53 with the skew symmetric matrix xi results in the planar homography constraint shown in Equation 3-54. xiHxy = 0 (3-54) Since H is linear, linear algebra techniques can be used to stack the entries of H as a column vector h and, therefore, Equation 3-54 can be rewritten to Equation 3-55, a'h = 0 (3-55) where a is the Kronecker product of xi and xl. Each feature point correspondence between frames provides two constraints in determining the entries of H. Therefore, to solve for a unique solution of H, Equation 3-55 requires at least four feature point correspondences. These additional constraints can be stacked to form a new constraint matrix ?, as shown in Equation 3-56. 'Y= aX1,a2,8 3, --, nn T (3-56i) Rewriting Equation 3-55 in terms of the new constraint matrix results in Equation 3-57. Wh = 0 (3-57) The standard least-squares estimation can be used to recover H up to a scale factor. Improvements can be made to the solution when more than four feature point correspondences are used in the least-squares solution. The scale factor is then determined as the second largest singular value of the solution H [102, 103], shown in Equation 3-58 for the unknown scaler h. |1| = G2 (H) (3-5 8) and incur cost and time delays. These additional expenses add complexity and eliminate the attractiveness of low cost autonomous systems. Meanwhile, the current appeal of these systems has been the use of low cost off-the-shelf components, such as cameras that are easily replaced. Maintaining a low cost product is a goal for UAV that can be accomplished by considering a vision system. If future operations require a stockpile of thousands of UAV or MAV ready to deploy, then the capability to switch out or replace components in a timely fashion with little cost is a tremendous functionality. Therefore, this dissertation describes a method that would enable cameras to be replaced rapidly and without the need for extensive calibration. 1.5 Contributions The goal of the work presented in this dissertation is to establish a methodology that estimates the states of a moving object using a monocular camera configuration in the presence of uncertainty. The estimates will provide not only critical information regarding target-motion estimation for autonomous systems but also retain confidence values through a distribution around a target's estimate. Previous work has investigated many problems and issues related to this topic but has neglected several key features. In particular, this thesis addresses (i) the physical effects of camera nonlinearities on state estimates, (ii) a multi-layered classification approach to object motion based on visual sensing that determines the confidence measure in the estimates, and (iii) the relationships between vehicle and sensor constraints coupled with sensor fusion in an autonomous system framework. The main contribution of this dissertation is the development of a state estimation process of a dynamic object using a monocular vision system for autonomous navigation. In addition to the main contribution, there exists some secondary contributions solved in the process of facilitating the main goal. The contributions presented in this dissertation consist of the following: *A homography approach to state estimation of moving objects is developed through a virtual camera to estimate the relative pose of the target relative to the true camera. This virtual camera facilitated the estimation process by maintaining a constant homography relative to a known reference object. CHAPTER 11 CONCLUSION Vision-based feedback can be an important tool for autonomous systems and is the primary focus of this dissertation in the context of an unmanned air vehicle. This dissertation describes a methodology for a vehicle, such as a UAV, to observe features within the environment and estimate the states of a moving target using various camera configurations. The complete equation of motion of an aircraft-camera system was derived in its general form that allows multiple cameras. Camera models were summarized and the effects of uncertainty regarding the intrinsic parameters was discussed. Expressions for worse-case bounds were derived for varies vision processing algorithms on a conservative level. A classification scheme was summarized to discern between stationary and moving objects within the image using a focus of expansion threshold method. The homography derivation proposed was the main contribution of this dissertation where the states of a moving target were formulated based on visual information. Some underlining assumptions were imposed on the features and the system to obtain feasible estimates. The two critical assumptions imposed on the features were the planar constraint and the requirement of the distance to a feature on the reference and target vehicles be known and equal. An additional assumption was placed on the system which involved a communication link that allows the vehicle to have access to the states of the reference vehicle. The modeling of the target position attempted to anticipate future locations to enable a predictive capability for the controller and to provide estimates when the features are outside the field of view. The approach summarized here consisted of a Hidden Markov method which has limitations for general 6-DOF motion due to incomplete motion models. Lastly, a standard control design is tuned for an aircraft performing waypoint navigation to use in closed-loop control where commands are generated from the state estimator. Simulations were presented to validated the proposed algorithms and to demonstrate the applications for autonomous vehicles. The first simulation verified the feature point and optic flow computation for a aircraft-camera system containing multiple cameras with time varying during reconstruction. R = UR ( + VT) T = URz( +\ EUT (3-48) 0 +1 0 whr Ry"2+0- 41 0 0 0 01 The eight-point algorithm fails with a non-unique solution when all points in 3D space lie on the same 2D plane [102, 103]. When this situation occurs one must use the planar homography approach, which is the topic of the next section. 3.6.3 Planar Homography The homography approach can be used to solve the degenerate cases of the eight-point algorithm. For instance, a very common case where the feature points of interest all lie on the same 2D plane in 3D space causes the algorithm to produce nonunique solutions. This case, in particular, is a crucial part of enabling autonomous systems to navigate in urban environments. Manmade structures such as buildings, roads, bridges, etc. all contain planar characteristics associated with their geometry. This characteristic also applies especially to aerial imagery at high altitudes where objects on the ground are essentially viewed as planar objects. Therefore, this section describes the planar case to estimating motion from two images of the same scene as shown in Ma et al. [102, 103]. Figure 3-5 depicts the geometry involved with planar homography. The fundamental relationship expressing a point feature in 3D space across a set of images is given through a rigid body transformation shown in Equation 3-49. r12 = Rrll + T (3-49) Recall that rla and rll are relative position vectors describing the same feature point in space with respect to camera 2 and camera 1, respectfully, and R and T are the relative rotation and translation motion between frames. 2.2.1 Localization Localizing the camera position and orientation relative to a stationary surrounding has been addressed using a number of methods. An early method presented by Longuet-Higgins [38, 39] used the coplanarity constraint also known as the epipolar constraint. Meanwhile, the subspace constraint has also been employed to localize camera motion [40]. These techniques have been applied to numerous types of autonomous systems. The mobile robotic community has applied these techniques for the development of navigation in various scenarios [41-45]. The applications have also extended into the research of UAV for aircraft state estimation. Gurfil and Rotstein [46] was the first to extend this application in the framework of a nonlinear aircraft model. This approach used optical flow in conjunction with the subspace constraint to estimate the angular rates of the aircraft and was extended in [47]. Webb et al. [48, 49] employed the epipolar constraint to the aircraft dynamics to obtain vehicle states. The foundation for both of these approaches is a Kalman filter in conjunction with a geometric constraint to estimate the camera motion. Some applications for aircraft state estimation have involved missions for autonomous UAV such as autonomous night landing [50] and road following [51]. 2.2.2 Mapping Location estimation of stationary targets using algorithms such as structure from motion has been extensively researched for non-static cameras with successful results. The foundation of these techniques still rely on the geometric constraints imposed on stationary targets. The decoupling of structure from the motion has been characterized in a number of papers by Soatto et al. [52-58]. These approaches employ the subspace constraint to reconstruct feature point position through an extended Kalman filter. Several survey papers have been published describing the current algorithms while comparing the performance and robustness [59-62]. Robust and adaptive techniques have been proposed that use an adaptive extended Kalman filter to account for model uncertainties [63]. In addition, Qian et al. [64] designed a recursive Hoo filter to estimate structure from motion in the presence of measurement and model uncertainties while b3i Figure 5-2. Camera-fixed coordinate frame Similar to the body-fixed coordinate frame, a transformation can be defined for the mapping between the body-fixed frame, B and the camera frame, I as seen in Equation 5-5 i; by i2. = RBI b2. (5-5) 13 b3B where RBI is the relative rotation between frame B and I, respectfully. This transformation is analogous to the aircraft's roll-pitch-yaw, where now these rotation angles define the roll, pitch and yaw of the camera relative to the aircraft's orientation. The coordinate rotation transformation, RBI, can be decomposed as a sequence of single-axis Euler rotations as seen in Equation 5-6, similar to the body-fixed rotation matrix. The orientation angles of the camera are required to determine the imaging used for vision-based feedback. The roll angle, #c, describes rotation about ;;3, the pitch angle, 8c, describes rotation about 12 and the yaw angle, c,, describes rotation about ii. RBI= l(cOle 2e>le Oc)3c)] (5-6) The matrix RBI in Equation 5-6 will transform a vector in body-fixed coordinates to camera-fixed coordinates. This transformation is required to relate camera measurements to on-board vehicle measurements from inertial sensors. The matrix again depends on the angular 9.2 Controller Development .. . .... .. 118 9.2.1 Altitude Control . .... ... . 18 9.2.2 Heading Control ......... .. .... .. 119 9.2.3 Depth Control . . .. ..... .21 10 SIMULATIONS ............. ..............123 10.1 Example 1: Feature Point Generation ..... .... .. .. 123 10.2 Example 2: Feature Point Uncertainty .... .... . .. 126 10.2.1 Scenario . .. .... .. .26 10.2.2 Optic Flow ......... ... .. .. 128 10.2.3 The Epipolar Constraint . .... .. .. 130 10.2.4 Structure From Motion . ... .. .. 132 10.3 Example 3: Open-loop Ground Vehicle Estimation .. . 133 10.3.1 System Model . ... ..... .. .34 10.3.2 Open-loop Results . .. .. .. .. 135 10.4 Example 4: Closed-loop Aerial Refueling of a UAV .. .. .. .. .. 138 10.4.1 System Model ... . ..... .. .39 10.4.2 Control Tuning . ..... .. . 140 10.4.3 Closed-loop Results . ... .... .14 10.4.4 Uncertainty Analysis ....... ... .. .. 148 11 CONCLUSION ............. ..............151 REFERENCES ......... . ..... .. 154 BIOGRAPHICAL SKETCH ......... .. ... .. 164 The expressions for features points, given in Equation 4-7 and Equation 4-8, can be substituted into Equation 4-11 to introduce uncertainty. The resulting expression in Equation 4-12 separates the known from unknown elements. J = +(4-12) Vog Votv -v A range of variations are allowed for optic flow due to the uncertainty in feature points. The expression for IJ can thus be written using nominal, IJo, and uncertain, Sy, terms as in Equation 4-13 where the uncertainty is bounded by Ay E R. g = {$o+8y 6~ : |Sy| The amount of uncertainty in optic flow depends on the uncertainty in each feature point. The maximum variation in velocity for a given point, determined by rl, is given in Equation 4-14. The actual bounds on the feature points, as noted in Equation 4-9 and Equation 4-10, varies depending on the location of each feature point so bounds of A,, and A,, are given for each vertical component and Av, and Av, are given for each horizontal component. As such, the bound on variation is noted in Equation 4-14 as specific to the rll and rl2 used to gather feature points in each image. Ay= max || (8,,, 8)2+ _6v 8,2 | (4-14) V1? I a,1 4.3 Epipolar Geometry State estimation using epipolar geometry, computed as a solution to Equation 3-44, requires a pin-hole camera whose intrinsic parameters are exactly known. Such a situation is obviously not realistic so the effect of uncertainty can be determined. A non-ideal camera will lose the (F,n 5T,n Feature point location on reference vehicle realtive and expressed in Earth-fixed coordinates Feature point location on target vehicle expressed in Earth-fixed coordinates Variance operator of a vector x Gradient threshold Attitude of aircraft about {$1, 82, ~3) aXeS Attitude of camera about {81l, c^2, 83) aXeS Roll command Heading command Angular rates of aircraft about {$1, 82 -3) aXeS Angular rates of camera about {FIl, c^2, c^3) aXeS Radial distortion uncertainty bound Focal length uncertainty bound Uncertainty bound in the entries of the planar homography matrix Uncertainty bound in the entries of the essential matrix Uncertainty bound in depth components Lateral deviation between vehicle and target Uncertainty bound in optic flow Uncertainty bound in pu Uncertainty bound v Two-view feature point matrix using the planar homography matrix Nominal two-view feature point matrix using the planar homography (#, 6, y) S= (p, q, r) me = (Pc, 4c, re) Ad CHAPTER 8 MODELING TARGET MOTION 8.1 Introduction Once state estimation of a moving target has been obtained the next step is to record these estimates over time to try and leamn the object's general motion. The purpose of understanding these motions are useful for prediction and allows for closed-loop control for applications such as autonomous docking and AAR. In essence, this prediction step provides the tracking vehicle with future state information of the target which assists the controller in both the tracking and docking missions. This chapter describes a probabilistic method that employs the time history estimates of the target's motion to determine future locations. In addition to providing state predictions, the modeling scheme also provides position updates when features are outside the field of view. Linear modeling is not sufficient for prediction in this situation, where the motion is stochastic. Linear techniques that estimate a transfer function, such as ARX, require that the inputs and outputs of the system are known. Although this is the case for many systems, it doesn't apply in this scenario because the inputs (i.e. the forces) on the target are assumed to be unknown. For example, in the AAR mission the target, or drogue, interacts with a flow field that is potentially turbulent due to the effects of the surrounding aircraft (i.e. tanker and receiver) and difficult to model. The drogue is also tethered by a flexible boom that applies reaction forces which are dictated from the tanker aircraft and the aerodynamic forces on the boom. These factors make the modeling task challenging to accurately represent the motion of a general target with unmodeled dynamics and disturbances. Therefore, the method considered in this dissertation will consist of a probabilistic approach to account for general motions with stochastic behavior. 8.2 Dynamic Modeling of an Object There are numerous modeling schemes in the research community. The probabilistic approaches can be separated into two main categories consisting of supervised and unsupervised learning algorithms. Supervised algorithms require training data that determines trends apriori and classifies the the motion under consideration to the trends observed during training. Currently, several universities have a research facility dedicated to the investigation of MAV, including Brigham Young University (BYU), Stanford University, Georgia Institute of Technology, and the University of Florida. The autonomous capabilities demonstrated by BYU incorporated an autopilot system for waypoint navigation that integrated traditional IMU sensors [3, 4]. Meanwhile, Stanford has examined motion planning strategies that optimize flight trajectories to maintain sensor integrity for improved state estimation [5]. The work at Georgia Tech and BYU has considered corporative control of MAV for autonomous formation flying [6] and consensus work for distributed task assignment [7]. Alternatively, vision based control has also been the topic of interest at both Georgia Tech and UF. Control schemes using vision have been demonstrated on platforms such as a helicopter at Georgia Tech [8], while UF implemented a MAV that integrated vision based stabilization into a navigation architecture [9, 10]. The University of Florida has also considered MAV designs that improve the performance and agility of these vehicles through morphing technology [11-13]. Fabrication facilities at UF have enabled rapid construction of design prototypes useful for both morphing and control testing. The fleet of MAV produced by UF are illustrated in Figure 1-3 where the wingspan of these vehicles range from 24 in down to 4 in. Figure 1-3. The UF MAV fleet There are a number of current difficulties associated with MAV due to their size. For example, characterizing their dynamics under flight conditions at such low Reynolds numbers is an extremely challenging task. The consequence of increased agility at this scale also gives rise to erratic behavior and a severe sensitivity to wind gust and other disturbances. Waszak et al. [14] performed wind tunnel experiments on 6 inch MAV and obtained the required stability derivatives for linear and nonlinear simulations. Another critical challenge toward MAV Camera Feature Point Moving Object State Sohsi x(0) = 0Model Tracker Distection Estimation Dermnti LIIIIIImage 13Processing Motion Controller Moc eling Prediction Figure 1-6. Closed-loop block diagram with visual state estimation simple image differencing, where the stationary background is segmented out; however, this approach does not apply to moving cameras. In the case of a moving camera, the background is no longer stationary and it begins to change over time as the vehicle progresses through the environment. Therefore, the images taken by a moving camera contain the motion due to the camera, commonly called ego-motion, and the motion of the object. Techniques that involve camera motion compensation or image registration have been proposed to work well when there exists no stationary objects close to the camera which cause high parallax. This dissertation will establish a technique to classify objects in the field of view as moving or stationary while accounting for stationary objects with high parallax. Therefore, with a series of observations of a particular scene, one can determine which objects are moving in the environment. Knowing which objects are moving in the image dictates the type of image processing required to accurately estimate the object's states. In fact, the estimation problem becomes infeasible for a monocular system when both the camera and the object are moving. This unattainable solution is cause by a number of factors including 1) inability to decouple the motion from the camera and target and 2) failure to triangulate the depth estimate of the object. For this configuration, relative information can be obtained and fused with additional information for state estimation. First, decoupling the motion requires known information regarding motion of the camera or the motion of the object, which could be obtained through other sensors such The homography solution is then decomposed into its rotational and translational components through a similar technique used in the eight-point algorithm. This approach uses SVD to rewrite the homography matrix, as shown in Equation 3-59. HTH = VEVT (3-59) The matrix C = diag [o0a~ 21 -- ] nd ,mthe vetor V,- alt~~re nn otn orma nl eigenvector c~~ol-~rresodn to the singular values of E. The columns of the matrix V can be written as V = [vl, v2, v3]. Defining two other unit-length vectors, shown in Equation 3-60, that are preserved in the homography mapping and will facilitate in the decomposition process. vi +v3 vi-v3 U11= /2. = (3-60) Furthermore, defining the matrices shown in Equation 3-61 will establish a homography solution expressed in terms of these known variables. Ui = [v2,111slT21/2] Wi = [HyI2-Hui~H 2Hul [ ] (3-61) U2 = 7Z2:l 82-1T22 W2 = [Hy2Hu(II2,H 2H2 The four solutions are shown in Table 3-1 in terms of the matrices given in Equations 3-61, 3-60 and the columns of the matrix V. Notice the translation component is estimated up to a 2 scale factor. This is the same scale ambiguity associated with the eight-point algorithm, which is caused by the loss of depth during the image plane transformation. Table 3-1. Solutions for homography decomposition R1 = W1 Uz R3 = R1 Solution 1 NI1 = itul Solution 3 N3 = -NI1 T = (H-R1)Nl ci73 --2T1 R 2 = W2 U2 R 4 = R 2 Solution 2 N2~ = Vi2 Solution 4 N4 q gT2 = (H -R2) N2 I f4 = -gf2 A unique solution for the homography is then found by imposing the positive depth constraint, which is associated with the physically possible solution. This imposition involves * A new approach to detecting moving objects in a sequence of images is developed. This method computes estimates for the focus of expansion (FOE) and then classifies each feature point into their respective motions through an iterative least-squares solution. The decision scheme for classification maintains a cost function, which determines if a feature point obeys a particular FOE, under a desired threshold. The dominant motion assumption is then used to determine which FOE class is considered stationary objects in the environment and which are associated with moving objects. * The nonlinear dynamics for an aircraft-camera system are derived for a general camera configuration and model. This structure allows multiple cameras with time varying positions and orientations within the derivation to compute image plane quantities such as feature point position and velocity. * A new method for obtaining error bounds on the target state is established to provide a region of where the estimate can lie from the effects of uncertainty. This method can be described as a deterministic framework that computes upper bound uncertainty and was implemented to describe variations to image plane coordinates and propagated through vision based algorithms. Although this upper bound or worse-case approach to uncertainty is a conservative technique, it provides a fast implementation scheme to account for inaccurate camera calibration. * The implementation of the homography of a moving target along with a model prediction scheme will be incorporated into a controls framework to enable closed-loop tracking of an unknown moving object of interest. The first chapter of this thesis describes the motivation for this research, some current objectives and limitations to address followed by a summary of the contribution and descriptions of potential applications for this research. Chapter 2 describes the related work and literature review that applies to this particular research topic. Chapter 3 introduces the foundation of computer vision and image processing. First the camera geometry is described along with the projection model followed by the constraints used to facilitate the estimation process. Lastly, traditional algorithms which estimate both the 3D motion of the camera and the motion of targets are described. Chapter 4 quantifies the effects of uncertainty in state estimation from variations in feature point position caused from camera calibration and feature point tracking. -1 )( Time (sec) Time (sec) A B Time (sec) Figure 10-16. Attitude states of the reference vehicle (pursuit vehicle): A) Roll, B) Pitch, and C) Yaw Time (sec) A Figure 10-17. Position states o Down I 100( 80( Q 20( 0 20 40 60 Time (sec) I 1000( 15 Time (sec) f the target vehicle (chase vehicle): A) North, B) East, and C) 10 20 40 60 10 Time (sec) 20 40 60 0 Time (sec) 20 40 Time (sec) Figure 10-18. Attitude states of the target vehicle (chase vehicle): A) Roll, B) Pitch, and C) Yaw motion from the UAV to the target of interest. The norm error of this motion are depicted in Figure 10-19. These results indicate that with synthetic images and perfect tracking of the vehicles nearly perfect motion can be extracted. Once noise in the image or tracking is introduced the estimates of the target deteriorate quickly even with minute noise. In addition, image artifacts such as interference and drop outs will also have an adverse affect on homography estimation. 136 'Yv Vertical angle for field of view 87 A variation in focal length 6d A variation in radial distortion 8,, A variation in pu 6v A variation in v 8 y A variation in optic flow Sc A variation in the two-view feature point matrix Sq A variation in the entries of the essential matrix Sq, A variation in the two-view feature point matrix using the planar homography matrix Sh A variation in the entries of the planar homography matrix 6A A variation in the two-view feature point matrix using structure from motion 8: A variation to the depth components in two-view camera geometry rl Position vector of feature point relative to and expressed in camera coordinate frame I rlF,n Feature point location on reference vehicle realtive and expressed in camera-fixed coordinates grl,, Feature point location on target vehicle realtive and expressed in camera-fixed coordinates TIVF,n Feature point location on reference vehicle realtive and expressed in virtual coordinates TIV,,, Feature point location on target vehicle realtive and expressed in virtual coordinates pu Vertical coordinate in the image plane pu (x) Mean operator of a vector x LIST OF TERMS Acceleration of the target in E Body-fixed coordinate frame components Position vector of camera center in camera-fixed coordinated frame Radial distortion Nominal radial distortion Earth-fixed coordinate frame components Focal length Nominal focal length Altitude state Stacked column vector of the entries of the planar homography matrix Altitude command Nominal entries of the planar homography matrix Image motion model Camera-fixed coordinate frame components Proportional gain on altitude error Proportional gain on pitch rate Proportional gain to roll rate Proportional gain on the lateral position error Integral gain on the lateral position error Proportional gain on pitch Proportional gain to roll Proportional gain to heading error Epipolar line in image i Translation from camera-fixed to reference-fixed coordinates expressed relative camera-fixed coordinates a (t) d do {&1l, e2, 83} f fo h h he ho h(x) k kg k, kY, kyi ke kg li mlF The outer-loop that connects altitude to pitch commands is considered. The gains for the inner-loop pitch tracking remained the same while the gain in altitude error was set to k = 1.25. The final compensation filter is given in Equation 10-1 and was designed in Stevens et al. [110]. A step response for this controller is illustrated in Figure 10-25 that shows a steady climb with no overshoot and a steady-state error of 2 ft. This response is realistic for an F-16 but not ideal for autonomous refueling mission where tolerances are on the cm level. The altitude transition is slow due to the compensator but one may consider more aggressive maneuvers for missions such as target tracking that may require additional agility. s2 + 0.35s +t 0.015 Gs2+-t2.41s+-t0.024 (01 2450 -Response Command 2350 2250 2200 2150 0 20 40 60 80 100 Time (sec) Figure 10-25. Altitude step response The next stage that was tuned in the control design was the heading controller. The inner-loop gains were chosen to be kg = -5.7 and kp = -1.6 for the roll tracker. The bode diagram for this controller of roll command to roll angle is shown in Figure 10-26 which shows attenuation in the lower frequency range. This attenuation removes any high frequency response from the aircraft which is desired during a refueling mission, especially in close proximity. Meanwhile, the coupling between lateral and longitudinal states during a turn was counteracted is related to q as in Equation 3-44. The matricx C, shown in Equation 3-45, is a nx 9 matrix of stacked feature points matched between two views. Cq = 0 (3-44) #1l,192,1 V1,19u2,1 #u2,1 91l,192,1 V1,192,1 V2,l #1l,1 V1,1 1 u1,2iu2,2 V1,2iu2,2 #u2,2 #u1,2V2,2 V1,2V2,2 V2,2 #u1,2 V1,2 1 3-5 i#1,niU2,n V1,niU2,n iU2,n i#1,nV2,n V1,nV2,n V2,n i#1,n V1,n 1 A unique solution for Equation 3-44 exists using a linear least-squares approach only if the number of matched features in each frame is at least 8 such that rank(C) = 8. Additionally, more feature points will obviously generate more constraints and, presumably, increase accuracy of the solution due to the residuals of the least-squares. In practice, the least-squares solution to Equation 3-44 will not exist due to noise, therefore, a minimization is used to find an estimate of the essential matrix, as shown in Equation 3-46. min||Cq||, ||q|| =1 (3-46) Once an estimate of the essential matrix is found, the next step is to decompose this matrix into its translational and rotational components. This decomposition is obtained through singular value decomposition (SVD) of the essential matrix, and is shown in Equation 3-47. Q = UEV' (3-47) where E = diag {ol, 2, 03 } are the singular values. In general, this solution is corrupted from noise and needs to be projected onto the essential space. This projection is performed by normalizing the singular values to C = diag {1, 1,0} and adjusting the corresponding U and V. The motion decomposition can now be obtained through Equation 3-48, where the translation T is found up to a scaling factor. These four solutions, which consist of all possible combinations of R and Tx, are checked to verify which combination generates a positive depth in terms of the relative position with a lens offset, c, relative to the camera frame. p = x x (3-5 If the origin of the camera frame is placed at the lens, (i.e., c = 0), Equations 3-5 and 3-6 reduce to the very common pin-hole camera model and is represented by Equations 3-7 and 3-8. p = f Ex(3-7) v =f (3-8) This projection is commonly written as a map H: H : RW3 ,2; X x (3-9) The ideal perspective projection given in Equations 3-7 and 3-8 can be expressed in homogeneous coordinates and is shown in Equation 3-10. pu f 0 0 rlx Ezv =0 f 0 Try (3-10) 1 0 0 1 rlz 3.2.2 Intrinsic Parameters The image plane that is acquired from physical cameras is more complicated than the ideal projection given in Equation 3-10. First, the image plane is discretized into a set of pixels, corresponding to the resolution of the camera. This discretization is based on scale factors that relate real-world length measures into pixel units for both the horizontal and vertical directions. These scaling terms are defined as s, and sv which have units of pixels per length, where the length could be in feet or meters. In general, these terms are different but when the pixels are square then s, = sy. Second, the origin of the image plane is translated from the center of the (R, T) Figure 3-5. Geometry of the planar homography If an assumption is made that the feature points are contained on the same plane, then a new constraint involving the normal vector can be established. Denote N = [121,122,13 T as the normal vector of the plane containing the feature points relative to camera frame 1. Then the projection onto the unit normal is shown in Equation 3-50, where D is the projected distance to the plane. N rll = nlrl,x+/t 2291,y ft l23T1,z= (3-50) Substituting Equation 3-50 into Equation 3-49 results in Equation 3-51, 82=R+TNI qir (3-51) where the planar homography matrix is defined to be the following H = R t _TNT (3-52) by an aileron-elevator connect. This connection involved a proportional gain of k, = 0.35 multiplied to the roll angle and added to the elevator position. Bode Diagram Gm = 15.9 dB (at 43.4 rad/sec) Pm = 179 deg (at 0.0583 rad/sec) From Bankcmd(pt 1) To r2d(pt 1) 50 r -100 -150 a, -90 -270 10-4 10-2 100 102 Frequency (rad/sec) Figure 10-26. Inner-loop roll to roll command Bode plot The step response for this bank controller is illustrated in Figure 10-27. The tracking performance is acceptable based on a rise time of 0.25 see, an overshoot of 6% and less than a 3% steady-state error. The outer-loop tuning for heading controller consisted of first tuning the gain on heading error. A gain of kw = 1.5 was chosen for this mission which demonstrated acceptable performance. Figure 10-28 shows the heading response using this controller for a right turn. The response reveal a steady rise time, no overshoot, and a steady-state error of less than 2 deg. Finally, the loop pertaining to lateral deviation was tuned to k,, = 0.5 and kyi = 0.025 which produced reasonable tracking and steady error for lateral position. The final stage of the controller involves the axial position. This stage was designed to increase thrust based on a velocity command once the lateral and altitude states were aligned. A proportional gain was tuned based on velocity error to achieve a slow steady approach speed image relative to an aircraft and then employing the moving object detection algorithm shown in Chapter 6. Once moving objects in the image are detected, the homography estimation algorithm proposed in this chapter is implemented for target state estimation. 7.2 State Estimation 7.2.1 System Description The system described in this paper consists of three independently moving vehicles or objects containing 6-DOF motion. To describe the motion of these vehicles a Euclidean space is defined with five orthonormal coordinate frames. The first frame is an Earth-fixed inertial frame, denoted as E, which represents the global coordinate frame. The remaining four coordinate frames are moving frames attached to the vehicles. The first vehicle contains two coordinate frames, denoted as B and I, to represent the vehicle's body frame and camera frame, as described in Chapter 5 in Figure 5-1. This vehicle is referred to as the chase vehicle and is instrumented with an on-board camera and GPS/IMU sensors for position and orientation. The second vehicle, denoted as F, is considered a reference vehicle that also contains GPS/IMU sensors and provides its states to the chase vehicle through a communication link. Lastly, the third vehicle, denoted as T, is the target vehicle of interest in which unknown state information is to be estimated. In addition, a fictitious coordinate frame will be used to facilitate the estimation process and is defined as the virtual coordinate system, V. The coordinates of this system are related through transformations containing both rotational and translational components. The rotational component is established using a sequence of Euler rotations in terms of the orientation angles to map one frame into another. Let the relative rotation matrices REB, RBI, REF, REV, RIV, RFV, RTy and RET denote the rotation from E to B, B to I, E to F, E to V, I to V, F to V, T to V, and E to T. Secondly, the translations are defined as TEB, F, XV, XT, F,n, T,n, TBI, XIV, mFI mIT, TF,n N1T,n, W1VF, FVT, TVF,n, and rlvr,n which denote the respective translations from E to B, E to F, E to V, E to T, E to the 12th feature point on the reference vehicle and target vehicles all expressed in E, B to I expressed in B, I to V, I to F, I to T, I to the 12th feature point on the reference and target vehicles expressed in I, V to F, V CHAPTER 3 IMAGE PROCESSING AND COMPUTER VISION Image processing and computer vision refers to the process of acquiring and interpreting 2-dimensional visual data to achieve awareness of the surrounding environment. This information is used to infer spatial properties of the environment that are necessary to perform essential tasks such as guidance and navigation through unfamiliar environments. An important breakthrough in computer vision occurred when algorithms were able to detect, track, and estimate locations of features in the environment. This dissertation relies on feature points as the foundation for any vision-based feedback. The term "features" allows one to establish a relationship between the scene geometry and the measured image. These points generally correlate to items in the environment of special significance. Some examples of items that often constitute feature points are corners, edges and light sources. Such feature points can provide information about the overall object in the sense that a set of corners can outline a building. Feature points do not necessarily provide enough information to completely describe an environment but, in practice, they usually provide sufficient information for target tracking and position estimation. To understand the algorithms that use feature points, an establishment of the fundamental equations governed by the physics of a camera will be described. 3.1 Camera Geometry A camera effectively maps the 3-dimensional environment onto a 2-dimensional image plane. This image plane is defined as the plane normal to the camera's central axis located a focal length, f, away from the origin of the camera basis. The geometry provided by a pin-hole camera lens is described in Figure 3-1. The vector, rl, represents the vector between the camera and a feature point in the environment relative to a defined camera-fixed coordinate system, as defined by I. This vector and its components are represented in Equation 3-1. For feature points that are stationary in the environment, the translational optic flow induced by the camera motion is constraint to radial lines emanating from the FOE, as shown in Figure 6-2. Consequently, feature points that violate this condition can be classified as independently moving objects. This characteristic observed from static features will be the basis for the classification scheme. 1 r c r r I ~JI r I I r ,r I ( I I ~ r'r r *~ 2- ~, r /* ir I 1. r E;OE J' Figure 6-2. FOE constraint on translational optic flow for static feature points The residual optical flow may contain independently moving objects within the environment that radiate from their own FOE. An example of a simple scenario is illustrated in Figure 6-3 for a single moving object on the left and a simulation with synthetic data of two moving vehicles on the right. Notice the two probable FOEs in picture on the left, one pertaining to the static environment and the other describing the moving object. In addition, the epipolar lines of the two distinct FOEs intersect at discrete points in the image. These properties of moving objects are also verified in the synthetic data shown in the plot on the right. Thus, a classification scheme must be designed to handle these scenarios to detect independently moving objects. The next The modeling scheme presented in Chapter 8 provides a method to estimate targets in Euclidean space when features do exit the image. This method works well for short periods of time after the target has left; however, the trust in the predicted value degrades tremendously as time increases. Consequently, when a feature leaves the image the controller can rely on the predicted estimates to steer the aircraft initially but may resort to alternative approaches beyond a specified time. As a last resort, the controller can command the aircraft to slow down and regain a broader perspective of the scene to recapture the target. differences between the axes in each coordinate system and the sequence of single-axis rotations. In particular, the rotation order used for this transformation was a 3-2-1 sequence. cos(e,) cos(Ve) sin(Qc) sin(ec) cos(Ve,) cos(Qc) sin(Ve,) cos(Qc) sin(ec) cos(Ve,) + sin(Qc) sin(Ve,) RBI = cos(8c) sin(Vec) sin(#c ) sin(8c) sin(Ve,) + cos(#c) cos(Ve,) cos(#c) sin(8c) sin(Ve) sin(#c) cos(Ve,) sin(ec) sin(Qc) cos(6c) cos(Qc) cos(6c) (5-7) The rates of change of these orientation angles are again required for coordinate frame transformations. The roll rate, pe, is the angular velocity about 13, the pitch rate, q,, describes rotation about i2, and the yaw rate, re, described rotation about ii. The vector, me, is given in Eq. 5-8 to represent these angles. OWe = rcil +t qc 2 tPc 3 (5-8) 5.2 System Geometry The fundamental scenario involves an aircraft-mounted camera and a feature point in the environment. This scenario, as outlined in Figure 5-3, thus relates the camera and the aircraft to the feature point along with some inertial origin. 2, ~ ;;;;~e Feature Point Figure 5-3. Scenario for vision-based feedback The bound on error, Az, can be expressed using Equation 4-29. This bound notes that the bound on variations in feature points, and ultimately the bound on solutions to structure from motion, depends on the location of those feature points. Az n || (Ao + BA)-1( T -(Ao + BA) o) || (4-29) m~ax Is1, P la 8#2 I < 2, test the system under practical conditions. Additionally, incorporating the modeling scheme presented in Chapter 8 into the refueling simulation will help the controller by providing state estimate when the target exits the field of view. Similarly to the previous example, vision-based feedback is generated using a flight simulation. The overall setup of this example is the same where a nonlinear model of an F-16 is used to fly through a cluttered environment while capturing images from an on-board camera. Camera settings, such as focal length and field of view, are kept the same from the previous example. The actual environment has been normalized based on the aircraft velocity so units are not presented. A set of obstacles, each with a feature point, are randomly distributed throughout the environment and are not the same same as the previous example. This environment is shown in Figure 10-4 along with a pair of points indicating the locations at which images will be captured. The aircraft is initially straight and level then translates forward while rolling 4.0 deg and yawing 1.5 degp at the final location. 100 *. * -10 00* ** o " -1000 *0 0 00 50 2 0 ** Figure 10-4. Virtual environment of obstacles (solid circles) and imaging locations (open circles) A) 3D view and B) top view A single camera is simulated at the center of gravity of the aircraft with line of sight aligned to the nose of the aircraft. The intrinsic parameters are chosen such that fo = 1.0 and do = 0.0 for the nominal values. The images for the nominal camera associated with the scenario in Figure 10-4 are presented in Figure 10-5 to show the variation between frames. The vision-based feedback is computed for a set of perturbed cameras. These perturbations range as 87 E [-0.2, 0.2] and 6d E [-0.02, 0.02]. Obviously the feature points in Figure 10-5 will vary as the camera parameters are perturbed. The amount of variation will depend on the feature angles of the aircraft, (#, 6,11). The velocity of the aircraft's center of mass is vb and is defined in Equation 5-27. As stated in Equation 5-27, the aircraft's velocity is expressed in the body-fixed coordinate frame. Each of these parameters will appear explicitly in the aircraft- camera equations. vb = ub1 t +vb2+ twb3 (5-27) The first six equations represent the force and moment equations, while the remaining equations are kinematic relationships. The aerodynamic parameters consist of both the aerodynamic forces, {K,, F, FZ}", on the aircraft and the aerodynamic moments, {L,1M, N}", which are all contained in the force and moment equations. Although these equations do not contain control inputs explicitly, the aerodynamic parameters are directly effected by the position of the control surfaces on the aircraft. In other words, when the control surface deflections are changed the flow over that surface also changes. This flow change over a surface results in changes of the aerodynamic forces, such as lift and drag, which directly produce forces and moments that roll, pitch, and yaw the aircraft and are described by the stability derivatives for each aircraft. Therefore, controlled maneuvers are accomplished by changing these aerodynamic parameters through the control surfaces. An alternative approach to solving the nonlinear equations is to linearize these equations about a trim condition using a Taylor series expansion. By linearizing these equations about a level flight condition, the aircraft equations become decoupled into two planar motions. This set of equations, although easy to solve, have limitations outside the chosen trim state, especially for smaller more maneuverable aircraft. The choice of what set of aircraft equations to use depends primarily on the aircraft and the application. 5.4 Aircraft-Camera System The preliminary definitions established in the previous sections will now be used to formulate the aircraft-camera system by using the systems described in this chapter. Here the dependence of image plane position and velocity on the aircraft states along with the kinematic 3.6 Two-View Image Geometry ........ .. .. 56 3.6.1 Epipolar Constraint ........ .. .. 57 3.6.2 Eight-Point Algorithm ....... .. .. 59 3.6.3 Planar Homography .. . .... .. 61 3.6.4 Structure from Motion . ...... ... .. 65 4 EFFECTS ON STATE ESTIMATION FROM VISION UNCERTAINTY .. .. .. 67 4. 1 Feature Points ......... . .. .. 67 4.2 Optical Flow ......... ... .. 70 4.3 Epipolar Geometry ......... . .. .. 71 4.4 Homography ........ . ... .. 73 4.5 Structure From Motion ....... ... .. 75 5 SYSTEM DYNAMICS ......... . ... .. 77 5.1 Dyanmic States ......... . ... .. 77 5.1.1 Aircraft ............. ........... 77 5.1.2 Camera ............. ........... 79 5.2 System Geometry ......... . ... .. 81 5.3 Nonlinear Aircraft Equations ....... .. .. 83 5.4 Aircraft-Camera System ......... ... .. 84 5.4.1 Feature Point Position ....... .. .. 85 5.4.2 Feature Point Velocity ..... ... .. 85 5.5 System Formulation ........ .... .. 86 5.6 Simulating ....._.. . ... .. 89 6 DISCERNING MOVING TARGET FROM STATIONARY TARGETS .. .. .. 90 6. 1 Camera Motion Compensation . ...... ... .. 90 6.2 Classification ......... . ... .. 95 7 HOMOGRAPHY APPROACH TO MOVING TARGETS ... .. .. .. .. 98 7.1 Introduction ......... . ... .. 98 7.2 State Estimation . . ...... .01 7.2.1 System Description .............10 7.2.2 Homography Estimation .............10 8 MODELING TARGET MOTION.............11 8.1 Introduction .............. ............ 111 8.2 Dynamic Modeling of an Object.............11 8.2.1 Motion Models .............12 8.2.2 Stochastic Prediction .............13 9 CONTROL DESIGN ..............17 9. 1 Control Objectives .............17 subject to Equation 3-22. One important limitation of this criterion occurs when the window in both images contains relatively constant intensity values. This results in the aperture problem where a number of solutions for h are obtained. Therefore, during the feature selection process it's beneficial to choose features that contain unique information in this window. hi=argm~in ||li(i-I2((hSi))|2 (3-23) There are two common techniques to solve Equation 3-23 for small baseline tracking: (1) using the brightness consistency constraint and (2) applying the sum of squared differences (SSD) approach. Each of these techniques employs a translational model to describe the image motion. Therefore, if one assumes a simple translational model then the general transformation is shown in Equation 3-24. h(x) = x +t Ax (3-24) The brightness consistency constraint is derived by substituting Equation 3-24 into Equation 3-22 while initially neglecting the noise term. Applying the Taylor series expansion to this expression about the point of interest, x, while retaining only the first term in the series results in Equation 3-25. aI dpu aI dv aI aSp dt av dt tat (-5 This equation relates the spatial-temporal gradients to the pixel motion assuming the brightness remains constant across images. Rewrting Equation 3-25 in matrix form results in Equation 3-26. AITu +t It = 0 (3-26) where u = [f ]. Equation 3-26 constitutes 1 equation with 2 unknown velocities; therefore, another constraint is needed to solve this problem. A unique solution for the velocities can be determined by enforcing an additional constraint on the problem, which entails restraining regions to a local window that moves at constant velocity. Upon these assumption one can minimize the error -Response - -Command U' 0 5 10 15 2( Time (sec) step response 120 --Response --Command 100 80 60 40 20 20 ",15 av d 10 5 Figure 10-27. Roll angle 0 20 40 60 Time (sec) 80 100 Figure 10-28. Heading response to the target. A gain of ks = 3.5 was determined for this loop which generates the desired approach. Lastly, to help limit the number of times the feature points exit the field of view a limit was imposed on the pitch angle. This limit was enforced when the approach achieve a specified distance. For this example, the distance was set to within 75 ft in the axial position of the body-fixed frame which was determined experimentally from the target's size. 10.4.3 Closed-loop Results The state estimation performance of the target drogue during this simulation was similar to the previous simulation regarding the tracking of a ground vehicle. The estimated target states CHAPTER 9 CONTROL DESIGN The control strategy considered in this dissertation uses the computed relative states found between a moving camera and a moving target of interest as shown in Chapter 7. Effectively, these quantities are the error signals used for control to track the moving camera toward a desired location based on the motion of the target. The framework presented here will use aircraft and UAV navigation schemes for the aerial missions described in Chapter 1. Therefore, the control design described in this chapter focuses on the homing mission to facilitate the AAR problem, which involves tracking the position states computed from the homography. Various types of guidance controllers can be implemented for these types of task once the relative position and orientation are known. Depending on the control objectives and how fast the dynamics of the moving target are, low pass filtering or a low gain controller may be required to avoid high rate commands to the aircraft. In the AAR problem, the success of the docking controller will directly rely on several components. The first component is the accuracy of estimated target location which during AAR needs to precise. Secondly, the dynamics of the drogue are stochastic. This causes the modeling task to be impractical in replicating real life so the controller is limited to the models considered in the design. In addition, the drogue's dynamics may not be dynamically feasible for the aircraft to track which may further reduce performance. Lastly, the controller ideally should make position maneuvers in stages by considering the altitude as one stage, the lateral position as another stage, and the depth position as the final stage. In close proximity, the controller should implement only small maneuvers to help maintain the vehicles in the FOV. 9.1 Control Objectives The control objectives for the AAR mission is to track and home on the target drogue and successfully dock with the receptacle. This controller is designed using a tracking methodology that regulates the relative distance to within a specified tolerance. For example, the tolerance required for aerial refueling is on the centimeter scale [15]. For a stationary feature point in space, the position vector, 5, is constant in magnitude and direction and is expressed in the inertial frame; therefore, this time derivative is zero. Likewise, the position vector of the aircraft's center of mass, TEB, is also expressed in the inertial basis and, therefore, the time derivative just becomes ~TEB. Meanwhile, the Derivative Theorem is employed on such terms as rl and TBI to express these terms in the moving frame. By applying this theorem and solving for feature point velocity with respect to the camera frame, Equation 5-29 can now be rewritten to Equation 5-30 for a non-stationary feature point. 'd(r) = ( ~TEB Bd(TBI) a~x TBI Eo Ix r (5-30) dt dt This equation can be reduced further if the cameras are constrained to have no translation relative to the aircraft so 3(TEI) = 0. Alternatively, this term is retained in the derivation to allow this degree of freedom in the camera setup. The angular velocity, E I, can be further decomposed using the Addition Theorem. The final step implements Equations 5-5 and 5-13 to transform each term into the camera frame. After some manipulation, the expression for the velocity of a feature point relative to the camera frame results in Equation 5-31. Ti = RBIREB ( TEBR RBITBI RBI( (x TBI) ((RBIO+ Wc) x 8) (5-31) The image plane velocity of a feature point relative to the camera frame is finalized by substituting both equations for position and velocity derived in Equation 5-28 and 5-31 into Equations 3-32 and 3-33. This result will provide a description of the optical flow for each feature point formed by either the camera traveling through the environment or the motion of the feature points themselves. To incorporate radial distortion effects into the optic flow computation requires the additional substitution into Equations 3-34 and 3-35. 5.5 System Formulation The derivation of the aircraft-camera equations can be easily extended to systems with multiple cameras all of which have their own position and orientation relative to the aircraft while acquiring numerous feature points in each image. Although this adds computational complexity, as GPS and IMUs. Second, the depth estimate can be acquired if some information is known regarding the target geometry (e.g. a fixed distance on the target). For the case of stereo vision, depth estimates can be obtained for each time step which is suitable for estimating the states of a moving object. Although this particular configuration addresses the depth estimation, additional issues involving the correspondence solution emerge when introducing multiple cameras [5]. Furthermore, the accuracy of the state estimates becomes poor for small baseline configurations, which occurs for MAV using stereo vision. These issues regarding target state estimation will be considered in this dissertation to show both the capabilities and limitations toward autonomous control and navigation. Another important task involved with target estimation is to determine a pattern (if any) in the object's motion based on the time history. The objects can then be classified into deterministic and stochastic motions according to past behavior. With this information, prediction models can be made based on previous images to estimate the position of an object at a later time with some level of confidence. The predicted estimates can then be used in feedback for tracking or docking purposes. For stochasticly classified objects, further concerns regarding docking or AAR are imposed on the control problem. The primary task of state estimation, for both the vehicle and objects in the environment, relies on accurate knowledge of the image measurements and the associated camera. Such knowledge is difficult to obtain due to uncertainties in these measurements and the internal components of the camera itself. For instance, the image measurements contain uncertainties associated with the detection of objects in the image, in addition to noise corruption. These drawbacks have prompted many robust algorithms to increase the accuracy of feature detection while handling noise during the estimation process. Alternatively, many techniques have been used to accurately estimate the internal parameters of the camera through calibration. The parameters that describe the internal components of the camera are referred to as intrinsic parameters and typically consist of focal length, radial distortion, skew factor, pixel size, and optical center. This calibration process can become cumbersome for a large number of cameras 3.2.4 Radial Distortion Other nonlinear camera effects that are not accounted for in the pin-hole model, such as radial distortion, can be addressed through additional terms. A standard lens distortion model is considered to account for such nonlinearities in the camera. The general distortion term, given in Equation 3-14, requires an infinite series of terms to approximate the value. d = dr r2+-td2r4+-td3r,6+--- HOT (3-14) The distortion model, shown in Equations 3-15 and 3-16, maps an undistorted image, (p', v'), which is not measurable on a physical camera, into a distorted image, (p'd d&), which is observable [104]. This distortion model only considers the first term in the infinite series to describe radial distortion and excludes tangential distortion. This approximation in distortion has been used to generate an accurate description of real cameras without additional terms [105], p'd = v'(1+t dr2) (3-15) vid = p'(1+ dr2) (3-16) where r2 __ /1 C1)2 /t _V 2)2 and d is the radial distortion parameter of the camera. Assuming the origin of the camera frame is placed at the lens, then this term becomes r2 __ 1u2 +t v'2. In addition, the radial distortion parameter, d, which is not described in Figure 3-1, attempts to model the curvature of the lens during the image plane mapping. This distortion in the image plane varies in a nonlinear fashion based on position. This effect demonstrates an axisymmetric mapping that increases radially from the image center. An example can be seen in Figure 3-3B and 3-3C which illustrates how radial distortion changes feature point locations of a fixed pattern in the image by comparing it to a typical pin-hole model shown in Figure 3-3A. Notice the distorted images seem to take on a convex or concave shape depending on the sign of the distortion. vector [102]. The same constraint holds for the image coordinates as well but also introduces an unknown scale factor. Employing this constraint, estimates of relative motion can be acquired for both camera-in-hand and fixed camera configurations. This dissertation deals with the camera-in-hand configuration while assuming a perfect feature point detection and tracking algorithm. This assumption enables the performance of the vision based state estimation to be tested before introducing measurement errors and noise. The homography constraint requires a few assumptions based on the quantity and the structure of the feature points. The algorithm first requires a minimum of four planar feature points to describe each vehicle. This requirement enables a unique solution to the homography equation based on the number of unknown quantities. The reference vehicle will have a minimum of four pixel values in each image which will be defined as pF,n = CUF,n-,VF,n] Vn2 feature points. Likewise, the target vehicle will have four pixel values and will be defined as pr,n = [PT,n, VT,n] Vn2 feature points. This array of feature point positions are computed at 30 Hz which is typical for standard cameras and the frame count is denoted by i. The final requirement is a known distance for both the reference and target vehicle. One distance represents the position vector to a feature on the reference vehicle in Euclidean space relative to the local frame F and the second distance represents the position vector to a feature on the target vehicle in Euclidean space relative to the local frame T. In addition, the length of these vectors also must be equal which allows the unknown scale factor to be determined. The vector describing the reference feature point will be denoted aS SF expressed in F, while the vector describing the target feature point is referred to as sT expressed in T. These feature point position vectors are also illustrated in Figure 7-2. The feature points are first represented by position vectors relative to the camera frame, I. The expressions for both the reference and target feature points are given in Equations 7-3 and 7-4. These vector components are then used to compute the image coordinates given in Equations 7-1 and 7-2. The computation in Equation 7-4 requires information regarding the target which is done solely to produce image measurements that normally would be obtained from the sensor. Remaining computations, regarding the homography, will only use sensor myr Translation from camera-fixed to target-fixed coordinates expressed relative camera-fixed coordinates mVF Translation from virtual to refemce-fixed coordinates expressed relative virtual coordinates my, Translation from virtual to target-fixed coordinates expressed relative virtual coordinates op Vertical image offset from center to upper left corner in pixel units ov Horizontal image offset from center to upper left corner in pixel units p (t) Position of the target in E pVF Image coordinates in the virtual camera of the reference vehicle pyr Image coordinates in the virtual camera of the target vehicle q Stacked column vector of the entries of the essential matrix qo Nominal entries of the essential matrix sqc Vertical unit length to pixel scaling sv Horizontal unit length to pixel scaling so Image skew factor u Time rate of change of (p, v) v (t) Velocity of the target in E Vb = (u, v, w) Velocity of the body-fixed frame (velocity of the aircraft in body-fixed coordinates) vc = (uc, Ve, Wc) Velocity of the camera-fixed frame along {$1,82, 3 } axes w (t) Random vector I Subset image specified by W xrv Translation from camera-fixed to virtual coordinates expressed in camera-fixed coordinates 0.4 ~ '* 0.4 . 0.2 0.2 -0.4~ 0.4 -0.6~ 0.6 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 v v A B Figure 10-5. Feature points for A) initial and B) final images point, as noted in Equations 4-9 and 4-10, but the effect can be normalized. The variation in feature point given nominal values of Po = vo = 1 is shown in Figure 10-6 for variation in both focal length and radial distortion. This surface can be scaled accordingly to consider the variation at other feature points. The perturbed surface shown in Figure 10-6 is propagated through three main image processing techniques for analysis. d 2 0. 10.2.2 Otic Flo Figure 10-6 t foceral lngth and ftrada doistorin ersnaiecmaio fotcfo o the nominal camera and a set of perturbed cameras is shown in Figure 10-7. image is a feature point which indicates some pixel of particular interest due to, for example, color or intensity gradient near that pixel. These intensity variations correlate well to physical features in the environment such as comers and edges which describe the character of buildings and vehicles within a scene as described in Chapter 3. Among the techniques that utilize feature points, the approach related to this paper involves epipolar geometry [39, 112]. The purpose of this technique is to estimate relative motion based on a set of pixel locations. This relative motion can describe either motion of the camera between two images or the relative distance of two objects of the same size from a single image. The 3D scene reconstruction of a moving target can be determined from the epipolar geometry through the homography approach described in Chapter 3. For the case described in this chapter, a moving camera attached to a vehicle observes a known moving reference object along with an unknown moving target object. The goal is to employ a homography vision-based approach to estimate the relative pose and translation between the two objects. Therefore, a combination of vision and traditional sensors such as a global positioning system (GPS) and an inertial measurement unit (IMU) are required to facilitate this problem for a single camera configuration. For example in the AAR case, GPS and IMU measurements are available for both the receiver and tanker aircraft. In general, a single moving camera alone is unable to reconstruct the 3D scene containing moving objects. This restriction is due to the loss of the epipolar constraint, where the plane formed by the position vectors relative to two camera positions in time to a point of interest and the translation vector is no longer valid. Techniques have been formulated to reconstruct moving objects viewed by a moving camera with various constraints [35, 113-116]. For instance, a homography based method that segments background from moving objects and reconstructs the target's motion has been achieved [117]. Their reconstruction is done by computing a virtual camera which fixes the target's position in the image and decomposes the homography solution into motion of the camera and motion caused by the target. This decomposition is done using a planar translation constraint which restricts the target's motion to a ground plane. Similarly, Han can be used. First, two solutions can be eliminated by using the positive depth constraint. The decision regarding the remaining two solutions is more difficult to decipher unless the normal vector is known or can be estimated, which in this case is known. Recall the normal vector, n, describes the plane containing the feature points of the reference vehicle. As a result, the homography solution is determined uniquely. The final step in this development is to use the homography solution to solve for the relative translation and rotation from T to B. The resulting equation for the rotation uses a sequence of transformations and is shown in Equation 7-31. RTB = RBIREBRETVRRBI (i 1) REB (i 1) RETF (i 1) (7-31) The translation is found through a series of scalings followed by a vector sum. The relative translation, xh, is first multiplied by D to scale distance which is given in Equation 7-29 to obtain x. Secondly, x is then divided by oc to scale the depth ratio resulting in the final x expressed in I. This result in conjunction with R is then used in Equation 7-22 to solve for myr. The next step is to compute the relative translation from I to V which is given in Equation 7-32. my=RE V EB+ETBTI (7-32) The relative translation from T to B is then given in Equation 7-33. xTB = REBRETV (mvT mIV) (7-33) In conclusion, Equations 7-31 and 7-33 represent the relative motion between the camera vehicle and the target vehicle. This information is valuable for the control tasks described earlier involving both tracking and homing applications. The next section will implement this algorithm in simulation to verify the state estimator for the noise free case. REFERENCES [1] Secretary of Defense, "Unmanned Aircraft Systems Roadmap 2005-2030," website: http:/uay. navair:navy, mil/roadmap05/roadmap. htm [2] Grasmeyer, J. M., and Keennon, M. T., "Development of the Black Widow Micro-Air Vehicle," 39th Aerospace Sciences 1Meeting and Exhibit, AIAA 2001-0127, Reno, NV, January 2001. [3] Beard, R., Kingston, D., Quigley, M., Snyder, D., Christiansen, R., Johnson, W., Mclain, T., and Goodrich, M., "Autonomous Vehicle Technologies for Small Fixed Wing UAVs,' AIAA Journal ofAerospace Computing, Information, and Communication, Vol. 2, No. 1, January 2005, p. 92-108. [4] Kingston, D., Beard, R., McLain, T., Larsen, M., and Ren, W., "Autonomous Vehicle Technologies for Small Fixed Wing UAVs, AIAA 2nd Unmanned Unlimited Systems, Technologies, and Operations-Aerospace, Land, and Sea Corkreticcr, and Workshop and Exhibit, AIAA-2003-6559, San Diego, CA, September 2003. [5] Frew, E., "Observer Trajectory Generation for Target-Motion Estimation Using Monocular Vision," PhD Dissertation, Stanford University, August 2003. [6] Sattigeri, R., Calise, A. J., Soo Kim, B., Volyanskyy, K., and Nakwan, K., "6-DOF Nonlinear Simulation of Vision-based Formation Flight," AIAA Guidance, Navigation and Control C~r~r,rirtic and Exhibit, AIAA-2005-6002, San Francisco, CA, August 2005. [7] Beard, R., Mclain, T., Nelson, D., and Kingston, D., "Decentralized Cooperative Aerial Surveillance using Fixed-Wing Miniature UAVs," IEEE Proceedings: Special Issue on IMulti-Robot Systems, Vol. 94, Issue 7, July 2006, pp. 1306-1324. [8] Wu, A. D., Johnson, E. N., and Proctor, A. A., "Vision-Aided Inertial Navigation for Flight Control," A1AA Guidance, Navigation, and Control ~r-rTkreticcl and Exhibit, AIAA 2005-5998, San Francisco, CA, August 2005. [9] Ettinger S. M., Nechyba, M. C., Ifju, P. G., and Waszak, M., "Vision-Guided Flight Stability and Control for Micro Air Vehicle," IEEE/RSJ international C~rr r, rrtic on Intelligent Robots and System, Vol. 3, September/October 2002, pp. 2134-2140. [10] Kehoe, J., Causey, R., Abdulrahim, M., and Lind, R., "Waypoint Navigation for a Micro Air Vehicle using Vision-Based Attitude Estimation," AIAA Guidance, Navigation, and Control C~r~r,rirtic and Exhibit, AIAA-2005-6400, San Francisco, CA, August 2005. [11] Abdulrahim, M., and Lind, R., "Control and Simulation of a Multi-Role Morphing Micro Air Vehicle," AIAA Guidance, Navigation and Control ~r-rTkreticcl and Exhibit, AIAA-2005-6481, San Francisco, CA, August 2005. [12] Abdulrahim, M., Garcia, G., Ivey, G. F., and Lind, R., "Flight Testing of a Micro Air Vehicle Using Morphing for Aeroservoelastic Control," A1AA Structures, Structural Dynamics, and Materials Cr itreticc,. AIAA-2004-1674, Palm Springs, CA, April 20)04. 20 1 1 0 20 20 1 0 2 5 > 5 **5 **-0.0 Fiogur 3-3. Radifal DisgtortionEfcs foria dsA)c f s = ls 0.5 te di = 0,liea B)ai i fo = d=-0005 n Theia cameora is Ceffectivey modeled usin the fetr oa lngth and radial distoration aong wither thae ohrterms. dhpesr ibe d in s Eqa tiohn 311Auch those farmers arun ed termeny oad theinrsc itis parameters and arebise f ounde thouh albr tion. Ant feature point mut eanlze ith respett these ~ ~ ~ ~ ~ inrni aaeest nuepoersatue estimatin DTherdaldsanefrmafetr pit t he cent ter of the esimage is dependent on bot the relatv oit t e ions o caeraand features alohngwt the focwable lniength This radial distance isalso related s viat anonlineare rltinhip s toth raia distcortison.nCleaprobe fly any nlsso h fatcuraely poins wireuirsiation otie of thlecamera peatraeers Chapoter 4 wnille dicussatertchnwiqetharset consider bondedn uncertainty toard the inrisicy pramonie tersuu an esalshqnes af bounded conditionong thfatue poinestht psitios.y Thes fritrst stepi thoe estimabutiohn pobemi rqirtes teaiity trcoor dtct intcerestig features, and curves. These features usually correlate to objects of interest in the environment such as buildings, vehicles, bridges, etc. Although this gradient-based criterion is good at detecting these features, it also produces a large number of detections from highly textured surfaces that The next model considered involves a random motion model. The assumed acceleration is shown in Equation 8-4 and is characterized by a random vector, w (t) and is scaled by a constant, p. The velocity corresponding to this acceleration is described in Equation 8-5. This model attempts to capture the stochastic behaviors by utilizing a probabilistic distribution function. a (t) = pw (t) (8-4) v (t) = v (t A) + p w (T) d (8-5) Alternatively, the model shown in Equation 8-4 can be modified to incorporate some dependence on the previous acceleration value. This dependence is achieved by weighting the previous acceleration in the model and is shown in Equation 8-6. The benefit to this type of model as oppose to Equation 8-4 requires some knowledge of the target; namely, that the target cannot achieve large abrupt changing in acceleration. The resulting velocity expression for this model is given in Equation 8-7. a (t) = poa (t At) +t p w (t) (8-6) v (t) = v i+ JN(t A) + a( -A) w(T) dt (8-7) 8.2.2 Stochastic Prediction The image sequence obtained from the camera are processed by the homography to obtain the position estimates of the target. These position estimates are then used to compute a velocity profile of the target, as shown in Equation 8-8 for the ith target and N image frames. The velocity profile is computed using a backwards difference method and is given in Equation 8-9. [vi (t -1) ,vi(t -2) ,...,vi (t N+1 ),vi (t -N)] (8-8) vi (t j) = pi (t j) pi (t j 1) (8-9) Similarly, an acceleration profile, defined in Equation 8-10, is obtained from the velocity profile given in Equation 8-8. The same backwards difference method is used to compute this 10.2.4 Structure From Motion The images taken during the simulation are analyzed using structure from motion to determine the location of the environmental features. The initial analysis used the nominal camera to ensure the approach is able to correctly estimate the locations in the absence of unknown perturbations. The actual and estimated locations are shown in Figure 10-10 to indicate that all errors were less than 10-6 *Actual a Estimate 500 0- .. N. * -500 * -1000 1000\ - 2000 1000 Figure 10-10. Nominal estimation using structure from motion The depths are also estimated using structure from motion to analyze images from the perturbed cameras. A representative set of these estimates are shown in Figure 10-11 as having clear errors. An interesting feature of the results is the dependence on sign of the perturbation to focal length. Essentially, the solution tends to estimate a depth larger than actual when using a positive perturbation and a depth smaller than actual when using a negative perturbation. Such a relationship is a direct result of the scaling effect that focal length has on the feature points. Estimates are computed for each of the perturbed cameras and compared to the nominal estimate. The worst-case errors in estimation are compared to the theoretical bound, given in Equation 4-29, to these errors. These numbers shown in Table 10-8 indicate the variation in structure from motion depends on the sign of the perturbation. The approach is actually seen to be less sensitive to positive perturbations, which causes a larger estimate in depth, than to negative perturbations. Also, the theoretical bound was greater than, or equal to, the error caused by each camera perturbation. processing algorithms such as SFM are not valid for moving objects viewed by a moving camera. This limitation is caused by the epipolar constraint no longer maintaining a coplanar property across image sequences; consequently, research has evolved for the detection of moving objects in a scene viewed by a non-stationary camera. The detection of moving object in an image sequence is an important step in image analysis. For cases involving a stationary camera, simple image differencing techniques are sufficient in determining moving objects [19-21]. Techniques for more realistic applications involve Kalman filtering [22] to account for lighting conditions and background modeling techniques using statistical approaches, such as expectation maximization and mixture of Gaussian, to account for other variations in real-time applications [23-28]. Although these techniques work well for stationary cameras, they are insufficient for the case of moving cameras due to the motion of the stationary background. Motion detection using a moving camera, as in the case of a camera mounted to a vehicle, becomes significantly more difficult because the motion viewed in the image could result from a number of sources. For instance, a camera moving through a scene will view motions in the image caused by camera induced motion, referred to as egomotion, changes in camera intrinsic parameters such as zoom, and independently moving objects. There are two classes of problems considered in literature for addressing this topic. The first considers the scenario where the 3D camera motion is known a priori then compensation can be made to account for this motion to determine stationary objects through an appropriate transformation [29, 30]. The second class of problems does not require knowledge of the camera motion and consists of a two stage approach to the motion detection. The first stage involves camera motion compensation while the last stage employs image differencing on the registered image [31] to retrieve non-static objects. The transformation used to account for camera motion is commonly solved by assuming the majority of image consists of a dominant background that is stationary in Euclidean space [32, 33]. This solution is obtained through a least-squares minimization process [32] or with the use of morphological filters [34]. The transformations obtained from these techniques typically where the individual single-axis rotations are commonly referred to as 3-2-1, or roll-pitch-yaw, [el(#) 2 6) e3(w)] respectfully. The full rotation matrix is represented by Equation 5-3. cos(0) cos(y) sin( ) sin(0) cos(W) cos( ) sin(W) cos( ) sin(0) cos(W) + sin( ) sin(W) REB = COS(8) Sin(W) Sin( ) Sin(8) Sin(W) COS( ) COS(W) COS( ) Sin(8) Sin(W) Sin( ) COS(W) sin(0) sin( ) cos(0) cos( ) cos(0) (5-3) The rates of change of these orientation angles also require a coordinate transformation. The roll rate, p, is the angular velocity about bl, the pitch rate, q, describes rotation about b2, and the yaw rate, r, describes rotation about $3. The vector, m,, is given in Equation 5-4 to represent these rates. o,= p61+-tq62 -t r3 (5-4) 5.1.2 Camera The camera is also described using a right-handed coordinate system defined using orthonormal basis vectors. The axes, as shown in Figure 5-2, use the traditional choice of i3 aligning through the center of view of the camera. The remaining axes are usually chosen with 12 aligned right of the view and it aligned out the top although some variation in these choices is allowed as long as the resulting axes retain the right-handed properties. The direction of the camera basis vectors are defined through the camera's orientation relative to the body-fixed frame. This framework is noted as the camera-fixed coordinate system because the origin is always located at a fixed point on the camera and moves in the same motion as the camera. The camera is allowed to move along the aircraft through a dynamic mounting which admits both rotation and translation. This functionality enables the tracking of features while the vehicle moves through an environment. The origin of the camera-fixed coordinate system is attached to this moving camera, consequently, the camera-fixed frame is not an inertial reference. A 6 degree-of-freedom model of the camera is assumed which admits a full range of motion. Figure 5-2 also illustrates the camera's sensing cone which describes both the image plane and the field of view constraint. the ith row of this matrix can then be written as Equation 4-21. 6iu = ~I,;+2-tpU 11 pillig V16p, -t26v1 8vi,; 8,;,?~ L (4-21) U16,v V28pi -t ivz V18vz V26vl -t vl vz A solution to Equation 3-57, when including the uncertain matrix in Equation 4-20, will exist, however, that solution will differ from the true solution. Essentially, the solution can be expressed as the nominal solution, ho, and an uncertainty, Sh, as in Equation 4-22. ('Fo +t 8,) (ho +t Sh) = 0 (4-22) The solution vector, h = ho +t Sh, for Equation 4-22 has variation which will be norm bounded by Ah as in Equation 4-23. h = {ho+8 h 16 8h ~h} (4-23) The size of this uncertainty, which reflects the size of error in the state estimation, can be bounded using Equation 4-24. This bound uses the relationship between uncertainties in Equation 4-21 through the constraint in Equation 4-22. Also, the size of this uncertainty depends on the location of each feature point so the bounds is noted as specific to the rll and rl2 obtained from Figure 3-4. Ahx = I (Fo 6v)-1 who|| (4-24) 8v, < av, Iav1 < Av2 The maximum variation of the entries of h = ho +t Ah, determined through Equation 4-24, can then be used directly to compute the variation in state estimates. The entries of h are first arranged back into matrix form to construct the new homography matrix that includes parameter 5.3 Nonlinear Aircraft Equations The equations of motion of an aircraft can be represented in several different fashions. The most general form of the aircraft equations are the nonlinear, highly coupled equations of motion. These equations of motion are the standard equations which have been derived in a typical aircraft mechanics book [108-110] and are repeated in Equation 5-14 to 5-26 for overall completeness. P' mg sin O = m(ui + qw ry) (5-14) F, + mg cos 6 sin = m(v + ru pw) (5-15) F, + mg cos ecos = m(wi,+ py qu) (5-16) L= A -Grqr(z -l) -1-!4(5-17) M =Ir,q+rp(l,- I,) + ,(P2_ -2) (5-18) N = -IeI + Ir~+ pq(1-I,-) + zqr (5-19) p = #- isine (5-20) q = O cos #+ ~icos 8sin (5-21) r = Qicos ecos @- 0sin (5-22) 0= qcos -rsin~ (5-23) S= p +q9sin tan O+ rcos ~tan 8 (5-24) i = (q sin + r cos #) sec 0 (5-25) =xd CeS, S4SecwcS, +CC, CSeS, -S4C, v (5-26) dZb -Se S4Ce C4Ce The shorthand notation for Sw sinW, Cw cosy, So sin6, Co cos6, and Sq sing, C4 cos # is used in Equation 5-26. The aircraft states of interest for the camera motion system consist of the position and velocity of the aircraft's center of mass, TEB and Vb, the angular velocity, co,, and the orientation Ci = {(Oli, Vi) if A (iPi, Vi) < Jmax } (6-12) else Hi={(ii)if J2 (Pui,Vi) > Jmax} (6-13) After all n feature points have been examined under this criterion, a set of m feature points are classified to the static background, C. Meanwhile, a set of n m feature points are classified as objects disobeying the static trend, 'L, and are considered moving objects. The class of moving objects can be further classified into distinct objects through a clustering method. This method removes all static features and uses the intersections of the epipolar lines pertaining to moving objects as data points in the clustering algorithm. The resulting data will produce distinct clusters around the FOEs pertaining to moving objects. The threshold Jax is a design parameter that segments the feature points into their respective classes and needs to be tuned to account for measurement noise. Chapter 5 derives the system dynamics for an aircraft-camera configuration by formulating the differential equations and observations into a controls framework. Chapter 6 describes a method that utilizes image processing techniques to detect and segment moving objects in a sequence of images. Chapter 7 formulates a homography technique that estimates relative position and orientation with respect to a moving reference object. The method fuses traditional guidance and navigation sensors with the computed homography to obtain relative state estimates of the unknown object with respect to the moving camera. This process applies directly to solving a significant portion of the AAR problem. Chapter 8 summaries a modeling technique for moving objects to predict the target's motion in inertial space. Chapter 9 discusses a control design scheme that exploits vision-based state estimates to track and home on a desired target. The control framework will be generalized for many mission scenarios involving autonomous UAV but will be discussed in the context of the AAR problem. Chapter 10 will implement in MATLAB the vision algorithms for both open and closed-loop architectures to demonstrate and verify the purposed methods. Chapter 11 discusses concluding remarks and proposes future research directions for this work. for the first iteration. It is assumed for the first iteration that the two features are static. The least-squares solution is then given in Equation 6-9 for the FOE coordinates (Pfoetv,Vfoe) (for the first iteration a least-squares solution is not necessary because two lines intersect at a single point) . = ar mm | b ||2 (6-9) where 1M= I # ##2l Prz Vi+1 (6-10a) -1 -1 -1 b ~ ~ ~ t = z- p 2 2 "" Vi+1 A Pi 1 (6-10b) The next iteration adds another feature into the system of equations and a new potential FOE point is obtained. If the new feature point is a static feature, then the new estimated FOE will be near the static FOE, which is found in the first iteration, causing a small residual. Alternatively, if the feature is point is due to a moving object then the epipolar line will not intersect the static FOE and shift the solution causing a large residual. Defining the new FOE coordinates as (pfoue27 1982). A cost function is then checked to verify if the new feature point contains a similar motion to that of the static background by checking the residual. This residual is defined as the Euclidean distance from the two FOE solutions found before and after adding the next feature. If cost is higher than some maximum threshold Jmax then the feature point is discarded into a set of points classified as moving, fl; else, the feature point is classified into the static FOE solution, C. This process is repeated until all n feature points have been checked using this cost function which is shown in Equation 6-11 for the ith iteration. Mathematically, the classification scheme for the ith iteration is given in Equations 6-12 and 6-13. 12 PVi Pfei -Pfoi-1 2 foei-Vfei-) 2(6-11) drogue, especially moments before the docking phase. The aerodynamic data acquired in these experiments confirmed several dependencies on turbulence, flight conditions, and geometry. 2.4 Uncertainty in Vision Algorithms The location of environmental features can be obtained using structure from motion. The basic concepts are mature but their application to complex problems is relatively limited due to complexities of real-time implementation. In particular, the noise issues involved with camera calibration and feature tracking cause considerable difficulties in reconstructing 3-dimensional states. A sampling-based representation of uncertainty was introduced to investigate robustness of state estimation [82]. Robustness was also analyzed using a least-square solution to obtain an expression for the error in terms of the motion variables [83]. The uncertainty in vision-based feedback is often chosen as variations within feature points; however, uncertainty in the camera model may actually be an underlying source of those variations. Essentially, the uncertainty may be associated with the image processing to extract feature points or with the camera parameters that generated the image. The proper characterization of camera uncertainty may be critical to determine a realistic level of feature pomnt uncertainty. The analysis of camera uncertainty is typically addressed in a probabilistic manner. A linear technique was presented that propagates the covariance matrix of the camera parameters through the motion equations to obtain the covariance of the desired camera states [84]. An analysis was also conducted for the epipolar constraint based on the known covariance in the camera parameters to compute the motion uncertainty [85]. A sequential Monte Carlo technique demonstrated by Qian et al. [86] proposed a new structure from motion algorithm based on random sampling to estimate the posterior distributions of motion and structure estimation. The experimental results in this paper revealed significant challenges toward solving for the structure in the presence of errors in calibration, feature point tracking, feature occlusion, and structure ambiguities. 10.2.3 The Epipolar Constraint State estimation is performed by considering the epipolar constraint to relate the pair of images. The evaluation of images generated using the nominal camera for this simulated case is able to estimate the correct states. An investigation of the epipolar lines shown in Figure 10-8 shows the quality of the estimation. Essentially, the epipolar geometry requires a feature point in one image to lie along the epipolar line. This epipolar line is constructed by the intersection between the plane formed by the epipolar constraint and the image plane at the last measurement. The data in Figure 10-8 show the features in the second image do indeed lie exactly on the epipolar lines. 0.4~ '* + 0.4~ . -0.2 ~ -0.2 -0.4~ 0.4 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 v v A B Figure 10-8. Epipolar lines between two image frames: A) initial frame and B) final frame with overlayed epipolar lines for nominal camera The introduction of uncertainty into the epipolar constraint will cause variations in the essential matrix which will also propagate through the computation of the epipolar line. These variations in the epipolar line are visual clues of the quality of the estimate in the essential matrix. These variations can occur as changes in the slope and the location of the epipolar line. Figure 10-9 illustrates the epipolar variations due to perturbations on 87 = 0.1 and 6d = 0.01 to the camera parameters. The feature points with uncertainty and the corresponding epipolar line was plotted along with the nominal case to illustrate the variations. The key point in this figures is the small variations in the slope of the epipolar lines and the significant variations in feature [25] Sheikh, Y., and Shah, M., Bayesian Object Detection in Dynamic Scenes," IEEE Computer Society C~rreakican on Computer Vision and Pattern Recognition, San Diego, CA, June 2005. [26] Stauffer, C., and Grimson, W. E. L., "Adaptive Background Mixture Models for Real-Time Tracking," IEEE CriC irthe, on Computer V/ision and Pattern Recognition, Fort Collins, CO, June 1999, pp. 246-252. [27] Toyama, K., Krumm, J., Brumitt, B., and Meyers, B., "Wallflower: Principles and Practice of Background Maintenance", International C~r rthrn ,l on Computer Vision, Corfu, Greece, September 1999. [28] Zhou, D., and Zhang, H, "Modified GMM Background Modeling and Optical Flow for Detection of Moving Objects," IEE International C~r ratherrn on System, Man, and Cybernetics, Big Island, Hawaii, October 2005. [29] Nelson, R. C., "Qualitative Detection of motion by a Moving Observer," International Journal of Computer Vision, Vol. 7, No. 1, 1991, pp. 33-46. [30] Thompson, W. B., and Pong, T. G., "Detecting Moving Objects," International Journal of Computer Vision, Vol. 4, 1990, pp. 39-57. [31] Odobez, J. M., and Bouthemy, P., "Detection of Multiple Moving Objects Using Multiscale MRP With Camera Motion Compensation," IEEE International C~rrTkratec, on Image Processing, Austin, TX, November 1994, pp. 245-249. [32] Irani, M., Rousso, B., and Peleg, S., "Detecting and Tracking Multiple Moving Objects Using Temporal integration," European C~rr rthrrn l on Computer Vision, Santa Margherita Ligure, Italy, May 1992 pp. 282-287. [33] Torr, P. H. S., and Murray, D. W., "Statistical Detection of Independent Movement from a Moving Camera," Image and Computing, Vol. 11, No. 4, May 1993, pp. 180-187. [34] Gandhi, T., Yang, M. T., Kasturi, R., Camps, O., Coraor, L., and McCandless, J., "Detection of Obstacles in the Flight Path of an Aircraft," IEEE Transactions on Aerospace and Electronic Systems, Vol. 39, No. 1, January 2003, pp. 176-191. [35] Irani, M., and Anandan, P., "A Unified Approach to Moving Object Detection in 2D and 3D Scenes," IEEE Transactions on Pattern Analysis and 1MAchine Intelligence, Vol. 20, No. 6, June 1998. [36] Sharma, R., and Aloimonos, Y., "Early Detection of Independent Motion from Active Control of Normal Flow Patterns," IEEE Transactions on Systems, 1Man, and Cybernetics, Vol. 26, No. 1, February 1996. [37] Frazier, J., and Nevatia, R., "Detecting Moving Objects from a Moving Platform," IEEE International C~rrTk ratec, on Robotics and Automation, Nice, France, May 1992, pp. 1627-1633. 10-30 10-31 10-32 10-33 10-34 10-35 10-36 10-37 Open-loop estimation of target's inertial attitude . Norm error for target state estimates ...... Closed-loop target position tracking ...... Position tracking error ...... Target attitude tracking ...... Tracking error in heading angle ....... Target's inertial position with uncertainty bounds Target's inertial attitude with uncertainty bounds . .. .. ... . 145 ........ .. .146 ........ .. .146 .......... .. 147 .......... .. 147 ........ .. .147 .... .. .. .149 . .... ... .. .149 epipolar constraint enables the relationship between image frames to compute the features of the target through Equation 7-15 PvT,n (xyTVRIV) pr,n= (7-15) where xyjv is the skew symmetric representation of the relative translation from I to V expressed in I and the new pixel coordinates determined from the virtual camera are denoted as pVF,n = [PVF,n, VVF,n] Vn2 for the reference vehicle and pyr,n = [PVr,n, Wa~n] Vn2 for the target vehicle. As a result of the virtual camera, the desired property is obtained regarding pixels of the reference vehicle computed from the camera at i 1 are equal to the pixels generated by the virtual camera at i. Mathematically, this property is expressed in Equation 7-16 which relies on the relative motion remaining constant to maintain the reference stationary in the image. PF,n ( 1) = PVF,n (i) (7-16) With this virtual camera in place and the reference pixels stationary, the computation of the homography between the reference and target vehicles is considered. First, the geometric relationships are established relative to the virtual camera of both the reference and target vehicles by denoting their feature point positions in Euclidean coordinates. The time varying position of a feature point on the reference vehicle expressed in V is given in Equation 7-17. Likewise, the time varying position of a feature point on the target vehicle expressed in V is given in Equation 7-18. rlVF,n = mvF +tRFVSF (7-17) rlvr,n = myr +tRTVSy (7-18) The components of these Euclidean coordinates are defined in Equations 7-19 and 7-20 and are relative to the virtual camera frame. rlVF,n (t) t yt t)(-9 vrn()a x t r()L t (7-20) CHAPTER 10 SIMULATIONS 10.1 Example 1: Feature Point Generation A simulation of vision-based feedback is presented to demonstrate the implementation, and resulting information, associated with sensor models and aircraft dynamics. This simulation utilizes a nonlinear model of the flight dynamics of an F-16 [110]. A baseline controller is implemented that allows the vehicle to follow waypoints based entirely on feedback from inertial sensors. Images are obtained from a set of cameras mounted on the aircraft. These cameras include a stationary camera mounted at the nose and pointing along the nose, a translating camera under the centerline that moves from the right wing to the left wing, and a pitching camera mounted under the center of gravity. The parameters for these cameras are given in Table 10-1 in values relative to the aircraft frame and functions of time given as t in seconds. Table 10-1. States of the cameras position (ft) orientation (deg) camera xe ye ze Wcc V 1 24 0 0 0 90 0 2 -10 15-3t 0 0 45 0 3 0 0 3 0 45-9t 0 The camera parameters are chosen as similar to an existing camera that has been flight tested [111]. The focal length is normalized so f = 1. Also, the field of view for this model correlates to angles of y;; = 32 deg and y, = 28 deg. The resulting limits on image coordinates are given in Table 10-2. Table 10-2. Limits on image coordinates coordinate minimum maximum pu -0.62 0.62 v -0.53 0.53 A virtual environment is established with some characteristics similar to an urban environment. This environment includes several buildings along with a moving car and a To my lovely wife, Liza P. Causey, that has supported me every step of the way. Her love and understanding through the years have brought my passion for life beyond boundaries. A commonly used algorithm that employed these equations with slight variations is the Harris comer detector [106]. This method can be extended to edge detection by considering the structure of the singular values of G. An example of this algorithm is the Canny edge detector [107]. 3.4 Feature Point Tracking Feature tracking or feature correspondence is the next step in the general state estimation problem. The correspondence problem is described as the association of a feature point between two or more images. In other words, the solution to this problem determines that a feature point in two or more images corresponds to the same physical point in 3D space. The most common approach to discerning how points are moving between images is the use of intensity or color matching. This brightness matching is typically performed over a small window, W(x), centered around the point of interest, as opposed to only matching a single brightness value, which could have numerous false solutions. The vector of brightness values over a small window set, 1, contained in the image is shown in Equation 3-21. 1(x) = {I(i)| IE W (x)} (3-21) This brightness vector can be compared across images, li and I2, and optimized to find the minimum error. If a feature point of interest is located at xl = [pt, v1] in image 1, li then a simple translational model of the same scene can be used as an image matching constraint. This relationship is shown in Equation 3-22, 11 (xl) = I2(h(x1)) +t n(h(x1)) (3-22) where h(x) defines a general motion transformation to the proceeding image and n(xl) is additive noise caused by ambiguities such as variations in lighting, reflections, and view point. Therefore, the correspondence solution is cast as a minimization problem that computes the best intensity match over a small window by minimizing the intensity error. An equation for the translation estimate can then be found from this minimization process through Equation 3-23, noise and unknown camera parameters so, in practice, an averaging process is often used to estimate the feature coordinates. There are two fundamental issues regarding the obtained solution. First, by relying on the solution provided by the eight-point algorithm, then the translation is only determined up to a scaling factor. The SFM solution will therefore be corrupted from this scale factor unless an alternative method is used to obtain this scaling. Second, the uncertainty due to intrinsic parameters, feature detection, feature tracking, along with the uncertainty in the solution of the eight-point algorithm contributes to large variations in the SFM solution. The solution obtained from Equation 3-64 is very sensitive to these uncertainties. Chapter 4 will discuss a method to obtain uncertainty bounds on the SFM estimates based on the sources described. [51] Silveira, G. F., Carvalho, J. R. H., Madirid, M. K., Rives, P., and Bueno, S. S., "A Fast Vision-Based Road Following Strategy Applied to the Control of Aerial Robots," IEEE Proceedings of XIV Brazilian Symposium on Computer Graphics and Image Processing, 1530-1834/01, Florianopolis, Brazil, October 2001, pp. 226-231. [52] Soatto, S., and Perona, P., "Dynamic Visual Motion Estimation from Subspace Constraints," IEEE, 0-8186-6950-0/94, 1994, pp. 333-337. [53] Soatto, S., Frezza, R., and Perona P., "Motion Estimation via Dynamic Vision," IEEE Transactions on Automatic Control, Vol. 41, No. 3, March 1996, pp. 393-413. [54] Soatto, S., and Perona, P., "Recursive 3-D Motion Estimation Using Subspace Constraints," international Journal of Computer Vision, Vol. 22, No. 3, 1997, pp. 235-259. [55] Soatto, S., and Perona, P., "Reducing 'Structure from Motion'," IEEE, 1063-6919/96, 1996, pp. 825-832. [56] Soatto, S., and Perona, P., "Visual Motion Estimation from Point Features: Unified View," IEEE international C~rr rtf rrtic on Image Processing, Vol. 3, October 1995, pp. 21-24. [57] Soatto, S., and Perona, P., "Reducing 'Structure from Motion': A General Framework for Dynamic Vision Part 1: Modeling," IEEE Transactions on Pattern Alrab\ \i\ and 1Machine Intelligence, Vol. 20, No. 9, September 1998, pp.933-942. [58] Soatto, S., and Perona, P., "Reducing 'Structure from Motion': A General Framework for Dynamic Vision Part 2: Implementation and Experimental Assessment," IEEE Transactions on Pattern Ata rl~li\ and 1Machine intelligence, Vol. 20, No. 9, September 1998, pp. 943-960. [59] Erol, A., Bebis, G., Nicolescu, M., Boyle, R. D., and Twombly, X., "A Review on Vision-Based Full DOF Hand Motion Estimation," IEEE Computer Society International ~r~rTforeticc on Computer Vision and Pattern Recognition, San Diego, CA, June 2005. [60] Huang, T. S., and Netravali, A. N., "Motion and Structure from Feature Correspondences: A Review," Proceedings of the IEEE, Vol. 82, No. 2, February 1994, pp. 252-268. [61] Stewart, C. V., "Robust Parameter Estimation in Computer Vision," Society of Industrial and Aplied Mathelr manil, \, Vol. 41, No. 3, 1999, pp. 513-537. [62] Weng, J., Huang, T. S., and Ahuja, N., "Motion and Structure from Two Perspective Views: Algorithms, Error Analysis, and Error Estimation," IEEE Transactions on Pattern Analysis and 1Machine intelligence, Vol.11, No. 5, May 1989, pp. 451-476. [63] Jianchao, Y., "A New Method for Passive Location Estimation from Image Sequence Using Adaptive Extended Kalman Filter," International Confrrticcr, on Signal Processing, Beijing, China, October 1998, pp. 1002-1005. [64] Qian, G., Kale, G., and Chellappa, R., "Robust Estimation of Motion and Structure using a Discrete H-infinity Filter," IEEE 0- 7803-6297-7/00, 2000, pp. 616-619. Finally, the Markov transition probability function is given explicitly in Equation 8-16 as three-dimensional Gaussian density function and is uniquely determined by the mean and variance. 1 1 I(ai (t) p (ai (t))) 2 P (ai (t))= ep(-6 AGn (ai (t))1 e 2 G2 a t (-6 The probability function is then extended over the entire time interval [t, t +t k] to estimate the prediction probability. Mathematically, this extension is expressed in Equation 8-17. P (ai (t +t j)) = P (ai (t +t j 1)) = .. = P (ai (t)) (8-17) Employing Equations 8-16 and 8-17, the predictive probability for object i at time t +t k is given as Equation 8-18. This framework enables the flexibility of computing the predicted estimates at any desired time in the future with the notion that further out in time the probability diminishes. k-1 Probe (ai (t + k)) = n P (ai (t. +j) (8-18) j= 0 A similar process is considered for computing the Markov transition functions for both velocity and position. First, the mean and variance vectors for velocity and position are defined in Equations 8-19 and 8-20 for the entire time interval. [p (vi (t + j)) G2 (Vi (t + j))] j ,1 ,k(8-19) [p(pi (t+- j)) ,G2 (pi(t + j))] j=01.k(8-20) The initial mean and variance expressions for the velocity are given in Equation 8-22 and 8-23. pu (vi (t)) = pu (vi (t 1) +t ai (t 1)) (8-21) = vi (t 1) +t p (ai (t 1)) G2 (Vi (t)) = G2 (Vi (t 1) +t ai (t 1)) (8-22) The simulation was then played in a virtual environment to enhance the graphics and illustrate the application of this algorithm. To add the vehicles within the virtual environment the velocities of each vehicle had to be scaled down to practical values that fit the scene. Snapshots are shown in Figure 10-22 of the camera view depicting the vehicles and the surrounding scene. The red vehicle was designated as the reference whereas the grey vehicle was the target vehicle. The next step in this process is to implement an actual feature tracking algorithm on the synthetic images that follows the vehicles. This modification alone will degrade the homography results immensely due to the troublesome characteristics of a feature point tracker. Figure 10-22. Virtual environment 10.4 Example 4: Closed-loop Aerial Refueling of a UAV A closed-loop simulation was executed in Matlab to replicate an autonomous aerial refueling task. As Chapter 1 described the motivation and the benefits of AAR, this section will demonstrate it by combining the control design given in Chapter 9 with the homography result in Chapter 7 to form a closed-loop visual servo control system. The vehicles involved in this simulation includes a Receiver UAV instrumented with a single camera, a tanker aircraft also are slight differences in direction due to the translating camera. Likewise, the optic flow observed by camera 3 is different due to the camera's orientation. -0 -4 02 0 02 04 06 -06 -04 -02 0 02 04 06 -06 -04 -02 0 0 4 0 A B C Figure 10-3. Optic flow Measurements at t = 2 sec for A) camera 1, B) camera 2, and C) camera A summary of the resulting image plane quantities, position and velocity, is given in Table 10-5 for the feature points of interest as listed in Table 10-3. The table is organized by the time at which the image was taken, which camera took the image, and which feature point is observed. This type of data enables autonomous vehicles to gain awareness of their surroundings for more advanced applications involving guidance, navigation and control. Table 10-5. Image coordinates of feature points Time (s) Camera Feature Point pu v p 9 2 1 1 0.157 0.162 0.610 0.044 2 1 3 0.051 0.267 0.563 -0.012 2 2 2 -0.308 0.075 0.464 -0.254 2 3 2 0.011 0.077 0.583 -0.235 4 2 2 -0.279 -0.243 -0.823 0.479 4 3 2 0.365 -0.248 -0.701 0.603 6 1 3 0.265 -0.084 0.267 -0.015 10.2 Example 2: Feature Point Uncertainty 10.2.1 Scenario Feature point uncertainty is demonstrated in this section by extending the previous example. This simulation will examine the uncertainty effects on vision processing algorithms using simulated feature points and perturbed camera intrinsic parameters. begin to falter due to the drastic changes in lighting conditions and the large view point change. Therefore, a more general motion transformation is used such as an affine transformation. Normalized cross-correlation techniques are also used for large baseline configurations to handle considerable changes in lighting conditions. An extremely important concern is the accuracy of these algorithms and how variations in feature point tracking can effects the final state estimation. These concerns will be addressed in detail in Chapter 4. 3.5 Optic Flow The next metric of interest in the image plane is an expression for the velocity of a feature point. This expression is found simply by taking the time derivative of the feature point position defined in Equations 3-7 and 3-8. The velocity expressions, shown in Equations 3-32 and 3-33, describe the movement of feature points in the image plane and is commonly referred to in literature as the optic flow. Likewise, the feature point velocity with radial distortion can be computed by differentiating Equations 3-15 and 3-16 while assuming c = 0 is as follows ild = ii(1+- dr2) +t 2pdri (3-34) itd = 9j(1+- dr2) +t 2vdrt (3-35) where pu v r = p- + 9 (3-36) 3.6 Two-View Image Geometry The two-view image geometry relates the measured image coordinates to the 3D scene. The camera configuration could be either two images taken over time of the same scene, as CHAPTER 7 HOMOGRAPHY APPROACH TO MOVING TARGETS 7.1 Introduction Autonomous vehicles have gained significant roles and assisted the military on the battlefield over the last decade by performing missions such as reconnaissance, surveillance, and target tracking with the aid of humans. These vehicles are now being considered for more complex missions that involve increased autonomy and decision making to operate in cluttered environments with less human interaction. One critical component that autonomous vehicles need for a successful mission is the ability to estimate the location and movement of other objects or vehicles within the scene. This capability, from a controls standpoint, enables autonomous vehicles to navigate in complex surroundings for tracking or avoidance purposes. Target state estimation is an attractive capability for many autonomous systems over a broad range of applications and is the focus of this dissertation. In particular, unmanned aerial vehicles (UAV) have shown a great need for this technology. With UAV becoming more prevalent in the aerospace community, researchers are striving to extend their capabilities while making them more reliable. The key applications of interest for future UAV regarding target estimation pertain to both civilian and military tasks. These tasks range from police car pursuits and border patrol to locating and pursuing enemy vehicles during lethal engagement. A major limitation to small UAV are their range, payload constraints and fuel capacity. These limitations generate the need for autonomous aerial refueling (AAR) to extend the vehicle's operational area. Target state estimation facilitates a portion of the AAR problem by estimating the receptacle's current position and orientation during approach. Therefore, the purpose of this paper is to demonstrate a method that estimates the motion of a target using an on-board monocular camera system to address these applications. Most techniques for vision-based feedback share some commonality; namely, a sequence of image processing and vision processing are performed on an image or a set of images to extract information which is then analyzed to make a decision. The basic unit of information from an stabilization design also included a roll to elevator connect to help counteract the altitude loss during a tumn. The outer-loop is completed by simply closing the loop around the roll tracker using a proportional gain to follow to desired heading. In addition, command limits of +600 were placed on roll to regulate aggressive turns and a yaw damper was also implemented that included a aileron-rudder interconnect which helps a tumn in a number of ways. The aileron-rudder interconnect helps to raise the nose up during a turn. Meanwhile, the yaw damper is employed to damp oscillations from the Dutch-roll mode during a heading maneuver. The design of the yaw damper is provided in Stevens et al. [110]. Consequently, the tumn smoother and contains less oscillations. Tracking heading is not sufficient to track the lateral position with the level of accuracy needed for refueling task. The final loop was added to account for any lateral deviation accumulated over time due to the delay in heading from position. This delay is mainly due to the time delay associated with sending a roll command and producing a heading change. Therefore, this loop was added to generate more roll for compensation. The loop commanded a change in aileron based of the error in lateral position. This deviation, referred to as Ay, was computed based on two successive target locations provided by the estimator. The current and previous (x, y) positions of the target were used to compute a line in space to provide a reference of the it's motion. The perpendicular distance from the vehicle's position to this line was considered the magnitude of the lateral command. In addition, the sign of the command was needed to assign the correct direction. This direction was determined from the relative y position, expressed in the body-fixed frame, that was found during estimation. Once the lateral deviation was determined, that signal was passed through a PI structure, as shown in Figure 9-2. The gains corresponding to the proportional, kyp, and integrator, kyi, were then summed and added to compute the final roll command. The complete expression for the roll command is shown in Equation 9-1. ky #cmd = kW (Vcmd W) +t ky p~y + ~Yi (9-1) (R, T) Figure 3-4. Geometry of the epipolar constraint The expressions in Equation 3-37 and Equation 3-38 reflect that the scalar triple product of three coplanar vectors is zero, which forms a plane in space. These relationships can be expanded using linear algebra [102, 103] to generate a standard form of the epipolar geometry as in Equation 3-39. This new form indicates a relationship between the rotation and translation, written as the essential matrix denoted as Q, to the intrinsic parameters of the camera and associated feature points. In this case, the equation is derived for a single feature point that is correlated between the frames, [#2 V2 fe 91 v1 fT = 0 (3-39) where Q = [T] xR and [T] x is defined as the skew-symmetric form of the translation T. The geometric relationship formed by this triangular plane is also seen in the epipolar lines of each image. The 3D plane formed through this triangle constrains a feature point in one image to lie on the epipolar line in the other image. These constraints can be mathematically expressed function given in Equation 3-27. El(u) = [VI (i, t)u(x) +tIt (1, t)]2 (3-27) w(x) The minimum of this function is obtained by setting VE1 = 0 to obtain Equation 3-28, LI,2 LCly + I<^ = 3- I C,tv LIS ut LIvT, O(-8 or, rewritten in matrix form results in the following Gu -tb= (3-29) where G(x) was derived in Equation 3-19. The final solution for the pixel velocity is found through a least-squares estimate given in Equation 3-30. These image velocities are also referred to as the optic flow. Once the optic flow is computed for a feature point then the image displacement for feature tracking is trivial to find. u = G b (3-30) On the other hand, the method using SSD, shown in Equation 3-24, attempts to estimate the Ax while not requiring the computation of image gradients. This approach also employs the translational model over a windowed region. The method considers the possible range that window could move, dpu and dv, in the time, dt. This consistency constraint then leads to a problem of minimizing the error over the possible windows within the described range. This error function is described mathematically in Equation 3-31. E2 (dpdv) = [I(p + dp, lv + dv, t +dt) I0pv, t)]2 (3-31) W( c,v) The solutions obtained are the displacement components, dpu and dv, of the specified window that correlates to the translation of the center pixel. This techniques is the foundation for the Lucas-Kanade tracker [17]. For large baseline tracking simple translational models section examines the motion detection problem through the residual optical flow to further classify static objects from dynamic objects in the field of view. --~---0.5 FOE2 -'- 0.5 SOE 1 10.5 0 -0.5 -1 Figure 6-3. Residual optic flow for dynamic environments 6.2 Classification The classification scheme proposed in this dissertation is an iterative approach to computing the FOE of the static environment using the residual optical flow given in Equation 6-8. An approximation for the potential location of the FOE is found by extending the translational optical-flow vectors to form the epipolar lines, as illustrated in Figure 6-3, and obtaining all possible points of intersection. As mentioned previously, the intersection points obtained will constitute a number of potential FOEs; however, only one will describe the static background while the rest are due to moving objects. The approach considered for this classification that essentially groups the intersection data together through a distance criterion is an iterative least-squares solution for the potential FOEs. The iteration procedure tests all intersection points as additional features are introduced to the system of equations each of which involves 2 unknown image plane coordinates of the FOE (Pfoe;, VSoe;). The process starts by considering 2 feature points and their FOE intersection 77VT,n /IT~n XV 11IT mlVT Target e eFeature Fly~ syPoin E lve~ sPoint sF FauePoint sF) Fetur n IF F A B Figure 7-2. Moving target vector description relative to A) camera I and B) virtual camera V distortion, discrete mapping into pixels, and field of view constraints which are further also specified in Chapter 3. Each extension to the model adds another parameter to know for the estimation problem and each can introduce uncertainty and large errors in the estimation result. Therefore, this chapter will only consider the field of view constraint and leave the nonlinear terms and the effects on estimation for future work. Recall the field of view constraints given in Chapter 3. These constraints can be represented as lower and upper bounds in the image plane and are dependent on the half angles (yh, t) which are unique to each camera. Mathematically, these bounds are shown in Equations 7-1 and 7-2 for the horizontal and vertical directions. [pip] = [-f tanh, f tanyh (7-1) [v, V] = [- f tany,, f tany,] (7-2) 7.2.2 Homography Estimation The implicit relationship between camera and environment is known as the epipolar constraint or, alternatively, the homography constraint. This constraint notes position vectors that describe a feature point, rln, at two instances in time are coplanar with the camera's translation initially travel north until the target vehicles makes a left turn and heads west and is subsequently followed by the pursuit vehicles. 2 100'"" -10000 00 20 40 60 <* 0 -00 20 40 60 Time (sec) Time (sec) Time (sec) A B C Figure 10-13. Position states of the UAV with on-board camera: A) North, B) East, and C) Down 60 10 5 50 0 0 20 40 60 -10 20 40 60 -0020 40 60 Time (sec) Time (sec) Time (sec) A B C Figure 10-14. Attitude states of the UAV with on-board camera: A) Roll, B) Pitch, and C) Yaw 1~ 5 800 400 0 5 -15000 020 00 20 40 60 <* 0 00 20 40 60 Time (sec) Time (sec) Time (sec) A B C Figure 10-15. Position states of the reference vehicle (pursuit vehicle): A) North, B) East, and C) Down 10.3.2 Open-loop Results The homography was computed for this simulation to find the relative rotation and translation between the ground vehicles. These results are then used to find the relative 135 and Kanade [115] proposed an algorithm that reconstructs 3D motion of a moving object using a factorization-based algorithm with the assumption that the object moves linearly with constant speeds. A nonlinear filtering method was used to solve the process model which involved both the kinematics and the image sequences of the target [118] This technique requires knowledge of the height above the target which was done by assuming the target traveled on the ground plane. This assumption allowed other sensors, such as GPS, to provide this information. The previous work of Mehta et al. [77] showed that a moving monocular camera system could estimate the Euclidean homographies for a moving target in reference to a known stationary object. The contribution of this chapter is to cast the formulation shown in Mehta el al. to a more general problem where both target and reference vehicles have general motion and are not restricted to planar translations. This proposed approach incorporates a known reference motion into the homography estimation through a transformation. Estimates of the relative motion between the target and reference vehicle are computed and related back through known transformations to the UAV. Relating this information with known measurements from GPS and IMU, the reconstruction of the target's motion can be achieved regardless of its dynamics; however, the target must remain in the image at all times. Although the formulation can be generalized for n cameras with independent position, orientation, translations, and rotation this chapter describes the derivation of a single camera setup. Meanwhile, cues on both the target and reference objects are achieved through LED lights or markers placed in a known geometric pattern of the same size. These markers facilitate the feature detection and tracking process by placing known features that stand out from the surroundings while the geometry and size of the pattern allows for the computation of the unknown scale factor that is customary to epiploar and homography based approaches. This chapter builds on the theory developed in Chapters 3 and 5 while relying on the moving object detection algorithm to isolate moving objects within an image. Recall the flow of the overall block diagram shown in Figure 1-6. The process started by computing features in the provide poor estimation if the motions of moving objects are not accounted for in the registration process or if the image contains stationary objects close to the camera that result in high parallax. A technique presented by Irani et al. [35] proposed a unified method to detect moving objects. This proposed method handles various levels of parallax in the image through a segmentation process that is performed in layers. The first layer extracts the background objects which are far away from the camera and have low parallax through a general transformation involving camera rotation, translation, and zoom through image differencing. The next layer contains the object with high parallax consisting of both objects close to the camera and objects that are moving independently of the camera. The parallax is then computed for the remaining pixels and compared to one pixel. This process separates the objects within the image based on their computed parallax. The selection may involve choosing a point on a known stationary object that contains high parallax so any object not obeying this parallax is classified as a moving object in the scene. Optic flow techniques are also used to estimate moving target locations once ego-motion has been estimated. A method that computes the normal image flow has been shown to obtain motion detection [36]. Coordinate transformations are sometimes used to facilitate this approach to detecting motion. For instance, a method using complex log mapping was shown to transform the radial motions into horizontal lines upon which vertical motion indicate independent motion [37]. Alternatively, spherical mapping was used geometrically to classify moving objects by segmenting motions which do not radiate from the focus of expansion (FOE) [29]. 2.2 State Estimation Using Vision Information The types of state estimation that can be obtained from an on-board vision system are (i) localization which estimates the camera motion between image frames from known stationary feature points, (ii) mapping which estimates the location of 3D feature points using reconstruction and structure from motion, and (iii) targpet-motion which estimates 3D feature points that have independent motion. The work related to these topics are described in this section. [91] Papaikolopoulos, N. P., Nelson, B. J., and Khosla, P. K., "Six Degree-of-Freedom Hand/Eye Visual Tracking with Uncertain Parameters," IEEE Transactions on Robotics and Automation, Vol. 11, No. 5, October 1995, pp. 725-732. [92] Sznaier, M., and Camps, O. I, "Control issues in Active Vision: Open Problems and Some Answers," IEEE C~rr rikrrtec on Decision and Control, Tampa, FL, December 1998, pp. 3238-3244. [93] Frezza, R., Picci, G., and Soatto, S., "Non-holonomic Model-based Predictive Output Tracking of an Unknown Three-dimensional Trajectory," IEEE ~r rTkratec, on Decision and Control, Tampa, FL, December 1998, pp. 3731-3735. [94] Papaikolopoulos, N. P., Khosla, P. K., and Kanade, T., "Visual Tracking of a Moving Target by a Camera Mounted on a Robot: A Combination of Control and Vision," IEEE Transactions on Robotics and Automation Vol. 9, No. 1, February 1993, pp. 14-35. [95] Papaikolopoulos, N. P., and Khosla, P. K., "Adaptive Robotic Visual Tracking: Theory and Experiments," IEEE Transactions on Automatic Control, Vol.38, No. 3, March 1993, pp. 429-445. [96] Zanne, P., Morel, G., and Plestan, F., Robust Vision Based 3D Trajectory Tracking using Sliding Mode Control," IEEE International C~rr rikrate on Robotics and Automation, San Francisco, CA, April 2000, pp. 2088-2093. [97] Zergeroglu, E., Dawson, D. M., de Queiroz, M. S., and Behal, A., "Vision-Based Nonlinear Tracking Controllers with Uncertain Robot-Camera Parameters," IEEE/ASIME International C~rrik reticc on Advanced 1Mechanics, Atlanta, GA, September 1999, pp. 854-859. [98] Valasek, J., Kimmett, J., Hughes, D., Gunnam, K., and Junkins, J. L., "Vision Based Sensor and Navigation System for Autonomous Aerial Refueling," A1AA 's 1st Technical C~r~rikicticc and Workshop on Unmanned Aerospace Vehicles, Portsmouth, Virginia, May 2002. [99] Pollini, L., Campa, G., Giulietti, F., and Innocenti, M., "Virtual Simulation Set-Up for UAVs Aerial Refueling," AIAA 1Modeling and Simulation Technologies C~rrakican c. Austin, TX, August 2003. [100] No, T. S., and Cochan, J. E., "Dynamics and COntrol of a Teathered Flight Vehicle," Journal of Guidance, Control, and Dynamics, Vol. 18,No. 1, January 1995, pp. 66-72. [101] Forsyth, D. A., and Ponce, J., "Computer Vision : A Modern Approach, Prentice-Hall Publishers, Upper Saddle River, NJ, 2003. [102] Ma, Y., Soatto, S., Kosecka, and Sastry, S. S., "An Invitation to 3-D Vision: From Images to Geometric Models," Springer-Verlag Publishing New York, NY, 2004. [103] Faugeras, O., "Three-Dimensional Computer Vision," The MIT Press, Cambridge Massachusetts, 2001. known [94]. Adaptive solutions presented in [91, 95-97] have shown control solutions for target tracking with uncertain camera parameters while estimating depth information. The homing control problem has numerous applications toward autonomous systems such as autonomous aerial refueling, spacecraft docking, missile guidance, and object retrieval using a robtotic manipulator. Kimmett et al. [15, 98] developed a candidate autonomous probe-and-drogue aerial refueling controller that uses a command generator tracker (CGT) to track time-varying motions of a non-stationary drogue. The CGT is an explicit model following control technique and was demonstrated in simulation for a moving drogue with known dynamics subject to light turbulence. Tandale et al. [16] extended the work of Kimmett and Valasek by developing a reference observer based tracking controller (ROTC) which does not require a drogue model or presumed knowledge of the drogue position. This system consist of a reference trajectory generation module that sends commands to an observer that estimates the desired states and control for the plant. The input to this controller is the relative position between the receiver aircraft and the drogue measured by the vision system. A similar vision approach to aerial refueling is also presented in [99], where models of the tanker and drogue are used in conjunction with an inferred camera. The drogue model used in this paper was taken from [100] that uses a multi-segment approach to deriving the dynamics of the hose. Meanwhile, Houshangi et al. [80] considered grasping a moving target by adaptively controlling a robot manipulator using vision interaction. The adaptive control scheme was used to account for modeling errors in the manipulator. In addition, this paper considered unknown target dynamics. An auto-regressive model approach was used to predict the target's position based on passed visual information and an estimated target velocity. Experimental test cases are documented that show tracking convergence. 9.2 Controller Development The control architecture chosen for this mission consisted of a Proportional, Integral and Derivative (PID) framework for waypoint tracking given in Stevens and Lewis [110]. The standard design approach was used by considering the longitudinal and lateral states separately as in typical waypoint control schemes. This approach separated the control into three segments: 1) Altitude control, 2) Heading Control and 3) Depth Control. 9.2.1 Altitude Control The first stage considered in the control design to home on a target is the altitude tracking. This stage considers the longitudinal states of the aircraft using the elevator as the control effector. The homography generates the altitude command necessary to track and dock with the refueling receptacle. The architecture for the altitude tracking system is shown in Figure 9-1. The first portion of this system is described as the inner-loop where pitch and pitch rate are used in feedback to stabilize and track a pitch command. Meanwhile, the second portion is referred to as the outer-loop which generates pitch commands for the inner-loop based on the current altitude error. The inner-loop design enables the tracking of a pitch command through proportional Figure 9-1. Altitude hold block diagram control. This pitch command in turn will affect altitude through the changes in forces on the horizontal tail from the elevator position. The two signals used for this inner-loop are pitch and pitch rate. The pitch rate feedback helps with short period damping and allows for rate variations in the transient response. A lead compensator was designed in Stevens et al. [110] to raise the loop gain and to achieve good gain and phase margins for the pitch command to pitch transfer function. The outer-loop design involved closing the loop in altitude. The altitude error signal is generated by the difference in current altitude and the commanded altitude computed by the estimation algorithm. The compensator designed for the inner-loop pitch is augmented to maintain the high loop gain and is defined as G, in Figure 9-1. This structure will provide good disturbance rejection during turbulent conditions. In addition, bounds were placed on the pitch command to alleviate any aggressive maneuvers during the refueling process. 9.2.2 Heading Control The next stage in the control design consist of the turn or heading coordination. This aspect involves the lateral directional states of the aircraft. The control surfaces that effect these states are ailerons and rudder. Similar to the altitude controller, the homography estimates a heading command that steers the aircraft in the desired direction toward the target of interest. The control architecture that accomplishes this objective is depicted in Figure 9-2. The inner-loop Figure 9-2. Heading hold block diagram component of Figure 9-2 deals with roll tracking. The feedback signals include both roll and roll rate through proportional control to command a change in aileron position. The inner-loop LIST OF TABLES Table 3-1 10-1 10-2 10-3 10-4 10-5 10-6 10-7 10-8 10-9 10-10 page .. 64 .... . .23 ..... . 13 .... . 14 .... . .25 .. .126 ..... . 19 .... . 31 ... .. . 33 . .. . 10 . .. .. .. . 10 Solutions for homography decomposition ...... States of the cameras ...... Limits on image coordinates ...... States of the feature points ...... Aircraft states ...... Image coordinates of feature points ....... Effects of camera perturbations on optic flow ...... Effects of camera perturbations on epipolar geometry ..... Effects of camera perturbations on structure from motion ... Maximum variations in position due to parametric uncertainty Maximum variations in attitude due to parametric uncertainty . through Equation 5-37 Xi(t) = (X (t), U(t), a(t), t) (5-35) X (0) = Ko (5-36) Y (t) = g (X (t), a(t), rl, t) (5-37) where U(t) is defined as a set of control inputs to the aircraft and u(t) is a vector containing the camera parameters aT = {ul, a2, ak T for k number of cameras. These equations that utilized feature position will be referred to as the Control Theoretic Form of the governing camera-aircraft equations. Alternatively, if the image plane velocities are employed instead of the image plane positions, as seen in Equation 5-37, then a different set of equations can be obtained which will be referred to as the Optic Flow Form of the governing aircraft-camera equations of motion. This system is given in Equation 5-40, which uses the optic flow expression given in Equations 3-34 and 3-35 as the observations. Xi(t) = (X (t), U(t), a(t), t) (5-38) X(0) = Ko (5-39) J"(t) = 3M(X (t), a(t), rl, t) (5-40) The two system equations just described both have applications to missions involving unmanned aerial vehicles. The Control Theoretic Form primarily applies to missions involving target tracking and surveillance such as aerial refueling and automated visual landing. Meanwhile, the Optic Flow Form is useful for guidance and navigation through unknown environments. The information provided by optic flow reveals magnitude and direction of each feature point in the image which gives a sense of objects in close proximity. Incorporating this information, along with some logic, a control system can be designed to avoid unforeseen obstacles throughout the desired path. x01 t 6x10 05 010 20 30 40 oO 10 20 30 40 Time (sec) Time (sec) A B Figure 10-31. Norm error for target state estimates A) translation and B) rotation ---Target 2300--are 8/ 1 -10i 0 10 20 30 40 -50 10 20 30 40 250 10 20 30 40 Time (sec) Time (sec) Time (sec) A B C Figure 10-32. Closed-loop target position tracking: A) North, B) East, and C) Altitude The components of the position error between the receiver and drogue are shown in Figure 10-33 to illustrate the performance of the tracking controller. These plots depict the initial offset error decaying over time which indicates the receiver's relatives distance is decreasing. The altitude showed a quick climb response where as the response in axial position was a slow steady approach which was desired to limit large changes in altitude and angle of attack. The lateral position is stable for the time period but contains oscillations due the roll to heading lag. The orientation angles shown in Figure 10-34 indicate the Euler angles for for the body-fixed transformations corresponding to the body-fixed frame of the receiver and the body-fixed frame of the drogue. Recall, the only signal being tracked in the control design was heading. This selection allowed the aircraft to steer and maintain a flight trajectory similar to the drogue without aligning roll and pitch. The receiver should fly close to a trim condition rather then matching the full orientation of the drogue, as illustrated in Figure 10-34 for pitch angle. |

Full Text |

PAGE 1 1 PAGE 2 2 PAGE 3 3 PAGE 4 ThisworkwassupportedjointlybyNASAunderNND04GR13HwithSteveJacobsonandJoePahleasprojectmanagersalongwiththeAirForceResearchLaboratoryandtheAirForceOfceofScienticResearchunderF49620-03-1-0381withJohnnyEvers,NealGlassman,SharonHeise,andRobertSierakowskiasprojectmonitors.Additionally,IthankDr.RickLindforhisremarkableguidanceandinspirationthatwilltrulylastalifetime.Finally,IthankmyparentsSandraandJamesCauseyformakingthisjourneypossiblebyprovidingmetheguidanceanddisciplineneededtobesuccessful. 4 PAGE 5 page ACKNOWLEDGMENTS .................................... 4 LISTOFTABLES ....................................... 8 LISTOFFIGURES ....................................... 9 LISTOFTERMS ........................................ 12 ABSTRACT ........................................... 19 CHAPTER 1INTRODUCTION .................................... 21 1.1Motivation ...................................... 21 1.2ProblemStatement ................................. 27 1.3PotentialMissions .................................. 27 1.4SystemArchitecture ................................. 30 1.5Contributions .................................... 33 2LITERATUREREVIEW ................................. 36 2.1DetectionofMovingObjects ............................ 36 2.2StateEstimationUsingVisionInformation ..................... 38 2.2.1Localization ................................. 39 2.2.2Mapping ................................... 39 2.2.3Target-MotionEstimation .......................... 40 2.3ModelingObjectMotion .............................. 41 2.4UncertaintyinVisionAlgorithms .......................... 42 2.5ControlUsingVisualFeedbackinDynamicEnvironments ............ 43 3IMAGEPROCESSINGANDCOMPUTERVISION .................. 45 3.1CameraGeometry .................................. 45 3.2CameraModel .................................... 47 3.2.1IdealPerspective .............................. 47 3.2.2IntrinsicParameters ............................. 48 3.2.3ExtrinsicParameters ............................. 49 3.2.4RadialDistortion .............................. 50 3.3FeaturePointDetection ............................... 51 3.4FeaturePointTracking ............................... 53 3.5OpticFlow ..................................... 56 5 PAGE 6 ............................. 56 3.6.1EpipolarConstraint ............................. 57 3.6.2Eight-PointAlgorithm ............................ 59 3.6.3PlanarHomography ............................. 61 3.6.4StructurefromMotion ............................ 65 4EFFECTSONSTATEESTIMATIONFROMVISIONUNCERTAINTY ........ 67 4.1FeaturePoints .................................... 67 4.2OpticalFlow ..................................... 70 4.3EpipolarGeometry ................................. 71 4.4Homography .................................... 73 4.5StructureFromMotion ............................... 75 5SYSTEMDYNAMICS .................................. 77 5.1DyanmicStates ................................... 77 5.1.1Aircraft ................................... 77 5.1.2Camera ................................... 79 5.2SystemGeometry .................................. 81 5.3NonlinearAircraftEquations ............................ 83 5.4Aircraft-CameraSystem .............................. 84 5.4.1FeaturePointPosition ............................ 85 5.4.2FeaturePointVelocity ............................ 85 5.5SystemFormulation ................................. 86 5.6Simulating ...................................... 89 6DISCERNINGMOVINGTARGETFROMSTATIONARYTARGETS ........ 90 6.1CameraMotionCompensation ........................... 90 6.2Classication .................................... 95 7HOMOGRAPHYAPPROACHTOMOVINGTARGETS ................ 98 7.1Introduction ..................................... 98 7.2StateEstimation ................................... 101 7.2.1SystemDescription ............................. 101 7.2.2HomographyEstimation .......................... 103 8MODELINGTARGETMOTION ............................ 111 8.1Introduction ..................................... 111 8.2DynamicModelingofanObject .......................... 111 8.2.1MotionModels ............................... 112 8.2.2StochasticPrediction ............................ 113 9CONTROLDESIGN ................................... 117 9.1ControlObjectives ................................. 117 6 PAGE 7 ............................... 118 9.2.1AltitudeControl ............................... 118 9.2.2HeadingControl ............................... 119 9.2.3DepthControl ................................ 121 10SIMULATIONS ...................................... 123 10.1Example1:FeaturePointGeneration ........................ 123 10.2Example2:FeaturePointUncertainty ....................... 126 10.2.1Scenario ................................... 126 10.2.2OpticFlow .................................. 128 10.2.3TheEpipolarConstraint ........................... 130 10.2.4StructureFromMotion ........................... 132 10.3Example3:Open-loopGroundVehicleEstimation ................ 133 10.3.1SystemModel ................................ 134 10.3.2Open-loopResults .............................. 135 10.4Example4:Closed-loopAerialRefuelingofaUAV ................ 138 10.4.1SystemModel ................................ 139 10.4.2ControlTuning ............................... 140 10.4.3Closed-loopResults ............................. 144 10.4.4UncertaintyAnalysis ............................ 148 11CONCLUSION ...................................... 151 REFERENCES ......................................... 154 BIOGRAPHICALSKETCH .................................. 164 7 PAGE 8 Table page 3-1Solutionsforhomographydecomposition ....................... 64 10-1Statesofthecameras .................................. 123 10-2Limitsonimagecoordinates .............................. 123 10-3Statesofthefeaturepoints ............................... 124 10-4Aircraftstates ...................................... 125 10-5Imagecoordinatesoffeaturepoints ........................... 126 10-6Effectsofcameraperturbationsonopticow ...................... 129 10-7Effectsofcameraperturbationsonepipolargeometry ................. 131 10-8Effectsofcameraperturbationsonstructurefrommotion ............... 133 10-9Maximumvariationsinpositionduetoparametricuncertainty ............ 150 10-10Maximumvariationsinattitudeduetoparametricuncertainty ............. 150 8 PAGE 9 Figure page 1-1TheUAVeet ...................................... 23 1-2AeroVironment'sMAV:TheBlackWidow ....................... 23 1-3TheUFMAVeet .................................... 24 1-4Refuelingapproachusingtheprobe-droguemethod .................. 28 1-5TrackingapursuitvehicleusingavisionequippedUAV ................ 30 1-6Closed-loopblockdiagramwithvisualstateestimation ................ 31 3-1Mappingfromenvironmenttoimageplane ....................... 46 3-2Imageplaneeldofview(topview) .......................... 46 3-3Radialdistortioneffects ................................. 51 3-4Geometryoftheepipolarconstraint ........................... 58 3-5Geometryoftheplanarhomography .......................... 62 4-1Featurepointdependenceonfocallength ........................ 68 4-2Featurepointdependenceonradialdistortion ..................... 68 5-1Body-xedcoordinateframe .............................. 78 5-2Camera-xedcoordinateframe ............................. 80 5-3Scenarioforvision-basedfeedback ........................... 81 6-1Epipolarlinesacrosstwoimageframes ......................... 91 6-2FOEconstraintontranslationalopticowforstaticfeaturepoints ........... 94 6-3Residualopticowfordynamicenvironments ..................... 95 7-1Systemvectordescription ................................ 102 7-2Movingtargetvectordescription ............................ 103 9-1Altitudeholdblockdiagram ............................... 118 9-2Headingholdblockdiagram .............................. 119 10-1Virtualenvironmentforexample1 ........................... 124 10-2Featurepointmeasurementsforexample1 ....................... 125 9 PAGE 10 ........................ 126 10-4Virtualenvironmentforexample2 ........................... 127 10-5Featurepointsacrosstwoimageframes ........................ 128 10-6Uncertaintyinfeaturepoint ............................... 128 10-7Uncertaintyresultsinopticow ............................ 129 10-8Nominalepipolarlinesbetweentwoimageframes ................... 130 10-9Uncertaintyresultsforepipolargeometry ........................ 131 10-10Nominalestimationusingstructurefrommotion .................... 132 10-11Uncertaintyresultsforstructurefrommotion ...................... 133 10-12Vehicletrajectoriesforexample3 ............................ 134 10-13PositionstatesoftheUAVwithon-boardcamera ................... 135 10-14AttitudestatesoftheUAVwithon-boardcamera ................... 135 10-15Positionstatesofthereferencevehicle ......................... 135 10-16Attitudestatesofthereferencevehicle ......................... 136 10-17Positionstatesofthetargetvehicle ........................... 136 10-18Attitudestatesofthetargetvehicle ........................... 136 10-19Normerror ........................................ 137 10-20Relativepositionstates ................................. 137 10-21Relativeattitudestates .................................. 137 10-22Virtualenvironment ................................... 138 10-23Inner-looppitchtopitchcommandBodeplot ..................... 141 10-24Pitchanglestepresponse ................................ 141 10-25Altitudestepresponse .................................. 142 10-26Inner-looprolltorollcommandBodeplot ....................... 143 10-27Rollanglestepresponse ................................. 144 10-28Headingresponse .................................... 144 10-29Open-loopestimationoftarget'sinertialposition .................... 145 10 PAGE 11 .................... 145 10-31Normerrorfortargetstateestimates .......................... 146 10-32Closed-looptargetpositiontracking .......................... 146 10-33Positiontrackingerror .................................. 147 10-34Targetattitudetracking ................................. 147 10-35Trackingerrorinheadingangle ............................. 147 10-36Target'sinertialpositionwithuncertaintybounds ................... 149 10-37Target'sinertialattitudewithuncertaintybounds .................... 149 11 PAGE 12 12 PAGE 13 13 PAGE 15 15 PAGE 18 18 PAGE 19 19 PAGE 20 20 PAGE 21 1 ].Thisincreaseincapabilityforsuchcomplextasksrequirestechnologyformoreadvancedsystemstofurtherenhancethesituationalawareness.Overthepastseveralyears,theinterestanddemandforautonomoussystemshasgrownconsiderably,especiallyfromtheArmedForces.Thisinteresthasleveragedfundingopportunitiestoadvancethetechnologyintoastateofrealizablesystems.Sometechnicalinnovationsthathaveemergedfromtheseefforts,fromahardwarestandpoint,consistmainlyofincreasinglycapablemicroprocessorsinthesensors,controls,andmissionmanagementcomputers.TheDefenseAdvancedResearchProjectsAgency(DARPA)hasfundedseveralprojectspertainingtotheadvancementofelectronicdevicesthroughsizereduction,improvedspeedandperformance.Fromthesedevelopments,thecapabilityofautonomoussystemhasbeendemonstratedonvehicleswithstrictweightandpayloadrequirements.Inessence,thecurrenttechnologyhasmaturedtoapointwhereautonomoussystemsarephysicallyachievableforcomplexmissionsbutnotyetalgorithmicallycapable.TheaerospacecommunityhasemployedmanyoftheresearchdevelopedforautonomoussystemsandappliedittoUnmannedAerialVehicles(UAV).Manyofthesevehiclesarecurrently 21 PAGE 22 1 ].FuturemissionsenvisionUAVtoconductmorecomplextasksuchasterrainmapping,surveillanceofpossiblethreats,maritimepatrol,bombdamageassessment,andeventuallyoffensivestrike.Thesemissionscanspanovervarioustypesofenvironmentsand,therefore,requireawiderangeofvehicledesignsandcomplexcontrolstoaccommodatetheassociatedtasks.TherequirementsanddesignofUAVareconsideredtoenableaparticularmissioncapability.Eachmissionscenarioisthedrivingforceoftheserequirementsandaredictatedbyrange,speed,maneuverability,andoperationalenvironment.CurrentUAVrangeinsizefromlessthan1poundtoover40,000pounds.SomepopularUAVthatareoperational,intestingphase,andintheconceptphasearedepictedinFigure 1-1 toillustratethevariousdesigns.ThetwoUAVontheleft,GlobalHawkandPredator,arecurrentlyinoperation.GlobalHawkisemployedasahighaltitude,longendurancereconnaissancevehiclewhereasthePredatorisusedforsurveillancemissionsatloweraltitudes.Meanwhile,theremainingtwopicturespresentJ-UCAS,whichisajointcollaborationforboththeAirForceandNavy.ThisUAVisdescribedasamediumaltitudeyerwithincreasedmaneuverabilityoverGlobalHawkandthePredatorandisconsideredforvariousmissions,someofwhichhavealreadybeendemonstratedinight,suchasweapondeliveryandcoordinatedight.Theadvancementsinsensorsandcomputingtechnology,mentionedearlier,hasfacilitatedtheminiaturizationoftheseUAV,whicharereferredtoasMicroAirVehicles(MAV).Thescaleofthesesmallvehiclesrangesfromafewfeetinwingspandowntoafewinches.DARPAhasalsofundedtherstsuccessfulMAVprojectthroughAeroVironment,asshowninFigure 1-2 ,wherebasicautonomywasrstdemonstratedatthisscale[ 2 ].Thesesmallscalesallowhighlyagilevehiclesthatcanmaneuverinandaroundobstaclessuchasbuildingsandtrees.ThiscapabilityenablesUAVtooperateinurbanenvironments,belowrooftoplevels,toprovide 22 PAGE 23 TheUAVeet thenecessaryinformationwhichcannotbeobtainedathigheraltitudes.ResearchersarecurrentlypursuingMAVtechnologytoaccomplishtheverysamemissionsstatedearlierfortheuniqueapplicationofoperatingatlowaltitudesinclutteredenvironments.Assensorandcontroltechnologiesevolve,theseMAVcanbeequippedwiththelatesthardwaretoperformadvancedsurveillanceoperationswherethedetection,tracking,andclassicationofthreatsaremonitoredautonomouslyonline.Althoughasinglemircoairvehiclecanprovidedistinctinformation,targetsmaybedifculttomonitorduetobothightpathandsensoreldofviewconstraints.ThislimitationhasmotivatedtheideaofacorporativenetworkoraswarmofMAVcommunicatingandworkingtogethertoaccomplishacommontask. Figure1-2. AeroVironment'sMAV:TheBlackWidow 23 PAGE 24 3 4 ].Meanwhile,Stanfordhasexaminedmotionplanningstrategiesthatoptimizeighttrajectoriestomaintainsensorintegrityforimprovedstateestimation[ 5 ].TheworkatGeorgiaTechandBYUhasconsideredcorporativecontrolofMAVforautonomousformationying[ 6 ]andconsensusworkfordistributedtaskassignment[ 7 ].Alternatively,visionbasedcontrolhasalsobeenthetopicofinterestatbothGeorgiaTechandUF.ControlschemesusingvisionhavebeendemonstratedonplatformssuchasahelicopteratGeorgiaTech[ 8 ],whileUFimplementedaMAVthatintegratedvisionbasedstabilizationintoanavigationarchitecture[ 9 10 ].TheUniversityofFloridahasalsoconsideredMAVdesignsthatimprovetheperformanceandagilityofthesevehiclesthroughmorphingtechnology[ 11 13 ].FabricationfacilitiesatUFhaveenabledrapidconstructionofdesignprototypesusefulforbothmorphingandcontroltesting.TheeetofMAVproducedbyUFareillustratedinFigure 1-3 wherethewingspanofthesevehiclesrangefrom24indownto4in. Figure1-3. TheUFMAVeet ThereareanumberofcurrentdifcultiesassociatedwithMAVduetotheirsize.Forexample,characterizingtheirdynamicsunderightconditionsatsuchlowReynoldsnumbersisanextremelychallengingtask.Theconsequenceofincreasedagilityatthisscalealsogivesrisetoerraticbehaviorandaseveresensitivitytowindgustandotherdisturbances.Waszaketal.[ 14 ]performedwindtunnelexperimentson6inchMAVandobtainedtherequiredstabilityderivativesforlinearandnonlinearsimulations.AnothercriticalchallengetowardMAV 24 PAGE 25 25 PAGE 26 3 .Thisdissertationwillfocusonthemonocularcameracongurationtoaddressthestateestimationproblemregardingmovingtargets.TheadvantageofthesetechniquesbecomesmoreapparenttoUAVwhenappliedtoguidance,navigation,andcontrol.Bymountingacameraonavehicle,stateestimationofthevehicleandobjectsintheenvironmentcanbeachievedinsomeinstancesthroughvisionprocessing.Oncestateestimatesareknown,theycanthenbeusedinfeedback.Controltechniquescanthenbeutilizedforcomplexmissionsthatrequirenavigation,pathplanning,avoidance,tracking,homing,etc.Thisgeneralframeworkofvisionprocessingandcontrolhasbeensuccessfullyappliedtovarioussystemsandvehiclesincludingroboticmanipulators,groundvehicles,underwatervehicles,andaerialvehiclesbuttherestillexistssomecriticallimitations.Theproblematicissueswithusingvisionforstateestimationinvolvescameranonlinearities,cameracalibration,sensitivitytonoise,largecomputationaltime,limitedeldofview,andsolvingthecorrespondenceproblem.Aparticularsetoftheseimageprocessingissueswillbeaddresseddirectlyinthisdissertationtofacilitatethecontrolofautonomoussystemsincomplexsurroundings. 26 PAGE 27 1. segmentingmovingtargetsfromstationarytargetswithinthescene 2. classifyingmovingtargetsintodeterministicandstochasticmotions 3. couplingthevehicledynamicsintothesensorobservations(i.e.images) 4. formulatingthehomographyequationsbetweenamovingcameraandtheviewabletargets 5. propagatingtheeffectsofuncertaintythroughthestateestimationequations 6. establishingcondenceboundsontargetstateestimationThedesignandimplementationofavision-basedcontrollerisalsopresentedinthisdissertationtoverifyandvalidatemanyoftheconceptspertainingtotrackingofmovingtargets. 27 PAGE 28 15 ].Thedrogueisdesignedinanaerodynamicshapethatpermitstheextensionfromthetankerwithoutinstability.Theprobe-and-droguemethodisconsideredthepreferredmethodforAAR,mainlyduetothehighpilotworkloadincontrollingtheboom[ 16 ].Figure 1-4 illustratestheviewobservedbyreceiveraircraftduringtherefuelingprocesswherefeaturepointshavebeenplacedonthedrogue. Figure1-4. Refuelingapproachusingtheprobe-droguemethod VisioncanbeusedtofacilitatetheAARproblembyaugmentingtraditionalaircraftsensorssuchasglobalpositioningsystem(GPS)andinertialmeasurementunit(IMU).GighprecisionGPS/IMUsensorscanproviderelativeinformationbetweenthetankerandthereceiverthenvisioncanbeusedtoproviderelativeinformationonthedrogue.Theadvantagetovisioninthiscaseisitspassivenaturewhicheliminatessensoremissionsduringrefuelingoverenemyair 28 PAGE 29 1-5 illustratesinasimulatedenvironmentthisscenariowhereaUAVobservesthe 29 PAGE 30 Figure1-5. TrackingapursuitvehicleusingavisionequippedUAV 1-6 ,wherecommandsaresenttoavehiclebasedonthemotionsobservedintheimages.ThevehicleconsideredinthisdissertationispredominatelyassumedanautonomousUAV,butisgeneralizedforanydynamicalsystemwithpositionandorientationstates.TheblockspertainingtothisdissertationarehighlightedinFigure 1-6 intheimageprocessingblockandconsistsofthemovingobjectdetection,stateestimationofamovingobject,andclassifyingdeterministicversusstochasticmotion.Abriefdiscussionofeachtopicisdescribedinthissection,whilethedetailsarecoveredintheirrespectivechapters.Distinguishingmovingobjectsfromstationaryobjectswithamovingcameraisachallengingtaskinvisionprocessingandistherststepinthestateestimationprocesswhenconsideringadynamicscene.Thisinformationisextremelyimportantforguidance,navigation,andcontrolofautonomoussystemsbecauseitidentiesobjectsthatpotentiallycouldbeinapathforcollision.Forastationarycamera,movingobjectsinthescenecanbeextractedusing 30 PAGE 31 Closed-loopblockdiagramwithvisualstateestimation simpleimagedifferencing,wherethestationarybackgroundissegmentedout;however,thisapproachdoesnotapplytomovingcameras.Inthecaseofamovingcamera,thebackgroundisnolongerstationaryanditbeginstochangeovertimeasthevehicleprogressesthroughtheenvironment.Therefore,theimagestakenbyamovingcameracontainthemotionduetothecamera,commonlycalledego-motion,andthemotionoftheobject.Techniquesthatinvolvecameramotioncompensationorimageregistrationhavebeenproposedtoworkwellwhenthereexistsnostationaryobjectsclosetothecamerawhichcausehighparallax.Thisdissertationwillestablishatechniquetoclassifyobjectsintheeldofviewasmovingorstationarywhileaccountingforstationaryobjectswithhighparallax.Therefore,withaseriesofobservationsofaparticularscene,onecandeterminewhichobjectsaremovingintheenvironment.Knowingwhichobjectsaremovingintheimagedictatesthetypeofimageprocessingrequiredtoaccuratelyestimatetheobject'sstates.Infact,theestimationproblembecomesinfeasibleforamonocularsystemwhenboththecameraandtheobjectaremoving.Thisunattainablesolutioniscausebyanumberoffactorsincluding1)inabilitytodecouplethemotionfromthecameraandtargetand2)failuretotriangulatethedepthestimateoftheobject.Forthisconguration,relativeinformationcanbeobtainedandfusedwithadditionalinformationforstateestimation.First,decouplingthemotionrequiresknowninformationregardingmotionofthecameraorthemotionoftheobject,whichcouldbeobtainedthroughothersensorssuch 31 PAGE 32 5 ].Furthermore,theaccuracyofthestateestimatesbecomespoorforsmallbaselinecongurations,whichoccursforMAVusingstereovision.Theseissuesregardingtargetstateestimationwillbeconsideredinthisdissertationtoshowboththecapabilitiesandlimitationstowardautonomouscontrolandnavigation.Anotherimportanttaskinvolvedwithtargetestimationistodetermineapattern(ifany)intheobject'smotionbasedonthetimehistory.Theobjectscanthenbeclassiedintodeterministicandstochasticmotionsaccordingtopastbehavior.Withthisinformation,predictionmodelscanbemadebasedonpreviousimagestoestimatethepositionofanobjectatalatertimewithsomelevelofcondence.Thepredictedestimatescanthenbeusedinfeedbackfortrackingordockingpurposes.Forstochasticlyclassiedobjects,furtherconcernsregardingdockingorAARareimposedonthecontrolproblem.Theprimarytaskofstateestimation,forboththevehicleandobjectsintheenvironment,reliesonaccurateknowledgeoftheimagemeasurementsandtheassociatedcamera.Suchknowledgeisdifculttoobtainduetouncertaintiesinthesemeasurementsandtheinternalcomponentsofthecameraitself.Forinstance,theimagemeasurementscontainuncertaintiesassociatedwiththedetectionofobjectsintheimage,inadditiontonoisecorruption.Thesedrawbackshavepromptedmanyrobustalgorithmstoincreasetheaccuracyoffeaturedetectionwhilehandlingnoiseduringtheestimationprocess.Alternatively,manytechniqueshavebeenusedtoaccuratelyestimatetheinternalparametersofthecamerathroughcalibration.Theparametersthatdescribetheinternalcomponentsofthecameraarereferredtoasintrinsicparametersandtypicallyconsistoffocallength,radialdistortion,skewfactor,pixelsize,andopticalcenter.Thiscalibrationprocesscanbecomecumbersomeforalargenumberofcameras 32 PAGE 33 33 PAGE 34 34 PAGE 35 35 PAGE 36 1-6 illustratesthecomponentsofinterestdescribedinthisdissertationforstateestimationandtrackingcontrolwithrespecttoamovingobjectwhichinvolvesobjectmotiondetection,objectstateestimation,andobjectmotionmodelingandprediction.Theliteraturereviewofthesetopicsisgiveninthissection. 17 18 ]hasservedasafoundationformanyalgorithms.Thistechniquereliesonasmoothnessconstraintimposedontheopticowthatmaintainsaconstantintensityacrosssmallbase-linemotionofthecamera.Manytechniqueshavebuiltuponthisalgorithmtoincreaserobustnesstonoiseandoutliers.Oncefeaturetrackinghasbeenobtained,thenextprocessinvolvessegmentingtheimageformovingobjects.Theneedforsuchaclassicationisduethefactthatstandardimage 36 PAGE 37 19 21 ].TechniquesformorerealisticapplicationsinvolveKalmanltering[ 22 ]toaccountforlightingconditionsandbackgroundmodelingtechniquesusingstatisticalapproaches,suchasexpectationmaximizationandmixtureofGaussian,toaccountforothervariationsinreal-timeapplications[ 23 28 ].Althoughthesetechniquesworkwellforstationarycameras,theyareinsufcientforthecaseofmovingcamerasduetothemotionofthestationarybackground.Motiondetectionusingamovingcamera,asinthecaseofacameramountedtoavehicle,becomessignicantlymoredifcultbecausethemotionviewedintheimagecouldresultfromanumberofsources.Forinstance,acameramovingthroughascenewillviewmotionsintheimagecausedbycamerainducedmotion,referredtoasegomotion,changesincameraintrinsicparameterssuchaszoom,andindependentlymovingobjects.Therearetwoclassesofproblemsconsideredinliteratureforaddressingthistopic.Therstconsidersthescenariowherethe3Dcameramotionisknownapriorithencompensationcanbemadetoaccountforthismotiontodeterminestationaryobjectsthroughanappropriatetransformation[ 29 30 ].Thesecondclassofproblemsdoesnotrequireknowledgeofthecameramotionandconsistsofatwostageapproachtothemotiondetection.Therststageinvolvescameramotioncompensationwhilethelaststageemploysimagedifferencingontheregisteredimage[ 31 ]toretrievenon-staticobjects.ThetransformationusedtoaccountforcameramotioniscommonlysolvedbyassumingthemajorityofimageconsistsofadominantbackgroundthatisstationaryinEuclideanspace[ 32 33 ].Thissolutionisobtainedthroughaleast-squaresminimizationprocess[ 32 ]orwiththeuseofmorphologicallters[ 34 ].Thetransformationsobtainedfromthesetechniquestypically 37 PAGE 38 35 ]proposedauniedmethodtodetectmovingobjects.Thisproposedmethodhandlesvariouslevelsofparallaxintheimagethroughasegmentationprocessthatisperformedinlayers.Therstlayerextractsthebackgroundobjectswhicharefarawayfromthecameraandhavelowparallaxthroughageneraltransformationinvolvingcamerarotation,translation,andzoomthroughimagedifferencing.Thenextlayercontainstheobjectwithhighparallaxconsistingofbothobjectsclosetothecameraandobjectsthataremovingindependentlyofthecamera.Theparallaxisthencomputedfortheremainingpixelsandcomparedtoonepixel.Thisprocessseparatestheobjectswithintheimagebasedontheircomputedparallax.Theselectionmayinvolvechoosingapointonaknownstationaryobjectthatcontainshighparallaxsoanyobjectnotobeyingthisparallaxisclassiedasamovingobjectinthescene.Opticowtechniquesarealsousedtoestimatemovingtargetlocationsonceego-motionhasbeenestimated.Amethodthatcomputesthenormalimageowhasbeenshowntoobtainmotiondetection[ 36 ].Coordinatetransformationsaresometimesusedtofacilitatethisapproachtodetectingmotion.Forinstance,amethodusingcomplexlogmappingwasshowntotransformtheradialmotionsintohorizontallinesuponwhichverticalmotionindicateindependentmotion[ 37 ].Alternatively,sphericalmappingwasusedgeometricallytoclassifymovingobjectsbysegmentingmotionswhichdonotradiatefromthefocusofexpansion(FOE)[ 29 ]. 38 PAGE 39 38 39 ]usedthecoplanarityconstraintalsoknownastheepipolarconstraint.Meanwhile,thesubspaceconstrainthasalsobeenemployedtolocalizecameramotion[ 40 ].Thesetechniqueshavebeenappliedtonumeroustypesofautonomoussystems.Themobileroboticcommunityhasappliedthesetechniquesforthedevelopmentofnavigationinvariousscenarios[ 41 45 ].TheapplicationshavealsoextendedintotheresearchofUAVforaircraftstateestimation.GurlandRotstein[ 46 ]wasthersttoextendthisapplicationintheframeworkofanonlinearaircraftmodel.Thisapproachusedopticalowinconjunctionwiththesubspaceconstrainttoestimatetheangularratesoftheaircraftandwasextendedin[ 47 ].Webbetal.[ 48 49 ]employedtheepipolarconstrainttotheaircraftdynamicstoobtainvehiclestates.ThefoundationforbothoftheseapproachesisaKalmanlterinconjunctionwithageometricconstrainttoestimatethecameramotion.SomeapplicationsforaircraftstateestimationhaveinvolvedmissionsforautonomousUAVsuchasautonomousnightlanding[ 50 ]androadfollowing[ 51 ]. 52 58 ].TheseapproachesemploythesubspaceconstrainttoreconstructfeaturepointpositionthroughanextendedKalmanlter.Severalsurveypapershavebeenpublisheddescribingthecurrentalgorithmswhilecomparingtheperformanceandrobustness[ 59 62 ].RobustandadaptivetechniqueshavebeenproposedthatuseanadaptiveextendedKalmanltertoaccountformodeluncertainties[ 63 ].Inaddition,Qianetal.[ 64 ]designedarecursiveHltertoestimatestructurefrommotioninthepresenceofmeasurementandmodeluncertaintieswhile 39 PAGE 40 65 ]investigatedtheoptimalapproachestotargetstateestimationanddescribedtheeffectsoflinearsolutionsonvariousnoisedistributions. 66 ].ThismethodextendedthepreviousworkofBroidaetal.[ 67 ]thatonlyconsideredafeaturepointapproach.Forthecaseofmovingmonocularcameraconguration,theproblembecomesextremelydifcultduetotheadditionalmotionofthecamera.Oneapproachusedinliteraturerelevanttomonocularcamerasystemsisbearings-only-tracking.Inthisapproach,thereareseveralassumptionsmade:(i)thevehiclehasknowledgeofitsposition,(ii)anadditionalrangesensor,suchassonarorlaserrangender,isusedtoprovideabearingmeasurement,and(iii)animagemeasurementistakenforanestimateoflateralposition.Theinitialresearchhasinvolvedtheestimationprocessanddesignwithimprovementstotheperformance[ 68 72 ].ThisapproachwasimplementbyFlew[ 5 ]toestimatethemotionoftargetwithinacomputedcovariance.Guanghuietal.[ 73 ]providedamethodforestimatingthemotionofapointtargetfromknowncameramotion.Theroboticcommunityhasexaminedthetarget-motionestimationproblemfromavisualservocontrolframework.Trackingrelativemotionofamovingtargethasbeenshownusinghomography-basedmethods.Thesemethodshavebeendemonstratedtocontrolanautonomousgroundvehicletoadesiredposedenedbyagoalimage,wherethecamerawasmountedonthegroundvehicle[ 74 ].Chenetal.[ 75 76 ]regulatedagroundvehicletoadesiredposeusingastationaryoverheadcamera.Mehtaetal.[ 77 ]extendedthisconceptforamovingcamera,whereacamerawasmountedtoanUAVandagroundvehiclewascontrolledtoadesiredpose. 40 PAGE 41 15 ]appliedavisionnavigationalgorithmcalledVisNAVthatwasdevelopedbyJunkinsetal.[ 78 ]toestimatethecurrentrelativepositionandorientationofthetargetdroguethroughaGaussianleast-squaresdifferentialcorrectionalgorithm.Thisalgorithmhasalsobeenappliedtospacecraftformationying[ 79 ]. 80 ]demonstratedthecontrolrequiredtograbanunknownmovingobjectwithroboticmanipulatorusinganauto-regressive(AR)model.Thismodelpredictsafuturepositionofthetargetbasedonvelocityestimatescomputedfromimagesequences.Foraerialvehicles,detectingotheraircraftintheskyiscriticalforcollisionavoidance.NASAhasconsideredvisioninthisscenariotoaidpilotsindetectingaircraftonacrossingtrajectory.AtechniquecombiningimageandnavigationdataestablishedapredictionmethodthroughaKalmanlterapproachtoestimatethepositionandvelocityofthetargetaircraftaheadintime[ 34 ].Similarly,theAARproblemrequiressomeformofmodelpredictionwhendockingtoamovingdrogue.Kimmettetal.[ 15 ]utilizedadiscretelinearmodelforthepredictionofthedrogue.Thepredictedstatesusedforcontrolwerecomputedusingthediscretemodel,thecurrentstates,andlightturbulenceasinputtothedroguedynamics.Successfuldockingwassimulatedforonlylightturbulenceandwithlowfrequencydynamicsimposedonthedrogue.NASAisextremelyinterestedinAARproblemandcurrentlyhasaprojectonthistopic.FlighttestshavebeenconductedbyNASAinanattempttomodelthedroguedynamics[ 81 ].Inthisstudy,theaerodynamiceffectsfromboththereceiveraircraftandthetankeraircraftwereexaminedonthe 41 PAGE 42 82 ].Robustnesswasalsoanalyzedusingaleast-squaresolutiontoobtainanexpressionfortheerrorintermsofthemotionvariables[ 83 ].Theuncertaintyinvision-basedfeedbackisoftenchosenasvariationswithinfeaturepoints;however,uncertaintyinthecameramodelmayactuallybeanunderlyingsourceofthosevariations.Essentially,theuncertaintymaybeassociatedwiththeimageprocessingtoextractfeaturepointsorwiththecameraparametersthatgeneratedtheimage.Thepropercharacterizationofcamerauncertaintymaybecriticaltodeterminearealisticleveloffeaturepointuncertainty.Theanalysisofcamerauncertaintyistypicallyaddressedinaprobabilisticmanner.Alineartechniquewaspresentedthatpropagatesthecovariancematrixofthecameraparametersthroughthemotionequationstoobtainthecovarianceofthedesiredcamerastates[ 84 ].Ananalysiswasalsoconductedfortheepipolarconstraintbasedontheknowncovarianceinthecameraparameterstocomputethemotionuncertainty[ 85 ].AsequentialMonteCarlotechniquedemonstratedbyQianetal.[ 86 ]proposedanewstructurefrommotionalgorithmbasedonrandomsamplingtoestimatetheposteriordistributionsofmotionandstructureestimation.Theexperimentalresultsinthispaperrevealedsignicantchallengestowardsolvingforthestructureinthepresenceoferrorsincalibration,featurepointtracking,featureocclusion,andstructureambiguities. 42 PAGE 43 87 ]intrafcsituations,lowaltitudeightofarotorcraft[ 88 ],avoidingobstaclesintheightpathofanaircraft[ 34 ],andnavigatingunderwatervehicles[ 89 ].Opticalowtechniqueshavealsobeenutilizedasatoolforavoidancebysteeringawayfromareaswithhighopticowwhichindicateregionsofcloseobstacles[ 90 ].Targettrackingisanotherdesiredcapabilityforautonomoussystems.Inparticular,themilitaryisinterestedinthistopicforsurveillancemissionsbothintheairandontheground.Thecommonapproachestotargettrackingoccurinbothfeaturepointandopticalowtechniques.Thefeaturepointmethodtypicallyconstrainsthetargetmotionintheimagetoadesiredlocationbycontrollingthecameramotion[ 91 92 ].Meanwhile,Frezzaetal.[ 93 ]imposedanonholonomicconstraintonthecameramotionandusedapredictiveoutput-feedbackcontrolstrategybasedontherecursivetrackingofthetargetwithfeasiblesystemtrajectories.Alternatively,opticalowbasedtechniqueshavebeenpresentedforrobotichand-in-eyecongurationtotracktargetsofunknown2Dvelocitieswherethedepthinformationis 43 PAGE 44 94 ].Adaptivesolutionspresentedin[ 91 95 97 ]haveshowncontrolsolutionsfortargettrackingwithuncertaincameraparameterswhileestimatingdepthinformation.Thehomingcontrolproblemhasnumerousapplicationstowardautonomoussystemssuchasautonomousaerialrefueling,spacecraftdocking,missileguidance,andobjectretrievalusingarobtoticmanipulator.Kimmettetal.[ 15 98 ]developedacandidateautonomousprobe-and-drogueaerialrefuelingcontrollerthatusesacommandgeneratortracker(CGT)totracktime-varyingmotionsofanon-stationarydrogue.TheCGTisanexplicitmodelfollowingcontroltechniqueandwasdemonstratedinsimulationforamovingdroguewithknowndynamicssubjecttolightturbulence.Tandaleetal.[ 16 ]extendedtheworkofKimmettandValasekbydevelopingareferenceobserverbasedtrackingcontroller(ROTC)whichdoesnotrequireadroguemodelorpresumedknowledgeofthedrogueposition.Thissystemconsistofareferencetrajectorygenerationmodulethatsendscommandstoanobserverthatestimatesthedesiredstatesandcontrolfortheplant.Theinputtothiscontrolleristherelativepositionbetweenthereceiveraircraftandthedroguemeasuredbythevisionsystem.Asimilarvisionapproachtoaerialrefuelingisalsopresentedin[ 99 ],wheremodelsofthetankeranddrogueareusedinconjunctionwithaninferredcamera.Thedroguemodelusedinthispaperwastakenfrom[ 100 ]thatusesamulti-segmentapproachtoderivingthedynamicsofthehose.Meanwhile,Houshangietal.[ 80 ]consideredgraspingamovingtargetbyadaptivelycontrollingarobotmanipulatorusingvisioninteraction.Theadaptivecontrolschemewasusedtoaccountformodelingerrorsinthemanipulator.Inaddition,thispaperconsideredunknowntargetdynamics.Anauto-regressivemodelapproachwasusedtopredictthetarget'spositionbasedonpassedvisualinformationandanestimatedtargetvelocity.Experimentaltestcasesaredocumentedthatshowtrackingconvergence. 44 PAGE 45 3-1 .Thevector,h,representsthevectorbetweenthecameraandafeaturepointintheenvironmentrelativetoadenedcamera-xedcoordinatesystem,asdenedbyI.ThisvectoranditscomponentsarerepresentedinEquation 3 45 PAGE 46 Mappingfromenvironmenttoimageplane Amajorconstraintplacedonthissensoristhecamera'seldofview(FOV).HeretheFOVcanbedescribedasthe3Dregionforwhichfeaturepointsarevisibletothecamera;hence,featuresoutsidetheFOVwillnotappearintheimage.Thethreephysicalparametersthatdenethisconstraintaretheeldofdepth,thehorizontalangleandtheverticalangle.AtopviewillustrationoftheFOVcanbeseeninFigure 3-2 ,wherethehorizontalFOVisdenedbythehalfangle,gh,andthedistancetotheimageplaneisoflengthf.Likewise,asimilarplotcanbeshowntoillustratetheverticalangle,whichcanbedenedasgv. Imageplaneeldofview(topview) 46 PAGE 47 3 ,whererh;visdenedasthelargestspatialextensioninthehorizontalandverticaldirections. 3 forthehorizontalcomponent. 3 fortheverticalangle. 3.2.1IdealPerspectiveAgeometricrelationshipbetweenthecamerapropertiesandafeaturepointisrequiredtodeterminetheimageplanecoordinates.Thisrelationshipismadebyrstseparatingthecomponentsofhthatareparalleltotheimageplaneintotwodirections.Theimageplanecoordinatesarethencomputedfromatangentrelationshipofsimilartrianglesbetweentheverticalandhorizontaldirectionsandthedepthwithascalefactoroffocallength.Thisrelationshipestablishesthestandard2Dimageplanecoordinatesreferredtoasthepin-holecameramodel[ 101 102 ].Equations 3 and 3 representageneralpin-holeprojectionmodel 47 PAGE 48 3 and 3 reducetotheverycommonpin-holecameramodelandisrepresentedbyEquations 3 and 3 3 and 3 canbeexpressedinhomogeneouscoordinatesandisshowninEquation 3 3 .First,theimageplaneisdiscretizedintoasetofpixels,correspondingtotheresolutionofthecamera.Thisdiscretizationisbasedonscalefactorsthatrelatereal-worldlengthmeasuresintopixelunitsforboththehorizontalandverticaldirections.Thesescalingtermsaredenedassandsnwhichhaveunitsofpixelsperlength,wherethelengthcouldbeinfeetormeters.Ingeneral,thesetermsaredifferentbutwhenthepixelsaresquarethens=sn.Second,theoriginoftheimageplaneistranslatedfromthecenterofthe 48 PAGE 49 3 ,wherepixelmapping,origintranslation,andskewnessareallconsidered. 3 isrewrittentoEquation 3 3 toobtainageneralequationthatmapsfeaturepointsintheinertialframetocoordinatesintheimageplaneforacalibratedcamera. 49 PAGE 50 3 ,requiresaninniteseriesoftermstoapproximatethevalue. 3 and 3 ,mapsanundistortedimage,(0;n0),whichisnotmeasurableonaphysicalcamera,intoadistortedimage,(0d;n0d),whichisobservable[ 104 ].Thisdistortionmodelonlyconsidersthersttermintheinniteseriestodescriberadialdistortionandexcludestangentialdistortion.Thisapproximationindistortionhasbeenusedtogenerateanaccuratedescriptionofrealcameraswithoutadditionalterms[ 105 ], 3-1 ,attemptstomodelthecurvatureofthelensduringtheimageplanemapping.Thisdistortionintheimageplanevariesinanonlinearfashionbasedonposition.Thiseffectdemonstratesanaxisymmetricmappingthatincreasesradiallyfromtheimagecenter.AnexamplecanbeseeninFigure 3-3B and 3-3C whichillustrateshowradialdistortionchangesfeaturepointlocationsofaxedpatternintheimagebycomparingittoatypicalpin-holemodelshowninFigure 3-3A .Noticethedistortedimagesseemtotakeonaconvexorconcaveshapedependingonthesignofthedistortion. 50 PAGE 51 B CFigure3-3. RadialDistortionEffectsforA)f=0:5d=0,B)f=0:5d=0:0005,andC)f=0:5d=+0:0005 3 .Assuch,theseparametersaretermedtheintrinsicparametersandarefoundthroughcalibration.Afeaturepointmustbeanalyzedwithrespecttotheseintrinsicparameterstoensureproperstateestimation.Theradialdistancefromafeaturepointtothecenteroftheimageisdependentonboththerelativepositionsofcameraandfeaturealongwiththefocallength.Thisradialdistanceisalsorelatedviaanonlinearrelationshiptotheradialdistortion.Clearlyanyanalysisofthefeaturepointswillrequireestimationofthecameraparameters.Chapter 4 willdiscussatechniquethatconsidersboundeduncertaintytowardtheintrinsicparametersandestablishesaboundedconditiononthefeaturepointpositions. 51 PAGE 52 3 and 3 [ 102 103 ].Theimagecoordinates(;n)intheseexpressionsarecomputedusingeitherEqaution 3 orEquation 3 dependingonthecameramodel. 3 [ 102 103 ].Thepixelvalueswithinthesearchwindowaredenedasx. 3 .IfEquation 3 issatisedthenthisisavalidfeaturepointbasedontheuserscriterion[ 102 103 ].Thisselectionisafunctionofboththewindowsize,W,andthethreshold,t. 52 PAGE 53 106 ].ThismethodcanbeextendedtoedgedetectionbyconsideringthestructureofthesingularvaluesofG.AnexampleofthisalgorithmistheCannyedgedetector[ 107 ]. 3 3 3 53 PAGE 54 3 .Oneimportantlimitationofthiscriterionoccurswhenthewindowinbothimagescontainsrelativelyconstantintensityvalues.Thisresultsintheapertureproblemwhereanumberofsolutionsforhareobtained.Therefore,duringthefeatureselectionprocessit'sbenecialtochoosefeaturesthatcontainuniqueinformationinthiswindow. 3 forsmallbaselinetracking:(1)usingthebrightnessconsistencyconstraintand(2)applyingthesumofsquareddifferences(SSD)approach.Eachofthesetechniquesemploysatranslationalmodeltodescribetheimagemotion.Therefore,ifoneassumesasimpletranslationalmodelthenthegeneraltransformationisshowninEquation 3 3 intoEquation 3 whileinitiallyneglectingthenoiseterm.ApplyingtheTaylorseriesexpansiontothisexpressionaboutthepointofinterest,x,whileretainingonlythersttermintheseriesresultsinEquation 3 dt+I 3 inmatrixformresultsinEquation 3 dt;dn 3 constitutes1equationwith2unknownvelocities;therefore,anotherconstraintisneededtosolvethisproblem.Auniquesolutionforthevelocitiescanbedeterminedbyenforcinganadditionalconstraintontheproblem,whichentailsrestrainingregionstoalocalwindowthatmovesatconstantvelocity.Upontheseassumptiononecanminimizetheerror 54 PAGE 55 3 3 3 .Thenalsolutionforthepixelvelocityisfoundthroughaleast-squaresestimategiveninEquation 3 .Theseimagevelocitiesarealsoreferredtoastheopticow.Oncetheopticowiscomputedforafeaturepointthentheimagedisplacementforfeaturetrackingistrivialtond. 3 ,attemptstoestimatetheDxwhilenotrequiringthecomputationofimagegradients.Thisapproachalsoemploysthetranslationalmodeloverawindowedregion.Themethodconsidersthepossiblerangethatwindowcouldmove,danddn,inthetime,dt.Thisconsistencyconstraintthenleadstoaproblemofminimizingtheerroroverthepossiblewindowswithinthedescribedrange.ThiserrorfunctionisdescribedmathematicallyinEquation 3 17 ].Forlargebaselinetrackingsimpletranslationalmodels 55 PAGE 56 4 3 and 3 .Thevelocityexpressions,showninEquations 3 and 3 ,describethemovementoffeaturepointsintheimageplaneandiscommonlyreferredtoinliteratureastheopticow. 3 and 3 whileassumingc=0isasfollows 56 PAGE 57 3-4 whereh1andh2denotethepositionvectorsofthefeaturepoint,P,inthecamerareferenceframes.Also,thevaluesofx1andx2representthepositionvectorsprojectedontothefocalplanewhileTindicatesthetranslationvectoroftheoriginofthecameraframes.AgeometricrelationshipbetweenthevectorsinFigure 3-4 isexpressedbyintroducingRasarotationmatrix.Thisrotationmatrixincludestheroll,pitchandyawanglesthattransformthecameraframesbetweenmeasurements.TheresultingepipolarconstraintisexpressedinEquation 3 3 ,assumesapin-holecamerawhichiscolinearwithitsprojectionintothefocalplane. 57 PAGE 58 Geometryoftheepipolarconstraint TheexpressionsinEquation 3 andEquation 3 reectthatthescalartripleproductofthreecoplanarvectorsiszero,whichformsaplaneinspace.Theserelationshipscanbeexpandedusinglinearalgebra[ 102 103 ]togenerateastandardformoftheepipolargeometryasinEquation 3 .Thisnewformindicatesarelationshipbetweentherotationandtranslation,writtenastheessentialmatrixdenotedasQ,totheintrinsicparametersofthecameraandassociatedfeaturepoints.Inthiscase,theequationisderivedforasinglefeaturepointthatiscorrelatedbetweentheframes, 58 PAGE 59 3 withl1andl2representingtheepipolarlinesinimage1andimage2beingproportionaltotheessentialmatrix,respectfully. 3 and 3 arerewrittenintermsofthefundamentalmatrix,F,andareshowninEquations 3 and 3 3 3 whichsolvesfortheentriesoftheessentialmatrix.ThisalgorithmwasdevelopedbyLonguet-Higgins[ 39 ]andisdescribedinthissection.TheexpressioninEquation 3 canactuallybeexpressedasinEquation 3 usingadditionalargumentsfromlinearalgebra[ 102 103 ].Thevector,q2R9,containsthestackedcolumnsoftheessentialmatrixQ. 3 ,foreachfeaturepointwheretheentriesoftheessemtialmatrixarestackedinthevectorq.Asetofrowvectorsarestackedtoformamatrix,C,ofnmatchedfeaturepointsand 59 PAGE 60 3 .ThematricxC,showninEquation 3 ,isan9matrixofstackedfeaturepointsmatchedbetweentwoviews. 3 existsusingalinearleast-squaresapproachonlyifthenumberofmatchedfeaturesineachframeisatleast8suchthatrank(C)=8.Additionally,morefeaturepointswillobviouslygeneratemoreconstraintsand,presumably,increaseaccuracyofthesolutionduetotheresidualsoftheleast-squares.Inpractice,theleast-squaressolutiontoEquation 3 willnotexistduetonoise;therefore,aminimizationisusedtondanestimateoftheessentialmatrix,asshowninEquation 3 3 3 ,wherethetranslationTisfounduptoascalingfactor.Thesefoursolutions,whichconsistofallpossiblecombinationsofRandT,arecheckedtoverifywhichcombinationgeneratesapositivedepth 60 PAGE 61 102 103 ].Whenthissituationoccursonemustusetheplanarhomographyapproach,whichisthetopicofthenextsection. 102 103 ].Figure 3-5 depictsthegeometryinvolvedwithplanarhomography.Thefundamentalrelationshipexpressingapointfeaturein3DspaceacrossasetofimagesisgiventhrougharigidbodytransformationshowninEquation 3 61 PAGE 62 Geometryoftheplanarhomography Ifanassumptionismadethatthefeaturepointsarecontainedonthesameplane,thenanewconstraintinvolvingthenormalvectorcanbeestablished.DenoteN=[n1;n2;n3]Tasthenormalvectoroftheplanecontainingthefeaturepointsrelativetocameraframe1.ThentheprojectionontotheunitnormalisshowninEquation 3 ,whereDistheprojecteddistancetotheplane. 3 intoEquation 3 resultsinEquation 3 62 PAGE 63 3 canbeextendedtoimagecoordinatesthroughEquation 3 3 withtheskewsymmetricmatrixbx2resultsintheplanarhomographyconstraintshowninEquation 3 3 canberewrittentoEquation 3 3 requiresatleastfourfeaturepointcorrespondences.TheseadditionalconstraintscanbestackedtoformanewconstraintmatrixY,asshowninEquation 3 3 intermsofthenewconstraintmatrixresultsinEquation 3 102 103 ],showninEquation 3 fortheunknownscalerl. 63 PAGE 64 3 3 ,thatarepreservedinthehomographymappingandwillfacilitateinthedecompositionprocess. 3 willestablishahomographysolutionexpressedintermsoftheseknownvariables. ThefoursolutionsareshowninTable 3-1 intermsofthematricesgiveninEquations 3 3 andthecolumnsofthematrixV.Noticethetranslationcomponentisestimateduptoa1 Table3-1. Solutionsforhomographydecomposition Solution1 64 PAGE 65 3-4 andassumesthattherotation,R,andtranslation,T,betweenframesisknown.Giventhat,thecoordinatesofh1andh2canbecomputed.Recall,thefundamentalrelationshiprepeatedhereinEquation 3 3 andEquation 3 .Theserelationshipsallowsomecomponentsofhxandhytobewrittenintermsofandnwhichareknownfromtheimages.Thus,theonlyunknownsarethedepthcomponents,h1;zandh2;z,foreachimage.TheresultingsystemcanbecastasEquation 3 andsolvedusingaleast-squaresapproach. 3 usingz=[h2;z;h1;z]asthedesiredvectorofdepths. 3 obtainsthedepthestimatesofafeaturepointrelativetobothcameraframes.Thisinformationalongwiththeimageplanecoordinatescanbeusedtocompute(h1;x;h1;y)and(h2;x;h2;y)bysubstitutingthesevaluesbackintoEquations 3 and 3 .Theresultingcomponentsofh1canthenbeconvertedtothecoordinateframeofthesecondimageanditshouldexactlymatchh2.Thesevalueswillnevermatchperfectlydueto 65 PAGE 66 3 isverysensitivetotheseuncertainties.Chapter 4 willdiscussamethodtoobtainuncertaintyboundsontheSFMestimatesbasedonthesourcesdescribed. 66 PAGE 67 3 .Oncefeaturepointsarelocatedandtrackedacrossimages,anumberstateestimationalgorithms,suchasopticow,epipolarconstraint,andstructurefrommotion,canbeemployed.Althoughcameracalibrationtechniqueshaveproventoprovideaccurateestimatesoftheintrinsicparameters,theprocesscanbecumbersomeandtimeconsumingwhenusingalargequantityoflowqualitycameras.Thischapterdescribesquantitativelytheeffectsonfeaturepointpositionduetouncertaintiesinthecameraintrinsicparametersandhowthesevariationsarepropagatedthroughthestateestimationalgorithms.Thisdeterministicapproachtouncertaintyisanefcientmethodthatdeterminesalevelofboundedvariationsonstateestimatesandcanbeusedforcameracharacterization.Inotherwords,themaximumallowablestatevariationinthesystemwillthendeterminetheaccuracyrequiredinthecameracalibrationstep. 3-1 .TheresultingvaluesarerepeatedinEquations 4 and 4 asafunctionoffocallength,f,andradialdistortion,d,intermsofthecomponentsofh. 67 PAGE 68 4-1 ,isdependentonboththerelativepositionsofcameraandthefeature.Thisradialdistance,asshowninFigure 4-2 ,isalsorelatedviaanonlinearrelationshiptotheradialdistortion.Theanalysisofthefeaturepointswillrequireestimationofthecameraparameters. BFigure4-1. FeaturePointDependenceonFocalLengthforA)f=0:5andB)f=0:25 BFigure4-2. FeaturePointDependenceonRadialDistortionforA)d=0:0001andB)d=0:0005 4 ,showstherangeofvaluesthatmustbeconsideredforanominalestimate, 68 PAGE 69 4 presentstherangeofvaluesforradialdistortion. 4 andEquation 4 aresubstitutedintothecameramodelofEquation 4 andEquation 4 .TheresultingexpressionsforfeaturepointsarepresentedinEquations 4 and 4 4 andEquation 4 donotdependonuncertaintysotheseportionsrepresentnominalvalues,oandno,whicharethecorrectlocationsoffeaturepoints.Thesecondtermswhichincludedfandddtermsaretheuncertainty,danddn,ineachfeaturepointwhichareboundedinnormbyDandDn.Assuch,thefeaturepointsmaybewrittenasinEquation 4 andEquation 4 toreecttheuncertainty. 69 PAGE 70 4 andEquation 4 3 andEquation 3 .Inpractice,thevelocitiesarecomputedbysubtractinglocationsofafeaturepointacrossapairofimagestakenatdifferenttimes.Suchanapproachassumesthatafeaturepointcanbetrackedandcorrelatedbetweentheseframes.TheopticowisthengivenasJusingEquation 4 forafeaturepointat1andn1inoneframeand2andn2inanotherframe. 70 PAGE 71 4 andEquation 4 ,canbesubstitutedintoEquation 4 tointroduceuncertainty.TheresultingexpressioninEquation 4 separatestheknownfromunknownelements. 4 wheretheuncertaintyisboundedbyDJ2R. 4 .Theactualboundsonthefeaturepoints,asnotedinEquation 4 andEquation 4 ,variesdependingonthelocationofeachfeaturepointsoboundsofD1andD2aregivenforeachverticalcomponentandDn1andDn2aregivenforeachhorizontalcomponent.Assuch,theboundonvariationisnotedinEquation 4 asspecictotheh1andh2usedtogatherfeaturepointsineachimage. 3 ,requiresapin-holecamerawhoseintrinsicparametersareexactlyknown.Suchasituationisobviouslynotrealisticsotheeffectofuncertaintycanbedetermined.Anon-idealcamerawilllosethe 71 PAGE 72 4 andEquation 4 ,whichareactuallycausedbyuncertaintyinthecameraparametersasnotedinEquation 4 andEquation 4 .TheconstraintmatrixfromEquation 3 canthenbewrittenasanominalcomponent,Co,plussomeuncertainty,dC,asinEquation 4 4 andEquation 4 .TheithrowofthismatrixcanthenbewrittenasEquation 4 3 ,whenincludingtheuncertaintymatrixinEquation 4 ,willexist;however,thatsolutionwilldifferfromthetruesolutionorthenominalsolution.Essentially,thesolutioncanbeexpressedasthenominalsolution,qo,andanuncertainty,dq,asinEquation 4 .Thisperturbedsystemcannowbesolvedusingalinearleast-squaresapproachfortheentriesoftheessentialmatrix. 4 hasvariationwhichwillbenormboundedbyDqasinEquation 4 whichindicatestheworse-casevariationimposedontheentriesofq. 72 PAGE 73 4 .ThisboundusestherelationshipbetweenuncertaintiesinEquation 4 throughtheconstraintinEquation 4 .Also,thesizeofthisuncertaintydependsonthelocationofeachfeaturepointsotheboundsisnotedasspecictotheh1andh2obtainedfromFigure 3-4 4 ,canthenbeuseddirectlytocomputethevariationinstateestimates.Theentriesofqarerstarrangedbackintomatrixformtoconstructthenewessentialmatrixthatincludesparametervariations.ThisnewessentialmatrixisthendecomposedusingSVDtechniquesdescribedinSection 3.6.1 3 .SubstitutingEquation 4 andEquation 4 intoEquation 3 resultswithavariationinthesystemmatrixY.Likewise,thenewsystemmatrixwithuncertainintrinsicparamterscanbewrittenasanominalmatrix,Y0plussomevariation,dY,asshowninEquation 4 4 andEquation 4 .correspondingly, 73 PAGE 74 4 3 ,whenincludingtheuncertainmatrixinEquation 4 ,willexist;however,thatsolutionwilldifferfromthetruesolution.Essentially,thesolutioncanbeexpressedasthenominalsolution,ho,andanuncertainty,dh,asinEquation 4 4 hasvariationwhichwillbenormboundedbyDhasinEquation 4 4 .ThisboundusestherelationshipbetweenuncertaintiesinEquation 4 throughtheconstraintinEquation 4 .Also,thesizeofthisuncertaintydependsonthelocationofeachfeaturepointsotheboundsisnotedasspecictotheh1andh2obtainedfromFigure 3-4 4 ,canthenbeuseddirectlytocomputethevariationinstateestimates.Theentriesofharerstarrangedbackintomatrixformtoconstructthenewhomographymatrixthatincludesparameter 74 PAGE 75 3.6.3 3 forthestructurefrommotionrelationship.Assuch,thematrixshouldbewrittenintermsofanominalvalue,Ao,andanuncertainperturbation,dA,asinEq. 4 4 andEquation 4 intoEquation 3 .TheperturbationisthenwrittenasEquation 4 3 whenconsideringEquation 4 willobviouslyresultinadepthestimatethatdiffersfromthecorrectvalue.Denezoastheactualdepthsthatwouldbecomputedusingtheknownparametersofthenominalcameraanddzasthecorrespondingerrorintheactualsolution.Theleast-squaresproblemcanthenbewrittenasEquation 4 andsolvedusingapseudo-inverseapproach. 4 .Thisrangeofsolutionswillliewithintheboundedrangedeterminedfromtheworst-casebound. 75 PAGE 76 4 .Thisboundnotesthattheboundonvariationsinfeaturepoints,andultimatelytheboundonsolutionstostructurefrommotion,dependsonthelocationofthosefeaturepoints. 76 PAGE 77 3 toderivethesystemequations. 5-1 alongwiththerespectiveorigins.Thebody-xedcoordinatesystemhastheoriginlocatedatthecenterofgravityoftheaircraft.Theaxesareorientedsuchthatb1alignsoutthenoseandb2alignsouttherightwingwithb3pointedoutthebottom.Themovementoftheaircraft,whichincludesaccelerating,willobviouslyaffectthecoordinatesystem;consequently,thebody-xedcoordinatesystemisnotaninertialreferenceframe. 77 PAGE 78 Body-xedcoordinateframe Theorientationanglesoftheaircraftareofparticularinterestformodelingavision-basedsensor.Therollangle,f,describesrotationaboutb1,thepitchangle,q,describesrotationaboutb2andtheyawangle,y,describesrotationaboutb3.ThetransformationfromavectorrepresentedintheEarth-xedcoordinatesystemtothebody-xedcoordinatesystemisrequiredtorelateon-boardmeasurementstoinertialmeasurements.Thistransformation,giveninEquation 5 ,usesREBwhichareEulerrotationsofroll,pitchandyaw[ 29 108 ], 5 .Theorderofthismatrixmultiplicationneedstobemaintainforcorrectcomputation. 78 PAGE 79 5 5 torepresenttheserates. 5-2 ,usethetraditionalchoiceofi3aligningthroughthecenterofviewofthecamera.Theremainingaxesareusuallychosenwithi2alignedrightoftheviewandi1alignedoutthetopalthoughsomevariationinthesechoicesisallowedaslongastheresultingaxesretaintheright-handedproperties.Thedirectionofthecamerabasisvectorsaredenedthroughthecamera'sorientationrelativetothebody-xedframe.Thisframeworkisnotedasthecamera-xedcoordinatesystembecausetheoriginisalwayslocatedataxedpointonthecameraandmovesinthesamemotionasthecamera.Thecameraisallowedtomovealongtheaircraftthroughadynamicmountingwhichadmitsbothrotationandtranslation.Thisfunctionalityenablesthetrackingoffeatureswhilethevehiclemovesthroughanenvironment.Theoriginofthecamera-xedcoordinatesystemisattachedtothismovingcamera;consequently,thecamera-xedframeisnotaninertialreference.A6degree-of-freedommodelofthecameraisassumedwhichadmitsafullrangeofmotion.Figure 5-2 alsoillustratesthecamera'ssensingconewhichdescribesboththeimageplaneandtheeldofviewconstraint. 79 PAGE 80 Camera-xedcoordinateframe Similartothebody-xedcoordinateframe,atransformationcanbedenedforthemappingbetweenthebody-xedframe,Bandthecameraframe,IasseeninEquation 5 5 ,similartothebody-xedrotationmatrix.Theorientationanglesofthecameraarerequiredtodeterminetheimagingusedforvision-basedfeedback.Therollangle,fc,describesrotationabouti3,thepitchangle,qc,describesrotationabouti2andtheyawangle,yc,describesrotationabouti1. 5 willtransformavectorinbody-xedcoordinatestocamera-xedcoordinates.Thistransformationisrequiredtorelatecamerameasurementstoon-boardvehiclemeasurementsfrominertialsensors.Thematrixagaindependsontheangular 80 PAGE 81 5 torepresenttheseangles. 5-3 ,thusrelatesthecameraandtheaircrafttothefeaturepointalongwithsomeinertialorigin. Figure5-3. Scenarioforvision-basedfeedback 81 PAGE 82 5 andEquation 5 aretypicallyrepresentedintheinertialreferenceframerelativetotheEarth-axisorigin. 5 ,istypicallygivenwithrespecttothebody-axisorigin.Thischoiceofcoordinatesystemsreectsthatthecameraisintrinsicallyaffectedbyanyaircraftmotion. 3 todescribetherelativepositionbetweenthecameraandthefeaturepoint.Recall,thisvectorwasgiveninthecamera-xedcoordinatesystemtonotetheresultingimageisdirectlyrelatedtopropertiesrelativetothecamera.TherepresentationofhisrepeatedhereinEquation 5 forcompleteness. 5 isused.Thisexpressionincorporatesthetranslationsinvolvedwiththeoriginsofeachcoordinateframethroughaseriesofsingle-axisrotationsuntilthecorrectframeisreached. 82 PAGE 83 108 110 ]andarerepeatedinEquation 5 to 5 foroverallcompleteness.Fxmgsinq=m(u+qwrv) 5 .Theaircraftstatesofinterestforthecameramotionsystemconsistofthepositionandvelocityoftheaircraft'scenterofmass,TEBandvb,theangularvelocity,w,andtheorientation 83 PAGE 84 5 .AsstatedinEquation 5 ,theaircraft'svelocityisexpressedinthebody-xedcoordinateframe.Eachoftheseparameterswillappearexplicitlyintheaircraft-cameraequations. 84 PAGE 85 5-3 forafeaturepointrelativetotheinertialframe.Therefore,thevectorsumcanbeusedtosolvefortherelativepositionbetweenthecameraanda3Dfeaturepoint.AftermakingthepropercoordinatetransformationsbyusingEquations 5 and 5 ,thisrelativepositioncanbeexpressedincameraframe,I,asshowninEquation 5 5 intoEquations 3 and 3 animagecanbeconstructedasafunctionofaircraftstates.Themajorassumptionoftheseequationsispriorknowledgeofthefeaturepointlocationrelativetotheinertialframe,whichmaybeprovidedbyGPSmaps.Furthermore,theimageresultsobtainedcanalsobepassedthroughEquations 3 and 3 toaddtheeffectsofradialdistortion.Thedistortedimagewillprovideamoreaccuratedescriptionofanimageseenbyaphysicalcamera,assumingtheintrinsicparametersofthecameraareknown. 5 withrespecttotheinertialframe,asshowninEquation 5 dt(h)=Ed dt(x)Ed dt(TEB)Ed dt(TBI)(5) 85 PAGE 86 5 cannowberewrittentoEquation 5 foranon-stationaryfeaturepoint. dt(h)=xTEBBd dt(TBI)wTBIEwIh(5)ThisequationcanbereducedfurtherifthecamerasareconstrainedtohavenotranslationrelativetotheaircraftsoBd dt(TEI)=0.Alternatively,thistermisretainedinthederivationtoallowthisdegreeoffreedominthecamerasetup.Theangularvelocity,EwI,canbefurtherdecomposedusingtheAdditionTheorem.ThenalstepimplementsEquations 5 and 5 totransformeachtermintothecameraframe.Aftersomemanipulation,theexpressionforthevelocityofafeaturepointrelativetothecameraframeresultsinEquation 5 5 and 5 intoEquations 3 and 3 .Thisresultwillprovideadescriptionoftheopticalowforeachfeaturepointformedbyeitherthecameratravelingthroughtheenvironmentorthemotionofthefeaturepointsthemselves.ToincorporateradialdistortioneffectsintotheopticowcomputationrequirestheadditionalsubstitutionintoEquations 3 and 3 86 PAGE 87 5 5 3 3 3 ,and 3 .Arrangingtheparametersforthekthcameraintoasinglevector,asshowninEquation 5 ,resultsthenintheformulationofagenericaircraft-camerasystemwithkcamerasallhavingindependentmotionthattracknfeaturepointsisobtained. 5 .ThisvectorcanbeextendedtoincludeothercamerafeaturessuchasCCDarraymisalignment,skewness,etc.ThefocalplanepositionscanthenbeassembledintoavectorofobservationsasshowninEquation 5 ,wherennumberoffeaturepointsareobtained.Likewise,thestatesoftheaircraftcanbecollectedandrepresentedasastatevectorasshowninEquation 5 .Inaddition,theinitialstatesofthevehiclearedenedasX0. 5 and 5 .TheobservationsusedinthisdissertationconsistofmeasureableimagesshowninEquations 3 and 3 whichcapturenonlinearitiessuchasradialdistortion.Thissystem,whichmeasuresimageplaneposition,isdescribedmathematically 87 PAGE 88 5 5 ,thenadifferentsetofequationscanbeobtainedwhichwillbereferredtoastheOpticFlowFormofthegoverningaircraft-cameraequationsofmotion.ThissystemisgiveninEquation 5 ,whichusestheopticowexpressiongiveninEquations 3 and 3 astheobservations.X(t)=F(X(t);U(t);a(t);t) 88 PAGE 89 89 PAGE 90 3 andhowthesefeaturesrelatetothestatesofanaircraftinChapter 5 ,theeffectsofindependentlymovingobjectsneedtobehandledinadifferentmanner.Forcasesinvolvingastationarycamera,suchasinsurveillanceapplications,simplelteringandimagedifferencingtechniquesareemployedtodetermineindependentlymovingobjects.Althoughthesetechniquesworkwellforstationarycameras,adirectimplementationtomovingcameraswillnotsufce.Foramovingcamera,theapparentimagemotioniscausedbyanumberofsources,suchascamerainducedmotion(i.e.ego-motion)andthemotionduetoindependentlymovingobjects.Acommonapproachtodetectingmovingobjectconsidersatwostageprocessthatincludes(i)acompensationroutinetoaccountforcameramotionand(ii)aclassicationschemetodetectindependentlymovingobjects. 3.6 .Thesecondapproachusesthesmoothnessconstraintinattempttominimizethesumofsquaredifferences(SSD)overeitheraselectnumberoffeaturesortheentireoweld.Thisapproachassumesthestationary 90 PAGE 91 3.6 .Theepipolarconstraintcanbeusedtorelatefeaturepointsacrossimagesequencesthrougharigidbodytransformation.TheepipolarlinesofastaticenvironmentarecomputedusingEquation 3 orEquation 3 dependingiftheessentialmatrixorthefundamentalmatrixisrequired.AnillustrationofthecomputedepipolarlinesisdepictedinFigure 6-1 forastaticenvironmentobservedbyamovingcamera.Noticeforthisstaticcase,thefeaturepointsinthesecondimage(therightimagecontainingtheoverlaidepipolarlines)areshowntoliedirectlyontheepipolarlines. BFigure6-1. EpipolarLinesAcrossTwoImageFrames:A)InitialFeaturePointsandB)FinalFeaturePointswithOverlayedEpipolarLines Oncecameramotionestimationhasbeenfound,theepipolarlinescanbeusedasanindicationofmovingobjectsintheimage.Forinstance,thefeaturepointscorrespondingtothestationarybackgroundwilllieontheepipolarlineswhilethefeaturepointscorrespondingtomovingobjectswillviolatethisconstraint.Similarly,thecomputationofopticalowcanalsobeusedfordetectingindependentlymovingobjects.Incomputingtheopticalow,themotioninducedbythecameraalongwithmovingobjectsisfusedtogetherinthemeasuredimage.Recall,theopticowexpressions 91 PAGE 92 3 and 3 orEquations 3 and 3 forradialdistortion.Decomposingtheopticalowintoitscomponentsofcamerarotation(r;nr)andtranslation(t;nt)andindependentlymovingobjects(i;ni)facilitatesthedetectionproblem.Therefore,thecomponentsoftheopticalowcanbewrittenasinEquation 6 5 :thetranslationalvelocity[u;v;w]Tandtheangularvelocity[p;q;r]Tofthecamera.TheresultingexpressionsareshowninEquations 6 and 6 andappliesonlytofeaturesstationaryintheenvironment.ThedetailsdescribingthesubstitutionofthecameramotionstatesaredescribedinChapter 5 hz1 111 ]thattherotationalstates[p;q;r]Tcanbeestimatedaccuratelyforastaticenvironmentthroughanonlinearminimizationprocedurefornfeatureswheren6.Theapproachusedavector-valuedoweldJ(x)andisgiveninEquation 6 92 PAGE 93 6 iscomposedofunknownvehiclestatesanddepthparameters. 6 thatminimizesthemagnitudeofthecostfunction. 2kJ(x)k2(6)Thesameapproachistakenherewithcaution.Recallthatthemeasuredopticalowalsocontainsmotionduetoindependentlymovingobjectsinadditiontotheinducedopticalowcausedbythecameramotion.Ingeneral,thesevariationsinthemeasuredopticalowwillintroduceerrorintothe[p;q;r]Testimates.Ifsomeassumptionsaremaderegardingtherelativeopticalowbetweenthestaticenvironmentandmovingobjects,thenerrorsinthestateestimatescanhaveminimaleffect.Forinstance,ifthestaticportionofthesceneisassumedtobethedominantmotionintheopticalowthentheestimateswillcontainminimalerrors.Employingthisassumption,estimatesfortheangularvelocities[p;q;r]Tofthecamera/vehicleareobtained.SubstitutingtheseestimatesintoEquation 6 resultsinestimatesfortherotationalportionoftheopticalow,asshowninEquation 6 6 .TheresidualopticalowRes;nRescontainsonlythecomponentsofthecameratranslationandindependentlymovingobjects.Fromthisexpression,constraintscanbeemployedonthecamerainducedmotiontodetectindependentlymovingobjects. 93 PAGE 94 6-2 .Consequently,featurepointsthatviolatethisconditioncanbeclassiedasindependentlymovingobjects.Thischaracteristicobservedfromstaticfeatureswillbethebasisfortheclassicationscheme. Figure6-2. FOEconstraintontranslationalopticowforstaticfeaturepoints TheresidualopticalowmaycontainindependentlymovingobjectswithintheenvironmentthatradiatefromtheirownFOE.AnexampleofasimplescenarioisillustratedinFigure 6-3 forasinglemovingobjectontheleftandasimulationwithsyntheticdataoftwomovingvehiclesontheright.NoticethetwoprobableFOEsinpictureontheleft,onepertainingtothestaticenvironmentandtheotherdescribingthemovingobject.Inaddition,theepipolarlinesofthetwodistinctFOEsintersectatdiscretepointsintheimage.Thesepropertiesofmovingobjectsarealsoveriedinthesyntheticdatashownintheplotontheright.Thus,aclassicationschememustbedesignedtohandlethesescenariostodetectindependentlymovingobjects.Thenext 94 PAGE 95 Residualopticowfordynamicenvironments 6 .AnapproximationforthepotentiallocationoftheFOEisfoundbyextendingthetranslationaloptical-owvectorstoformtheepipolarlines,asillustratedinFigure 6-3 ,andobtainingallpossiblepointsofintersection.Asmentionedpreviously,theintersectionpointsobtainedwillconstituteanumberofpotentialFOEs;however,onlyonewilldescribethestaticbackgroundwhiletherestareduetomovingobjects.Theapproachconsideredforthisclassicationthatessentiallygroupstheintersectiondatatogetherthroughadistancecriterionisaniterativeleast-squaressolutionforthepotentialFOEs.Theiterationproceduretestsallintersectionpointsasadditionalfeaturesareintroducedtothesystemofequationseachofwhichinvolves2unknownimageplanecoordinatesoftheFOEfoei;nfoei.Theprocessstartsbyconsidering2featurepointsandtheirFOEintersection 95 PAGE 96 6 fortheFOEcoordinatesfoe1;nfoe1(fortherstiterationaleast-squaressolutionisnotnecessarybecausetwolinesintersectatasinglepoint). 2kM264n375bk2(6) where 6 fortheithiteration.Mathematically,theclassicationschemefortheithiterationisgiveninEquations 6 and 6 foeifoei12+nfoeinfoei12(6) 96 PAGE 97 97 PAGE 98 98 PAGE 99 3 .Amongthetechniquesthatutilizefeaturepoints,theapproachrelatedtothispaperinvolvesepipolargeometry[ 39 112 ].Thepurposeofthistechniqueistoestimaterelativemotionbasedonasetofpixellocations.Thisrelativemotioncandescribeeithermotionofthecamerabetweentwoimagesortherelativedistanceoftwoobjectsofthesamesizefromasingleimage.The3DscenereconstructionofamovingtargetcanbedeterminedfromtheepipolargeometrythroughthehomographyapproachdescribedinChapter 3 .Forthecasedescribedinthischapter,amovingcameraattachedtoavehicleobservesaknownmovingreferenceobjectalongwithanunknownmovingtargetobject.Thegoalistoemployahomographyvision-basedapproachtoestimatetherelativeposeandtranslationbetweenthetwoobjects.Therefore,acombinationofvisionandtraditionalsensorssuchasaglobalpositioningsystem(GPS)andaninertialmeasurementunit(IMU)arerequiredtofacilitatethisproblemforasinglecameraconguration.ForexampleintheAARcase,GPSandIMUmeasurementsareavailableforboththereceiverandtankeraircrafts.Ingeneral,asinglemovingcameraaloneisunabletoreconstructthe3Dscenecontainingmovingobjects.Thisrestrictionisduetothelossoftheepipolarconstraint,wheretheplaneformedbythepositionvectorsrelativetotwocamerapositionsintimetoapointofinterestandthetranslationvectorisnolongervalid.Techniqueshavebeenformulatedtoreconstructmovingobjectsviewedbyamovingcamerawithvariousconstraints[ 35 113 116 ].Forinstance,ahomographybasedmethodthatsegmentsbackgroundfrommovingobjectsandreconstructsthetarget'smotionhasbeenachieved[ 117 ].Theirreconstructionisdonebycomputingavirtualcamerawhichxesthetarget'spositionintheimageanddecomposesthehomographysolutionintomotionofthecameraandmotioncausedbythetarget.Thisdecompositionisdoneusingaplanartranslationconstraintwhichrestrictsthetarget'smotiontoagroundplane.Similarly,Han 99 PAGE 100 115 ]proposedanalgorithmthatreconstructs3Dmotionofamovingobjectusingafactorization-basedalgorithmwiththeassumptionthattheobjectmoveslinearlywithconstantspeeds.Anonlinearlteringmethodwasusedtosolvetheprocessmodelwhichinvolvedboththekinematicsandtheimagesequencesofthetarget[ 118 ].Thistechniquerequiresknowledgeoftheheightabovethetargetwhichwasdonebyassumingthetargettraveledonthegroundplane.Thisassumptionallowedothersensors,suchasGPS,toprovidethisinformation.ThepreviousworkofMehtaetal.[ 77 ]showedthatamovingmonocularcamerasystemcouldestimatetheEuclideanhomographiesforamovingtargetinreferencetoaknownstationaryobject.ThecontributionofthischapteristocasttheformulationshowninMehtaelal.toamoregeneralproblemwherebothtargetandreferencevehicleshavegeneralmotionandarenotrestrictedtoplanartranslations.Thisproposedapproachincorporatesaknownreferencemotionintothehomographyestimationthroughatransformation.EstimatesoftherelativemotionbetweenthetargetandreferencevehiclearecomputedandrelatedbackthroughknowntransformationstotheUAV.RelatingthisinformationwithknownmeasurementsfromGPSandIMU,thereconstructionofthetarget'smotioncanbeachievedregardlessofitsdynamics;however,thetargetmustremainintheimageatalltimes.Althoughtheformulationcanbegeneralizedforncameraswithindependentposition,orientation,translations,androtationthischapterdescribesthederivationofasinglecamerasetup.Meanwhile,cuesonboththetargetandreferenceobjectsareachievedthroughLEDlightsormarkersplacedinaknowngeometricpatternofthesamesize.Thesemarkersfacilitatethefeaturedetectionandtrackingprocessbyplacingknownfeaturesthatstandoutfromthesurroundingswhilethegeometryandsizeofthepatternallowsforthecomputationoftheunknownscalefactorthatiscustomarytoepiploarandhomographybasedapproaches.ThischapterbuildsonthetheorydevelopedinChapters 3 and 5 whilerelyingonthemovingobjectdetectionalgorithmtoisolatemovingobjectswithinanimage.RecalltheowoftheoverallblockdiagramshowninFigure 1-6 .Theprocessstartedbycomputingfeaturesinthe 100 PAGE 101 6 .Oncemovingobjectsintheimagearedetected,thehomographyestimationalgorithmproposedinthischapterisimplementedfortargetstateestimation. 7.2.1SystemDescriptionThesystemdescribedinthispaperconsistsofthreeindependentlymovingvehiclesorobjectscontaining6-DOFmotion.TodescribethemotionofthesevehiclesaEuclideanspaceisdenedwithveorthonormalcoordinateframes.TherstframeisanEarth-xedinertialframe,denotedasE,whichrepresentstheglobalcoordinateframe.Theremainingfourcoordinateframesaremovingframesattachedtothevehicles.Therstvehiclecontainstwocoordinateframes,denotedasBandI,torepresentthevehicle'sbodyframeandcameraframe,asdescribedinChapter 5 inFigure 5-1 .Thisvehicleisreferredtoasthechasevehicleandisinstrumentedwithanon-boardcameraandGPS/IMUsensorsforpositionandorientation.Thesecondvehicle,denotedasF,isconsideredareferencevehiclethatalsocontainsGPS/IMUsensorsandprovidesitsstatestothechasevehiclethroughacommunicationlink.Lastly,thethirdvehicle,denotedasT,isthetargetvehicleofinterestinwhichunknownstateinformationistobeestimated.Inaddition,actitiouscoordinateframewillbeusedtofacilitatetheestimationprocessandisdenedasthevirtualcoordinatesystem,V.Thecoordinatesofthissystemarerelatedthroughtransformationscontainingbothrotationalandtranslationalcomponents.TherotationalcomponentisestablishedusingasequenceofEulerrotationsintermsoftheorientationanglestomaponeframeintoanother.LettherelativerotationmatricesREB,RBI,REF,REV,RIV,RFV,RTVandRETdenotetherotationfromEtoB,BtoI,EtoF,EtoV,ItoV,FtoV,TtoV,andEtoT.Secondly,thetranslationsaredenedasTEB,xF,xV,xT,xF;n,xT;n,TBI,xIV,mIF,mIT,hF;n,hT;n,mVF,mVT,hVF;n,andhVT;nwhichdenotetherespectivetranslationsfromEtoB,EtoF,EtoV,EtoT,EtothenthfeaturepointonthereferencevehicleandtargetvehiclesallexpressedinE,BtoIexpressedinB,ItoV,ItoF,ItoT,ItothenthfeaturepointonthereferenceandtargetvehiclesexpressedinI,VtoF,V PAGE 102 7-1 foracameraonboardaUAVwhilethevectorsrelatingthefeaturepointstoboththerealandvirtualcamerasaredepictedinFigure 7-2 .TheestimatedquantitiescomputedfromthevisionalgorithmaredenedasRTBandxTBwhicharetherelativerotationandtranslationfromTtoBexpressedinB. Figure7-1. Systemvectordescription Thecameraismodeledthroughatransformationthatmaps3-dimensionalfeaturepointsontoa2-dimensionalimageplaneasdescribedinChapter 3 .Thistransformationisageometricrelationshipbetweenthecamerapropertiesandthepositionofafeaturepoint.Theimageplanecoordinatesarecomputedbasedonatangentrelationshipfromthecomponentsofhn.ThecamerarelationshipusedinthischapterisreferredtoasthecontinuouspinholecameramodelandisgiveninEquations 3 and 3 forazerolensoffset,wherefisthefocallengthofthecameraandhx;n,hy;n,hz;narethe(x;y;z)componentsofthenthfeaturepoint.Thispinholemodelisacontinuousmappingthatcanbefurtherextendedtocharacterizepropertiesofaphysicalcamera.Somecommonadditionstothismodelincludeskewness,radial 102 PAGE 103 BFigure7-2. MovingtargetvectordescriptionreltivetoA)cameraIandB)virtualcameraV 3 .Eachextensiontothemodeladdsanotherparametertoknowfortheestimationproblemandeachcanintroduceuncertaintyandlargeerrorsintheestimationresult.Therefore,thischapterwillonlyconsidertheeldofviewconstraintandleavethenonlineartermsandtheeffectsonestimationforfuturework.RecalltheeldofviewconstraintsgiveninChapter 3 .Theseconstraintscanberepresentedaslowerandupperboundsintheimageplaneandaredependentonthehalfangles(gh;gv)whichareuniquetoeachcamera.Mathematically,theseboundsareshowninEquations 7 and 7 forthehorizontalandverticaldirections. 103 PAGE 104 102 ].Thesameconstraintholdsfortheimagecoordinatesaswellbutalsointroducesanunknownscalefactor.Employingthisconstraint,estimatesofrelativemotioncanbeacquiredforbothcamera-in-handandxedcameracongurations.Thisdissertationdealswiththecamera-in-handcongurationwhileassumingaperfectfeaturepointdetectionandtrackingalgorithm.Thisassumptionenablestheperformanceofthevisionbasedstateestimationtobetestedbeforeintroducingmeasurementerrorsandnoise.Thehomographyconstraintrequiresafewassumptionsbasedonthequantityandthestructureofthefeaturepoints.Thealgorithmrstrequiresaminimumoffourplanarfeaturepointstodescribeeachvehicle.Thisrequirementenablesauniquesolutiontothehomographyequationbasedonthenumberofunknownquantities.ThereferencevehiclewillhaveaminimumoffourpixelvaluesineachimagewhichwillbedenedaspF;n=[F;n;nF;n]8nfeaturepoints.Likewise,thetargetvehiclewillhavefourpixelvaluesandwillbedenedaspT;n=[T;n;nT;n]8nfeaturepoints.Thisarrayoffeaturepointpositionsarecomputedat30Hzwhichistypicalforstandardcamerasandtheframecountisdenotedbyi.Thenalrequirementisaknowndistanceforboththereferenceandtargetvehicle.OnedistancerepresentsthepositionvectortoafeatureonthereferencevehicleinEuclideanspacerelativetothelocalframeFandtheseconddistancerepresentsthepositionvectortoafeatureonthetargetvehicleinEuclideanspacerelativetothelocalframeT.Inaddition,thelengthofthesevectorsalsomustbeequalwhichallowstheunknownscalefactortobedetermined.ThevectordescribingthereferencefeaturepointwillbedenotedassFexpressedinF,whilethevectordescribingthetargetfeaturepointisreferredtoassTexpressedinT.ThesefeaturepointpositionvectorsarealsoillustratedinFigure 7-2 .Thefeaturepointsarerstrepresentedbypositionvectorsrelativetothecameraframe,I.TheexpressionsforboththereferenceandtargetfeaturepointsaregiveninEquations 7 and 7 .ThesevectorcomponentsarethenusedtocomputetheimagecoordinatesgiveninEquations 7 and 7 .ThecomputationinEquation 7 requiresinformationregardingthetargetwhichisdonesolelytoproduceimagemeasurementsthatnormallywouldbeobtainedfromthesensor.Remainingcomputations,regardingthehomography,willonlyusesensor 104 PAGE 105 7 and 7 arethetruerotationsmatricesfromFtoBandTtoB,repectfully,andareshowninEquations 7 and 7 77 ].Inthiscase,boththereferenceandtargetvehiclesareinmotionandarebeingviewedbyamovingcamera.Therefore,thenextstepistotransformthecameratoavirtualcongurationthatobservesthereferencevehiclemotionlessintheimageovertwoframes.Inotherwords,thisapproachcomputesaEuclideantransformationthatmapsthecamera'sstatesati1toavirtualcamerathatmaintainstherelativepositionandorientationbetweenframestoxthefeaturepointsofthereferencevehicle.Thistransformationisdonebymakinguseofthepreviousimageframeandstateinformationati1fromboththecameraandthereferencevehicle.Afterthevirtualcameraisestablishedthehomographyequationscanbeemployedforstateestimation.TocomputethelocationandposeofthevirtualcameraatitherelativepositionandorientationfromItoFati1isrequired.ThisrelativemotioniscomputedthroughknownmeasurementsfromGPS/IMUandtheexpressionsareshowninEquations 7 and 7 fortranslationandrotationati1,respectfully. 105 PAGE 106 7 and 7 forthecurrentframei. 3 and 3 .TheexpressionsforthenewvectorshVF;nandhVT;nintermsofthevirtualcameraaregiveninEquations 7 and 7 forthereferenceandtargetvehicles. 7 and 7 areonewaytocomputeimagecoordinatesforthevirtualcamera,butthereareunknowntermsinEquation 7 thataren'tmeasurableorcomputeableinthiscase.Therefore,analternativemethodmustbeusedtocomputeimagevaluesofthetargetinthevirtualcamera.Usingthepositionandorientationofthevirtualcamera,asgiveninEquations 7 and 7 ,therelativemotioniscomputedfromcameraItocameraVwhileusingepipolargeometrytocomputethenewpixellocations.ThisrelativecameramotionisgiveninEquations 7 and 7 wherethetranslationisexpressedinI. 106 PAGE 107 7 7 whichreliesontherelativemotionremainingconstanttomaintainthereferencestationaryintheimage. 7 .Likewise,thetimevaryingpositionofafeaturepointonthetargetvehicleexpressedinVisgiveninEquation 7 7 and 7 andarerelativetothevirtualcameraframe. 107 PAGE 108 7 7 and 7 whichdescribetherelativemotionbetweenthereferenceandtargetobjects. 7 andthereferencevehiclelocationisknownthroughGPSalongwiththefeaturepointlocations;therefore,aprojecteddistancecanbecomputedthatscalesthedepthofthescene.Tocomputethisdistancethenormalvector,n,thatdenestheplanewhichthereferencefeaturepointslieisrequiredandcanbecomputedfromknowninformation.Ultimately,theprojectivedistancecanbeobtainedandisdenedinEquation 7 throughtheuseofthereferenceposition. 7 intoEquation 7 resultsinanintermediateexpressionfortheEuclideanhomographyandisshowninEquation 7 DnThVF;n(7)Tofacilitatethesubsequentdevelopment,thenormalizedEuclideancoordinatesareusedanddenedinEquations 7 and 7 108 PAGE 109 7 7 ,and 7 thenormalizedEuclideanhomographyisestablishedwhichrelatesthetranslationandrotationbetweencoordinateframesFandT.ThishomographyexpressionisshowninEquation 7 intermsofthenormalizedEuclideancoordinates. {z }hVF;naH(7)InEquation 7 7 D(7)TheEuclideanhomographycannowbeexpressedintermsofimagecoordinatesorpixelvaluesthroughtheidealpin-holecameramodelgiveninEquations 3 and 3 .Thisexpressingisdonebyrstrewritingthecameramodelintomatrixformwhichisreferredtoasthecameracalibrationmatrix,K.SubstitutingthecameramappingintoEquation 7 andusingthecameracalibrationmatrix,K,thehomographyintermsofpixelcoordinatesisobtainedandgiveninEquation 7 .ThisnalexpressionrelatestherotationandtranslationofthetwovehiclesFandTintermsoftheirimagescoordinates.Therefore,toobtainasolutionfromthishomographyexpressionbothvehiclesneedtobeviewableintheimageframe. {z }pVF;nG(7)ThematrixG(t)isdenotedasaprojectivehomographyinEquation 7 whichareasetofequationsthatcanbesolveduptoascalefactorusingalinearleastsquaresapproach.OncethecomponentsofhomographymatrixareestimatedthematrixneedstobedecomposedintotranslationalandrotationalcomponentstoobtainxhandR.Thisdecompositionisaccomplishedusingtechniquessuchassingularvaluedecompositionandgeneratesfourpossiblesolutions[ 119 120 ].Todetermineauniquesolutionsomephysicalcharacteristicsoftheproblem 109 PAGE 110 7 7 toobtainx.Secondly,xisthendividedbyatoscalethedepthratioresultinginthenalxexpressedinI.ThisresultinconjunctionwithRisthenusedinEquation 7 tosolveformVT.ThenextstepistocomputetherelativetranslationfromItoVwhichisgiveninEquation 7 7 7 and 7 representtherelativemotionbetweenthecameravehicleandthetargetvehicle.Thisinformationisvaluableforthecontroltasksdescribedearlierinvolvingbothtrackingandhomingapplications.Thenextsectionwillimplementthisalgorithminsimulationtoverifythestateestimatorforthenoisefreecase. 110 PAGE 111 111 PAGE 112 121 ]thatemploysaHiddenMarkovModeltopredictthemotionofmovingobjects.ThebenetsinusingaHiddenMarkovModelincludeatimedependenceframeworkincorporatedintotheprobabilisticmodelaswellastheabilitytohandlestochasticprocesses.TheunderliningconceptofaHiddenMarkovModeldescribestheprobabilityofaprocesstosequentiallygofromonestatetoanother.Thissequentialpropertyprovidesthenecessaryframeworkfortimedependencemodeling,whichisanattractiveapproachfortheapplicationsconsidered,wherethetimehistorydataisacriticalpieceofinformationincludedinthemodeling. 8 .Therefore,thevelocityandpositionareupdatedthroughEquations 8 and 8 .Althoughthismodelislimited,itdescribesafoundationformodelingtargetmotionandcoversthebasicmodelconstantvelocity. 112 PAGE 113 8 andischaracterizedbyarandomvector,w(t)andisscaledbyaconstant,r.ThevelocitycorrespondingtothisaccelerationisdescribedinEquation 8 .Thismodelattemptstocapturethestochasticbehaviorsbyutilizingaprobabilisticdistributionfunction. 8 canbemodiedtoincorporatesomedependenceonthepreviousaccelerationvalue.ThisdependenceisachievedbyweightingthepreviousaccelerationinthemodelandisshowninEquation 8 .ThebenettothistypeofmodelasopposetoEquation 8 requiressomeknowledgeofthetarget;namely,thatthetargetcannotachievelargeabruptchanginginacceleration.TheresultingvelocityexpressionforthismodelisgiveninEquation 8 8 fortheithtargetandNimageframes.ThevelocityproleiscomputedusingabackwardsdifferencemethodandisgiveninEquation 8 8 ,isobtainedfromthevelocityprolegiveninEquation 8 .Thesamebackwardsdifferencemethodisusedtocomputethis 113 PAGE 114 8 .Thisaccelerationtimehistoryiscomputedimplicitlythroughthepositionestimatesobtainedfromthehomographyalgorithm 8 and 8 providetheinitialmotionstatedescriptionthatpropagatestheMarkovtransitionprobabilityfunction.TheformoftheMarkovtransitionprobabilityfunctionisassumedtobeaGaussiandensityfunctionthatonlyrequirestwoparametersforitsrepresentation.TheparametersneededforthisfunctionincludethemeanandvariancevectorsfortheaccelerationprolegiveninEquation 8 .Note,duringthischapter(x)isthemeanoperatorandnottheverticalcomponentintheimageplane.Likewise,s2(x)isreferredtoasthevarianceoperator. 8 ,wheretheargumentsconsistofthemeanandvariancepertainingtotheestimatedacceleration. 8 and 8 forthetransitionfunction.Thefunctionsfandfsarechosenbasedonthedesiredweightingofthetimehistoryandcansimplybeaweightedlinearcombinationofthearguments.Theseinitialstatisticalparametersareusedinthepredictionstepandupdatedonceanewmeasurementisobtained. PAGE 115 8 asthree-dimensionalGaussiandensityfunctionandisuniquelydeterminedbythemeanandvariance. 2(ai(t)(ai(t)))2 8 8 and 8 ,thepredictiveprobabilityforobjectiattimet+kisgivenasEquation 8 .Thisframeworkenablestheexibilityofcomputingthepredictedestimatesatanydesiredtimeinthefuturewiththenotionthatfurtheroutintimetheprobabilitydiminishes. 8 and 8 fortheentiretimeinterval. 8 and 8 PAGE 116 8 and 8 2ai(t1) 2(ai(t1)) 2ai(t1) 2s2(ai(t1))Lastly,theprobabilityfunctionsforvelocityandpositionareusedtocomputethepredictiveprobabilitiesforobjectithataregiveninEquations 8 and 8 forvelocityandposition,respectfully. 8 istheprobabilitythattargetiislocatedinpositionp(x;y;z).Sothetheoverallprocessisaniterativemethodthatusesthemotionmodels,giveninSection 8.2.1 ,toprovideguessesforpositionandvelocityinattempttomaximizetheprobabilityfunctionsgiveninEquations 8 and 8 .ThepositionthatmaximizesEquation 8 isthemostlikelylocationofthetargetatt+kwithaknownprobability. 116 PAGE 117 7 .Effectively,thesequantitiesaretheerrorsignalsusedforcontroltotrackthemovingcameratowardadesiredlocationbasedonthemotionofthetarget.TheframeworkpresentedherewilluseaircraftandUAVnavigationschemesfortheaerialmissionsdescribedinChapter 1 .Therefore,thecontroldesigndescribedinthischapterfocusesonthehomingmissiontofacilitatetheAARproblem,whichinvolvestrackingthepositionstatescomputedfromthehomography.Varioustypesofguidancecontrollerscanbeimplementedforthesetypesoftaskoncetherelativepositionandorientationareknown.Dependingonthecontrolobjectivesandhowfastthedynamicsofthemovingtargetare,lowpasslteringoralowgaincontrollermayberequiredtoavoidhighratecommandstotheaircraft.IntheAARproblem,thesuccessofthedockingcontrollerwilldirectlyrelyonseveralcomponents.TherstcomponentistheaccuracyofestimatedtargetlocationwhichduringAARneedstoprecise.Secondly,thedynamicsofthedroguearestochastic.Thiscausesthemodelingtasktobeimpracticalinreplicatingreallifesothecontrollerislimitedtothemodelsconsideredinthedesign.Inaddition,thedrogue'sdynamicsmaynotbedynamicallyfeasiblefortheaircrafttotrackwhichmayfurtherreduceperformance.Lastly,thecontrollerideallyshouldmakepositionmaneuversinstagesbyconsideringthealtitudeasonestage,thelateralpositionasanotherstage,andthedepthpositionasthenalstage.Incloseproximity,thecontrollershouldimplementonlysmallmaneuverstohelpmaintainthevehiclesintheFOV. 15 ]. 117 PAGE 118 110 ].Thestandarddesignapproachwasusedbyconsideringthelongitudinalandlateralstatesseparatelyasintypicalwaypointcontrolschemes.Thisapproachseparatedthecontrolintothreesegments:1)Altitudecontrol,2)HeadingControland3)DepthControl. 9-1 .Therstportionofthissystemisdescribedastheinner-loopwherepitchandpitchrateareusedinfeedbacktostabilizeandtrackapitchcommand.Meanwhile,thesecondportionisreferredtoastheouter-loopwhichgeneratespitchcommandsfortheinner-loopbasedonthecurrentaltitudeerror.Theinner-loopdesignenablesthetrackingofapitchcommandthroughproportional 6 Altitudeholdblockdiagram control.Thispitchcommandinturnwillaffectaltitudethroughthechangesinforcesonthehorizontaltailfromtheelevatorposition.Thetwosignalsusedforthisinner-looparepitchandpitchrate.Thepitchratefeedbackhelpswithshortperioddampingandallowsforratevariationsinthetransientresponse.AleadcompensatorwasdesignedinStevensetal.[ 110 ]toraisethe 118 PAGE 119 9-1 .Thisstructurewillprovidegooddisturbancerejectionduringturbulentconditions.Inaddition,boundswereplacedonthepitchcommandtoalleviateanyaggressivemaneuversduringtherefuelingprocess. 9-2 .Theinner-loop -+fcmd 6y ?Dy 6f Headingholdblockdiagram componentofFigure 9-2 dealswithrolltracking.Thefeedbacksignalsincludebothrollandrollratethroughproportionalcontroltocommandachangeinaileronposition.Theinner-loop 119 PAGE 120 110 ].Consequently,theturnsmootherandcontainslessoscillations.Trackingheadingisnotsufcienttotrackthelateralpositionwiththelevelofaccuracyneededforrefuelingtask.Thenalloopwasaddedtoaccountforanylateraldeviationaccumulatedovertimeduetothedelayinheadingfromposition.Thisdelayismainlyduetothetimedelayassociatedwithsendingarollcommandandproducingaheadingchange.Therefore,thisloopwasaddedtogeneratemorerollforcompensation.Theloopcommandedachangeinaileronbasedoftheerrorinlateralposition.Thisdeviation,referredtoasDy,wascomputedbasedontwosuccessivetargetlocationsprovidedbytheestimator.Thecurrentandprevious(x;y)positionsofthetargetwereusedtocomputealineinspacetoprovideareferenceoftheit'smotion.Theperpendiculardistancefromthevehicle'spositiontothislinewasconsideredthemagnitudeofthelateralcommand.Inaddition,thesignofthecommandwasneededtoassignthecorrectdirection.Thisdirectionwasdeterminedfromtherelativeyposition,expressedinthebody-xedframe,thatwasfoundduringestimation.Oncethelateraldeviationwasdetermined,thatsignalwaspassedthroughaPIstructure,asshowninFigure 9-2 .Thegainscorrespondingtotheproportional,kyp,andintegrator,kyi,werethensummedandaddedtocomputethenalrollcommand.ThecompleteexpressionfortherollcommandisshowninEquation 9 120 PAGE 121 121 PAGE 122 8 providesamethodtoestimatetargetsinEuclideanspacewhenfeaturesdoexittheimage.Thismethodworkswellforshortperiodsoftimeafterthetargethasleft;however,thetrustinthepredictedvaluedegradestremendouslyastimeincreases.Consequently,whenafeatureleavestheimagethecontrollercanrelyonthepredictedestimatestosteertheaircraftinitiallybutmayresorttoalternativeapproachesbeyondaspeciedtime.Asalastresort,thecontrollercancommandtheaircrafttoslowdownandregainabroaderperspectiveofthescenetorecapturethetarget. 122 PAGE 123 110 ].Abaselinecontrollerisimplementedthatallowsthevehicletofollowwaypointsbasedentirelyonfeedbackfrominertialsensors.Imagesareobtainedfromasetofcamerasmountedontheaircraft.Thesecamerasincludeastationarycameramountedatthenoseandpointingalongthenose,atranslatingcameraunderthecenterlinethatmovesfromtherightwingtotheleftwing,andapitchingcameramountedunderthecenterofgravity.TheparametersforthesecamerasaregiveninTable 10-1 invaluesrelativetotheaircraftframeandfunctionsoftimegivenastinseconds. Table10-1. Statesofthecameras position(ft) orientation(deg) camera 24 0 0 0 90 0 2 -10 15-3t 0 0 45 0 3 0 0 3 0 45-9t 0 Thecameraparametersarechosenassimilartoanexistingcamerathathasbeenighttested[ 111 ].Thefocallengthisnormalizedsof=1.Also,theeldofviewforthismodelcorrelatestoanglesofgh=32degandgv=28deg.TheresultinglimitsonimagecoordinatesaregiveninTable 10-2 Table10-2. Limitsonimagecoordinates coordinate minimum maximum 0.62 0.53 Avirtualenvironmentisestablishedwithsomecharacteristicssimilartoanurbanenvironment.Thisenvironmentincludesseveralbuildingsalongwithamovingcaranda 123 PAGE 124 10-3 ,isassociatedwitheachfeaturefordirectidenticationinthecameraimages. Table10-3. Statesofthefeaturepoints position(ft) featurepoint north east altitude 1 3500 200 -1500 2 1000+200t 500 -500 3 6000 200cos(2p TheightpaththroughthisenvironmentisshowninFigure 10-1 alongwiththefeatures.TheaircraftinitiallyiesstraightandleveltowardtheNorthbutthenturnssomewhattowardstheEastandbeginstodescendfromadivemaneuver. BFigure10-1. VirtualEnvironmentforExample1:A)3DViewandB)TopView ImagesaretakenatseveralpointsthroughouttheightasindicatedinFigure 10-1 bymarkersalongthetrajectory.ThestatesoftheaircraftattheseinstancesaregiveninTable 10-4 .Theimageplanecoordinates(;n)areplottedinFigure 10-2 forthethreecamerasatt=2sec.ThiscomputationisaccomplishedbyusingEquation 5 inconjunctionwithEquations 3 and 3 whileapplyingtheeldofviewconstraintshowninEquations 3 and 3 .Allthreecamerascontainsomeportionoftheenvironmentalongwithdistinctviewsofthefeaturepointsofinterest.Forexample,camera1containsaforwardlookingviewofastationary 124 PAGE 125 Aircraftstates Time North East Down v w (ft) (ft) (ft) (ft=s) (ft=s) 1196.9 0.44 -2174.8 573.52 56.79 -126.46 4 2112.7 143.04 -1645.4 527.37 -54.94 17.77 6 2989.8 353.63 -1100.7 528.30 4.26 45.57 Time q y q r (deg) (deg) (deg) (deg=s) (deg=s) -13.92 -22.43 -1.81 -13.56 -36.82 1.38 4 -39.21 -37.90 22.79 32.31 28.41 0.04 6 6.98 -14.85 11.93 7.63 -9.34 -1.15 pointonthecornerofabuildingaswellasthemovinghelicopter.Meanwhile,cameras2and3observeatopviewofthemovinggroundvehicletravelingforwarddownaroad.Theseimagemeasurementsprovideasignicantamountofdataandallowformoreadvancedalgorithmsforstateestimationandreconstruction. B CFigure10-2. FeaturepointMmasurementsatt=2secforA)camera1,B)camera2,andC)camera3 Figure 10-3 depictstheopticowcomputedforthesamedatasetshowninFigure 10-2 .Thisimagemeasurementgivesasenseofrelativemotioninmagnitudeanddirectioncausedbycameraandfeaturepointmotion.TheexpressionsrequiredtocomputeopticowconsistedofEqs. 5 5 3 3 3 3 3 and 3 .Inthisexample,theopticowhasmanycomponentscontributingtothenalvalue.Forinstance,theaircraft'svelocityandangularratescontributealargeportionoftheopticowbecauseoftheirlargemagnitudes.Inaddition,thesmallercomponentsinthisexamplearecausedfromvehicleandcameramotionwhicharesmallerinmagnitudebuthaveasignicanteffectondirection.Comparingcameras1and2,there 125 PAGE 126 B CFigure10-3. OpticowMeasurementsatt=2secforA)camera1,B)camera2,andC)camera3 Asummaryoftheresultingimageplanequantities,positionandvelocity,isgiveninTable 10-5 forthefeaturepointsofinterestaslistedinTable 10-3 .Thetableisorganizedbythetimeatwhichtheimagewastaken,whichcameratooktheimage,andwhichfeaturepointisobserved.Thistypeofdataenablesautonomousvehiclestogainawarenessoftheirsurroundingsformoreadvancedapplicationsinvolvingguidance,navigationandcontrol. Table10-5. Imagecoordinatesoffeaturepoints Time(s) Camera FeaturePoint 1 1 0.157 0.162 0.610 0.044 2 1 3 0.051 0.267 0.563 -0.012 2 2 2 -0.308 0.075 0.464 -0.254 2 3 2 0.011 0.077 0.583 -0.235 4 2 2 -0.279 -0.243 -0.823 0.479 4 3 2 0.365 -0.248 -0.701 0.603 6 1 3 0.265 -0.084 0.267 -0.015 10.2.1ScenarioFeaturepointuncertaintyisdemonstratedinthissectionbyextendingthepreviousexample.Thissimulationwillexaminetheuncertaintyeffectsonvisionprocessingalgorithmsusingsimulatedfeaturepointsandperturbedcameraintrinsicparameters. 126 PAGE 127 10-4 alongwithapairofpointsindicatingthelocationsatwhichimageswillbecaptured.Theaircraftisinitiallystraightandlevelthentranslatesforwardwhilerolling4.0degandyawing1.5degatthenallocation. BFigure10-4. Virtualenvironmentofobstacles(solidcircles)andimaginglocations(opencircles)A)3DviewandB)topview Asinglecameraissimulatedatthecenterofgravityoftheaircraftwithlineofsightalignedtothenoseoftheaircraft.Theintrinsicparametersarechosensuchthatfo=1:0anddo=0:0forthenominalvalues.TheimagesforthenominalcameraassociatedwiththescenarioinFigure 10-4 arepresentedinFigure 10-5 toshowthevariationbetweenframes.Thevision-basedfeedbackiscomputedforasetofperturbedcameras.Theseperturbationsrangeasdf2[0:2;0:2]anddd2[0:02;0:02].ObviouslythefeaturepointsinFigure 10-5 willvaryasthecameraparametersareperturbed.Theamountofvariationwilldependonthefeature 127 PAGE 128 BFigure10-5. FeaturepointsforA)initialandB)nalimages point,asnotedinEquations 4 and 4 ,buttheeffectcanbenormalized.Thevariationinfeaturepointgivennominalvaluesofo=no=1isshowninFigure 10-6 forvariationinbothfocallengthandradialdistortion.Thissurfacecanbescaledaccordinglytoconsiderthevariationatotherfeaturepoints.TheperturbedsurfaceshowninFigure 10-6 ispropagatedthroughthreemainimageprocessingtechniquesforanalysis. Figure10-6. Uncertaintyinfeaturepoint 10-6 tofocallengthandradialdistortion.ArepresentativecomparisonofopticowforthenominalcameraandasetofperturbedcamerasisshowninFigure 10-7 128 PAGE 129 B CFigure10-7. Opticalowfornominal(black)andperturbed(red)camerasforA)f=1:1andd=0,B)f=1:0andd=0:01,andC)f=1:1andd=0:01 10-7 indicateseveraleffectsofcameraperturbationsnotedinEquations 4 and 4 .Theperturbationstofocallengthscalethefeaturepointssothemagnitudeofopticowisuniformlyscaled.Theperturbationstoradialdistortionhavelargereffectasthefeaturepointmovesawayfromthecenteroftheimagesotheopticowvectorsarealteredindirection.Thecombinationofperturbationsclearlychangestheopticowinbothmagnitudeanddirectionanddemonstratesthefeedbackvariationsthatcanresultfromcameravariations.Theopticowiscomputedforimagescapturedbyeachoftheperturbedcameras.ThechangeinopticowfortheperturbedcamerasascomparedtothenominalcameraisrepresentedasdJandisboundedinmagnitude,asderivedinEquation 4 ,byDJ.ThegreatestvalueofdJpresentedbythesecameraperturbationsiscomparedtotheupperboundinTable 10-6 .ThesenumbersindicatethevariationsinopticowareindeedboundedbythetheoreticalboundderivedinChapter 4 andindicatethelevelofowvariationsinducedfromthevariationsincameraparameters. Table10-6. Effectsofcameraperturbationsonopticow PerturbationSet Analyzeonlywithdf kDJk kdJk kDJk kdJk kDJk 0.0476 0.0040 0.0040 0.0496 0.0543 0.0476 0.0020 0.0040 0.0252 0.0543 0.0476 0.0020 0.0040 0.0264 0.0543 0.0476 0.0040 0.0040 0.0543 0.0543 129 PAGE 130 10-8 showsthequalityoftheestimation.Essentially,theepipolargeometryrequiresafeaturepointinoneimagetoliealongtheepipolarline.Thisepipolarlineisconstructedbytheintersectionbetweentheplaneformedbytheepipolarconstraintandtheimageplaneatthelastmeasurement.ThedatainFigure 10-8 showthefeaturesinthesecondimagedoindeedlieexactlyontheepipolarlines. BFigure10-8. Epipolarlinesbetweentwoimageframes:A)initialframeandB)nalframewithoverlayedepipolarlinesfornominalcamera Theintroductionofuncertaintyintotheepipolarconstraintwillcausevariationsintheessentialmatrixwhichwillalsopropagatethroughthecomputationoftheepipolarline.Thesevariationsintheepipolarlinearevisualcluesofthequalityoftheestimateintheessentialmatrix.Thesevariationscanoccuraschangesintheslopeandthelocationoftheepipolarline.Figure 10-9 illustratestheepipolarvariationsduetoperturbationsondf=0:1anddd=0:01tothecameraparameters.Thefeaturepointswithuncertaintyandthecorrespondingepipolarlinewasplottedalongwiththenominalcasetoillustratethevariations.Thekeypointinthisguresisthesmallvariationsintheslopeoftheepipolarlinesandthesignicantvariationsinfeature 130 PAGE 131 BFigure10-9. Uncertaintyresultsforepipolargeometry:A)initialframeandB)nalframewithoverlayedepipolarlinesforcameraswithf1:0andd=0:0(black)andf=1:1andd=0:01(red) Theessentialmatrixiscomputedfortheimagestakenusingasetofcameramodels.EachmodelisperturbedfromthenominalconditionusingthevariationsinFigure 10-6 .Thechangeinestimatedstatesbetweennominalandperturbedcamerasisgivenbydqovertheuncertaintyrangeandisbounded,asderivedinEquation 4 ,byDq.ThevalueofdqforaspecicperturbationisshownincomparisontotheupperboundinTable 10-7 whichalsoindicatethevariationinenteriesoftheessentailmatrixwhichpropagatetothecamerastates. Table10-7. Effectsofcameraperturbationsonepipolargeometry PerturbationSet Analyzeonlywithdf kDqk kdqk kDqk kdqk kDqk 293.14 4.45 4.45 288.75 297.34 293.14 2.19 2.19 288.75 297.34 293.14 2.11 2.19 288.75 297.34 293.14 4.15 4.45 288.75 297.34 131 PAGE 132 10-10 toindicatethatallerrorswerelessthan106. Figure10-10. Nominalestimationusingstructurefrommotion Thedepthsarealsoestimatedusingstructurefrommotiontoanalyzeimagesfromtheperturbedcameras.ArepresentativesetoftheseestimatesareshowninFigure 10-11 ashavingclearerrors.Aninterestingfeatureoftheresultsisthedependenceonsignoftheperturbationtofocallength.Essentially,thesolutiontendstoestimateadepthlargerthanactualwhenusingapositiveperturbationandadepthsmallerthanactualwhenusinganegativeperturbation.Sucharelationshipisadirectresultofthescalingeffectthatfocallengthhasonthefeaturepoints.Estimatesarecomputedforeachoftheperturbedcamerasandcomparedtothenominalestimate.Theworst-caseerrorsinestimationarecomparedtothetheoreticalbound,giveninEquation 4 ,totheseerrors.ThesenumbersshowninTable 10-8 indicatethevariationinstructurefrommotiondependsonthesignoftheperturbation.Theapproachisactuallyseentobelesssensitivetopositiveperturbations,whichcausesalargerestimateindepth,thantonegativeperturbations.Also,thetheoreticalboundwasgreaterthan,orequalto,theerrorcausedbyeachcameraperturbation. 132 PAGE 133 B CFigure10-11. Estimationusingstructurefrommotionfornominal(black)andperturbed(red)cameraswithA)f=1:1andd=0,B)f=1:0andd=0:01,andC)f=1:1andd=0:01 Effectsofcameraperturbationsonstructurefrommotion PerturbationSet Analyzeonlywithdf kDzk kdzk kDzk kdzk kDzk 4679.8 75.02 75.02 4903.5 4903.5 4679.8 36.90 75.02 1076.6 4903.5 4679.8 35.73 75.02 498.76 4903.5 4679.8 70.34 75.02 1092.5 4903.5 1 involvingapolicepursuitisdemonstratedthroughthissimulation.Thesetupconsistedofthreevehicles:anUAVyingabovewithamountedcamera,electronicsandcommunication,areferencegroundvehiclewhichisconsideredthepolicepursuitcar,andatargetvehicledescribingthesuspectsvehicle.ThegoalofthismissionisfortheUAVtotrackbothvehiclesintheimage,whilereceivingpositionupdatesfromthereferencevehicle,andestimatethetarget'slocationusingtheproposedestimationalgorithm.ThecamerasetupconsideredinthisproblemconsistofasingledownwardpointingcameraattachedtotheUAVwithxedpositionandorientation.Whileinightthecamerameasuresandtracksfeaturepointsonboththetargetvehicleandthereferencevehicleforuseintheestimationalgorithm.Thissimulationassumesperfectcameracalibration,featurepointextraction,and 133 PAGE 134 7 thegeometryofthefeaturepointsarepredescribedandaknowndistanceisprovidedforeachvehicle.AfurtherdescriptionofthisassumptionisgiveninSection 7.2.2 .Futureworkwillexaminemorerealisticaspectsofthecamerasystemtoreproduceamorepracticalscenarioandtrytoalleviatethelimitationsimposedonthefeaturepoints. 10-12 ,forillustration.Theinitialframeforthissimulationislocatedattheaircraft'spositionwhenthesimulationstarts.Thevelocityofthegroundvehicleswerescaleduptotheaircraft'svelocitywhichresultedinlargedistancesbutalsohelpedtomaintainthevehiclesintheimage. BFigure10-12. Vehicletrajectoriesforexample3:A)3DviewandB)topview ThepositionandorientationstatesofthethreevehiclesareplottedinFigures 10-13 10-18 andallarerepresentedintheinertialframe,E.Thepositionsindicatethatallthreevehicle 134 PAGE 135 B CFigure10-13. PositionstatesoftheUAVwithon-boardcamera:A)North,B)East,andC)Down B CFigure10-14. AttitudestatesoftheUAVwithon-boardcamera:A)Roll,B)Pitch,andC)Yaw B CFigure10-15. Positionstatesofthereferencevehicle(pursuitvehicle):A)North,B)East,andC)Down 135 PAGE 136 B CFigure10-16. Attitudestatesofthereferencevehicle(pursuitvehicle):A)Roll,B)Pitch,andC)Yaw B CFigure10-17. Positionstatesofthetargetvehicle(chasevehicle):A)North,B)East,andC)Down B CFigure10-18. Attitudestatesofthetargetvehicle(chasevehicle):A)Roll,B)Pitch,andC)Yaw motionfromtheUAVtothetargetofinterest.ThenormerrorofthismotionaredepictedinFigure 10-19 .Theseresultsindicatethatwithsyntheticimagesandperfecttrackingofthevehiclesnearlyperfectmotioncanbeextracted.Oncenoiseintheimageortrackingisintroducedtheestimatesofthetargetdeterioratequicklyevenwithminutenoise.Inaddition,imageartifactssuchasinterferenceanddropoutswillalsohaveanadverseaffectonhomographyestimation. 136 PAGE 137 BFigure10-19. NormerrorforA)relativetranslationandB)relativerotation Figures 10-20 and 10-21 showtherelativetranslationandrotationdecomposedintotheirrespectivecomponentsandexpressedinthebodyframe,B.Thesecomponentsrevealtherelativeinformationneededforfeedbacktotrackorhomeinonthetargetofinterest. B CFigure10-20. Relativepositionstates:A)X,B)Y,andC)Z B CFigure10-21. Relativeattitudestates:A)Roll,B)Pitch,andC)Yaw 137 PAGE 138 10-22 ofthecameraviewdepictingthevehiclesandthesurroundingscene.Theredvehiclewasdesignatedasthereferencewhereasthegreyvehiclewasthetargetvehicle.Thenextstepinthisprocessistoimplementanactualfeaturetrackingalgorithmonthesyntheticimagesthatfollowsthevehicles.Thismodicationalonewilldegradethehomographyresultsimmenselyduetothetroublesomecharacteristicsofafeaturepointtracker. Figure10-22. Virtualenvironment 1 describedthemotivationandthebenetsofAAR,thissectionwilldemonstrateitbycombiningthecontroldesigngiveninChapter 9 withthehomographyresultinChapter 7 toformaclosed-loopvisualservocontrolsystem.ThevehiclesinvolvedinthissimulationincludesaReceiverUAVinstrumentedwithasinglecamera,atankeraircraftalso 138 PAGE 139 5 withadditionalstatessuchasV,a,b,theaccelerationterms,Machnumber,anddynamicpressure.Althoughthecontrollerwillnotuseallstates,theassumptionoffullstatefeedbackwasmadetoallowallstatesaccessiblebythecontroller.Thecontrollerusesthesestatesoftheaircraftalongwiththeestimatedresultstocomputeactuatorcommandsaroundthespeciedtrimcondition. 139 PAGE 140 9 isintegratedandtunedforthenonlinearF-16modeltoaccomplishthissimulation.Itwasassumedthatfullstatefeedbackoftheaircraftstatesweremeasurableincludingposition.Theunitsusedinthissimulationaregiveninftanddegwhichmeansthegainsdeterminedinthecontrolloopswerealsofoundbasedontheseunits.First,thepitchtrackingforaltitudecontrollerisconsidered.Theinner-loopgainsforthiscontrolleraregivenaskq=3andkq=2:5.ThebodediagramforpitchcommandtopitchangleisdepictedinFigure 10-23 forthespeciedgains.Thisdiagramrevealsthedamping 140 PAGE 141 Figure10-23. Inner-looppitchtopitchcommandBodeplot ThestepresponseforthepitchcontrollerisgiveninFigure 10-24 andshowsacceptableperformance.Theouter-loopcontrolwillnowbedesignedusingthiscontrollertotrackaltitude. Figure10-24. Pitchanglestepresponse 141 PAGE 142 10 andwasdesignedinStevensetal.[ 110 ].AstepresponseforthiscontrollerisillustratedinFigure 10-25 thatshowsasteadyclimbwithnoovershootandasteady-stateerrorof2ft.ThisresponseisrealisticforanF-16butnotidealforautonomousrefuelingmissionwheretolerancesareonthecmlevel.Thealtitudetransitionisslowduetothecompensatorbutonemayconsidermoreaggressivemaneuversformissionssuchastargettrackingthatmayrequireadditionalagility. Figure10-25. Altitudestepresponse Thenextstagethatwastunedinthecontroldesignwastheheadingcontroller.Theinner-loopgainswerechosentobekf=5:7andkp=1:6fortherolltracker.ThebodediagramforthiscontrollerofrollcommandtorollangleisshowninFigure 10-26 whichshowsattenuationinthelowerfrequencyrange.Thisattenuationremovesanyhighfrequencyresponsefromtheaircraftwhichisdesiredduringarefuelingmission,especiallyincloseproximity.Meanwhile,thecouplingbetweenlateralandlongitudinalstatesduringaturnwascounteracted 142 PAGE 143 Figure10-26. Inner-looprolltorollcommandBodeplot ThestepresponseforthisbankcontrollerisillustratedinFigure 10-27 .Thetrackingperformanceisacceptablebasedonarisetimeof0:25sec,anovershootof6%andlessthana3%steady-stateerror.Theouter-looptuningforheadingcontrollerconsistedofrsttuningthegainonheadingerror.Againofky=1:5waschosenforthismissionwhichdemonstratedacceptableperformance.Figure 10-28 showstheheadingresponseusingthiscontrollerforarightturn.Theresponserevealasteadyrisetime,noovershoot,andasteady-stateerroroflessthan2deg.Finally,thelooppertainingtolateraldeviationwastunedtokyp=0:5andkyi=0:025whichproducedreasonabletrackingandsteadyerrorforlateralposition.Thenalstageofthecontrollerinvolvestheaxialposition.Thisstagewasdesignedtoincreasethrustbasedonavelocitycommandoncethelateralandaltitudestateswerealigned.Aproportionalgainwastunedbasedonvelocityerrortoachieveaslowsteadyapproachspeed 143 PAGE 144 Rollanglestepresponse Figure10-28. Headingresponse tothetarget.Againofkx=3:5wasdeterminedforthisloopwhichgeneratesthedesiredapproach.Lastly,tohelplimitthenumberoftimesthefeaturepointsexittheeldofviewalimitwasimposedonthepitchangle.Thislimitwasenforcedwhentheapproachachieveaspecieddistance.Forthisexample,thedistancewassettowithin75ftintheaxialpositionofthebody-xedframewhichwasdeterminedexperimentallyfromthetarget'ssize. 144 PAGE 145 10-29 forpositionandFigure 10-30 fororientationandrevealedacorrectt.Thisresultdemonstratesthefunctionalityoftheestimatorwithanaccuracyontheorderof109.ThiserrorwasplottedinFigure 10-31 forbothpositionandorientation. B CFigure10-29. Open-loopestimationoftarget'sinertialposition:A)North,B)East,andC)Altitude B CFigure10-30. Open-loopestimationoftarget'sinertialattitude:A)Roll,B)Pitch,andC)Yae Furthermore,theclosed-loopresultsforthissimulationwereplottedinFigures 10-32 and 10-34 forpositionandorientationofboththereceiveraircraftandthetargetdroguerelativetotheearth-xedframe.Thetrackingofthiscontrollershowedreasonableperformanceforthedesiredpositionandheadingsignals.Theremainingorientationangleswerenotconsideredinfeedbackbutestimatedforthepurposeofmakingsurethedrogue'spitchandrollarewithinthedesiredvaluesbeforedocking.AsseeninFigure 10-32 ,thereceiverwasabletotrackthegrossmotionofdroguewhilehavingsomedifcultlytrackingtheprecisemotion. 145 PAGE 146 BFigure10-31. NormerrorfortargetstateestimatesA)translationandB)rotation B CFigure10-32. Closed-looptargetpositiontracking:A)North,B)East,andC)Altitude ThecomponentsofthepositionerrorbetweenthereceiveranddrogueareshowninFigure 10-33 toillustratetheperformanceofthetrackingcontroller.Theseplotsdepicttheinitialoffseterrordecayingovertimewhichindicatesthereceiver'srelativesdistanceisdecreasing.Thealtitudeshowedaquickclimbresponsewhereastheresponseinaxialpositionwasaslowsteadyapproachwhichwasdesiredtolimitlargechangesinaltitudeandangleofattack.Thelateralpositionisstableforthetimeperiodbutcontainsoscillationsduetherolltoheadinglag.TheorientationanglesshowninFigure 10-34 indicatetheEuleranglesforforthebody-xedtransformationscorrespondingtothebody-xedframeofthereceiverandthebody-xedframeofthedrogue.Recall,theonlysignalbeingtrackedinthecontroldesignwasheading.Thisselectionallowedtheaircrafttosteerandmaintainaighttrajectorysimilartothedroguewithoutaligningrollandpitch.Thereceivershouldyclosetoatrimconditionratherthenmatchingthefullorientationofthedrogue,asillustratedinFigure 10-34 forpitchangle. 146 PAGE 147 B CFigure10-33. Positiontrackingerror:A)North,B)East,andC)Altitude TheerrorinheadingisdepictedinFigure 10-35 whichshowsacceptabletrackingperformanceoverthetimeinterval. B CFigure10-34. Targetattitudetracking:A)Roll,B)Pitch,andC)Yaw Figure10-35. Trackingerrorinheadingangle Theresultsshownintheseplotsindicatethatthetrackinginthelateralpositionandaltitudearenearlysufcientfortherefuelingtask.Thesimulationrevealsboundederrorsinthese 147 PAGE 148 8 willhelptoaidthecontroller,oratleasttohelpdeterminearegionofwherethefeaturesmostlikelyhavetraveled. 7 .Toseewhatlevelsofvariationsexistintheseresultsanuncertaintyanalysiswasperformed.Chapter 4 derivedamethodtocomputeworse-caseboundsonstateestimatesfromthehomographyapproachusingvisualinformation.ThetechniquedescribedinChapter 4 wasusedforthisuncertaintyanalysis.ThetargetestimatesforabsolutepositionandorientationalongwithupperandlowerboundswerecomputedforthissimulationandareshowninFigures 10-36 and 10-37 .These 148 PAGE 149 B CFigure10-36. Target'sinertialpositionwithuncertaintybounds:A)North,B)East,andC)Altitude B CFigure10-37. Target'sinertialattitudewithuncertaintybounds:A)Roll,B)Pitch,andC)Yaw Themaximumuncertaintiesintargetpositionrelativetotheearth-xedframearesummarizedinTable 10-9 .Meanwhile,Table 10-10 containsthemaximumuncertaintiesintargetorientation.Thethreelevelsofuncertaintyareincludedinthesetables.Thiscomparisonhelpstoverifythatthemaximumstatevariationcorrespondstothemaximumcameraparameter 149 PAGE 150 Table10-9. Maximumvariationsinpositionduetoparametricuncertainty uncertaintyparameter north(ft) 4.10 20.54 10.53 14.40 15.09 30.82 Table10-10. Maximumvariationsinattitudeduetoparametricuncertainty uncertaintyparameter 0 0 4.48 2.29 7.94 3.48 150 PAGE 151 151 PAGE 152 152 PAGE 153 8 intotherefuelingsimulationwillhelpthecontrollerbyprovidingstateestimatewhenthetargetexitstheeldofview. 153 PAGE 154 [1] SecretaryofDefense,UnmannedAircraftSystemsRoadmap2005-2030,website:http://uav.navair.navy.mil/roadmap05/roadmap.htm Grasmeyer,J.M.,andKeennon,M.T.,DevelopmentoftheBlackWidowMicro-AirVehicle,39thAerospaceSciencesMeetingandExhibit,AIAA2001-0127,Reno,NV,January2001. [3] Beard,R.,Kingston,D.,Quigley,M.,Snyder,D.,Christiansen,R.,Johnson,W.,Mclain,T.,andGoodrich,M.,AutonomousVehicleTechnologiesforSmallFixedWingUAVs,AIAAJournalofAerospaceComputing,Information,andCommunication,Vol.2,No.1,January2005,p.92-108. [4] Kingston,D.,Beard,R.,McLain,T.,Larsen,M.,andRen,W.,AutonomousVehicleTechnologiesforSmallFixedWingUAVs,AIAA2ndUnmannedUnlimitedSystems,Technologies,andOperationsAerospace,Land,andSeaConferenceandWorkshopandExhibit,AIAA-2003-6559,SanDiego,CA,September2003. [5] Frew,E.,ObserverTrajectoryGenerationforTarget-MotionEstimationUsingMonocularVision,PhDDissertation,StanfordUniversity,August2003. [6] Sattigeri,R.,Calise,A.J.,SooKim,B.,Volyanskyy,K.,andNakwan,K.,-DOFNonlinearSimulationofVision-basedFormationFlight,AIAAGuidance,NavigationandControlConferenceandExhibit,AIAA-2005-6002,SanFrancisco,CA,August2005. [7] Beard,R.,Mclain,T.,Nelson,D.,andKingston,D.,DecentralizedCooperativeAerialSurveillanceusingFixed-WingMiniatureUAVs,IEEEProceedings:SpecialIssueonMulti-RobotSystems,Vol.94,Issue7,July2006,pp.1306-1324. [8] Wu,A.D.,Johnson,E.N.,andProctor,A.A.,Vision-AidedInertialNavigationforFlightControl,AIAAGuidance,Navigation,andControlConferenceandExhibit,AIAA2005-5998,SanFrancisco,CA,August2005. [9] EttingerS.M.,Nechyba,M.C.,Ifju,P.G.,andWaszak,M.,Vision-GuidedFlightStabilityandControlforMicroAirVehicle,IEEE/RSJinternationalConferenceonIntelligentRobotsandSystem,Vol.3,September/October2002,pp.2134-2140. [10] Kehoe,J.,Causey,R.,Abdulrahim,M.,andLind,R.,WaypointNavigationforaMicroAirVehicleusingVision-BasedAttitudeEstimation,AIAAGuidance,Navigation,andControlConferenceandExhibit,AIAA-2005-6400,SanFrancisco,CA,August2005. [11] Abdulrahim,M.,andLind,R.,ControlandSimulationofaMulti-RoleMorphingMicroAirVehicle,AIAAGuidance,NavigationandControlConferenceandExhibit,AIAA-2005-6481,SanFrancisco,CA,August2005. [12] Abdulrahim,M.,Garcia,G.,Ivey,G.F.,andLind,R.,FlightTestingofaMicroAirVehicleUsingMorphingforAeroservoelasticControl,AIAAStructures,StructuralDynamics,andMaterialsConference,AIAA-2004-1674,PalmSprings,CA,April2004. 154 PAGE 155 Garcia,H.,Abdulrahim,M.,andLind,R.,RollControlforaMicroAirVehicleUsingActiveWingMorphing,AIAAGuidance,NavigationandControlConferenceandExhibit,AIAA-2003-5347,Austin,TX,August2003. [14] Waszak,M.R.,Jenkins,L.N.,andIfju,P.G.,StabilityandControlPropertiesofanAeroelasticFixedWingMicroAirVehicle,AIAAAtmosphericFlightMechanicsConferenceandExhibit,AIAA2001-4005,Montreal,Canada,August2001. [15] Kimmett,J.,Valasek,J.,andJunkinsJ.K.,VisionBasedControllerforAutonomousAerialRefueling,IEEEInternationalConferenceonControlApplications,Glasgow,Scotland,U.K.,September2002,pp.1138-1143. [16] Tandale,M.D.,Bowers,R.,andValasek,J.,RobustTrajectoryTrackingControllerforVisionBasedProbeandDrogueAutonomousAerialRefueling,AIAAGuidance,Navigation,andControlConferenceandExhibit,AIAA2005-5868,SanFrancisco,CA,August2005. [17] Lucas,B.,andKanade,T.,AnIterativeImageRegistrationTechniquewithanApplicationtoStereoVision,ProceedingsoftheDARPAImageUnderstandingWorkshop,1981,pp.121-130. [18] Tomasi,C.,andKanade,T.,DetectionandTrackingofPointFeatures,Tech.ReportCMU-CS-91-132,CarnegieMellonUniversity,April1991. [19] Kanade,T.,Collins,R.,Lipton,A.,Burt,P.,andWixson,L.,AdvancesinCooperativeMulti-SensorVideoSurveillance,ProceedingsofDARPAImageUnderstandingWork-shop,Vol.1,November1998,pp.3-24. [20] Piccardi,M.,BackgroundSubtractionTechniques:AReview,IEEEInternationalConferenceonSystems,ManandCybernetics,TheHague,TheNetherlands,October2004. [21] Schunck,B.G.,MotionSegmentationbyConstraintLineClusttering,IEEEWorkshoponComputerVision:RepresentationandControl,1984,pp.58-62. [22] Ridder,C.,Munkelt,O.,andKirchner,H.,AdaptiveBackgroundEstimationandForegroundDetectionusingKalman-Filtering,InternationalConferenceonRecentAdvancedinMechatronics,Istanbul,Turkey,June1995,pp.193-199. [23] Bailo,G.,Bariani,M.,Ijas,P.,andRaggio,M.,BackgroundEstimationwithGaussianDistributionforImageSegmentation,aFastApproach,IEEEInternationalWorkshoponMeasurementSystemsforHomelandSecurity,ContrabandDetectionandPersonalSafety,Orlando,FL,March2005. [24] Friedman,N.,andRussel,S.,ImageSegmentationinVideoSequences:AProbabilisticApproach,InternationalProceedingsoftheThirteenthConferenceofUncertaintyinArticialIntelligence(UAI),Providence,RI,August1997. 155 PAGE 156 Sheikh,Y.,andShah,M.,BayesianObjectDetectioninDynamicScenes,IEEEComputerSocietyConferenceonComputerVisionandPatternRecognition,SanDiego,CA,June2005. [26] Stauffer,C.,andGrimson,W.E.L.,AdaptiveBackgroundMixtureModelsforReal-TimeTracking,IEEEConferenceonComputerVisionandPatternRecognition,FortCollins,CO,June1999,pp.246-252. [27] Toyama,K.,Krumm,J.,Brumitt,B.,andMeyers,B.,Wallower:PrinciplesandPracticeofBackgroundMaintenance,InternationalConferenceonComputerVision,Corfu,Greece,September1999. [28] Zhou,D.,andZhang,H,ModiedGMMBackgroundModelingandOpticalFlowforDetectionofMovingObjects,IEEEInternationalConferenceonSystem,Man,andCybernetics,BigIsland,Hawaii,October2005. [29] Nelson,R.C.,QualitativeDetectionofmotionbyaMovingObserver,InternationalJournalofComputerVision,Vol.7,No.1,1991,pp.33-46. [30] Thompson,W.B.,andPong,T.G.,DetectingMovingObjects,InternationalJournalofComputerVision,Vol.4,1990,pp.39-57. [31] Odobez,J.M.,andBouthemy,P.,DetectionofMultipleMovingObjectsUsingMultiscaleMRPWithCameraMotionCompensation,IEEEInternationalConferenceonImageProcessing,Austin,TX,November1994,pp.245-249. [32] Irani,M.,Rousso,B.,andPeleg,S.,DetectingandTrackingMultipleMovingObjectsUsingTemporalIntegration,EuropeanConferenceonComputerVision,SantaMargheritaLigure,Italy,May1992pp.282-287. [33] Torr,P.H.S.,andMurray,D.W.,StatisticalDetectionofIndependentMovementfromaMovingCamera,ImageandComputing,Vol.11,No.4,May1993,pp.180-187. [34] Gandhi,T.,Yang,M.T.,Kasturi,R.,Camps,O.,Coraor,L.,andMcCandless,J.,DetectionofObstaclesintheFlightPathofanAircraft,IEEETransactionsonAerospaceandElectronicSystems,Vol.39,No.1,January2003,pp.176-191. [35] Irani,M.,andAnandan,P.,AUniedApproachtoMovingObjectDetectionin2Dand3DScenes,IEEETransactionsonPatternAnalysisandMAchineIntelligence,Vol.20,No.6,June1998. [36] Sharma,R.,andAloimonos,Y.,EarlyDetectionofIndependentMotionfromActiveControlofNormalFlowPatterns,IEEETransactionsonSystems,Man,andCybernetics,Vol.26,No.1,February1996. [37] Frazier,J.,andNevatia,R.,DetectingMovingObjectsfromaMovingPlatform,IEEEInternationalConferenceonRoboticsandAutomation,Nice,France,May1992,pp.1627-1633. 156 PAGE 157 Liu,Y.,Huang,T.S.,andFaugeras,O.D.,DeterminationofCameraLocationfrom2Dto3DlineandPointCorrespondence,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.12,No.1,January1990pp.28-37. [39] Longuet-Higgins,H.C.,AComputerAlgorithmforReconstructingaScenefromTwoProjections,Nature,Vol.293,September1981,pp.133-135. [40] Heeger,D.J.,andJepson,A.D.,SubspaceMethodforRecoveringRigidMotion1:AlgorithmandImplementation,InternationalJournalofComputerVision,Vol.7,No.2,January1992. [41] Gutmann,J.S.,andFox,D.,AnExperimentalComparisonofLocalizationMethodsContinued,IEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,Lausanne,Switzerland,October2002. [42] Martin,M.C.,andMoravec,H.,RobotEvidenceGrids,TechnicalReportCMU-RI-TR-96-06,RoboticsInstitute,CarnegieMellonUniversity,March1996. [43] Olson,C.F.,SelectingLandmarksforLocalizationinNaturalTerrains,AutonomousRobots,Vol.12,2002,pp.201-210. [44] Olson,C.F.,andMatthies,L.H.,MaximumLikelihoodRoverLocalizationbyMatchingRangeMaps,IEEEInternationalConferenceonRoboticsandAutomation,Leuven,Belgium,May1998,pp.272-277. [45] Volpe,R.,Estlin,T.,Laubach,S.,Olson,C.,andBalaram,J.,EnhancedMarsRoverNavigationTechniques,IEEEInternationalConferenceonRoboticsandAutomation,SanFrancisco,CA,April2000,pp.926-931. [46] Gurl,P.,andRotstein,H.,PartialAircraftStateEstimationfromVisualMotionUsingtheSubspaceContraintsApproach,JournalofGuidance,Control,andDynamics,Vol.24,No.5,September-October2001,pp.1016-1028. [47] Markel,M.D.,Lopez,J.,Gebert,G.,andEvers,J.,Vision-AugmentedGNC:PassiveRangingfromImageFlow,AIAAGuidance,Navigation,andControlConferenceandExhibit,AIAA2002-5026,Monterey,CA,August2002. [48] Webb,T.P.,Prazenica,R.J.,Kurdila,A.J.,andLind,R.,Vision-BasedStateEstimationforAutonomousMicroAirVehicles,AIAAGuidance,Navigation,andControlConfer-enceandExhibit,Providence,RI,August2004. [49] Webb,T.P.,Prazenica,R.J.,Kurdila,A.J.,andLind,R.,Vision-BasedStateEstimationforUninhibitedAerialVehicles,AIAAGuidance,Navigation,andControlConferenceandExhibit,AIAA2005-5869,SanFrancisco,CA,August2005. [50] Chatterji,G.B.,Vision-BasedPositionandAttitudeDeterminationforAircraftNightLanding,AIAAGuidance,NavigationandControlConferenceandExhibit,AIAA-96-3821,July1996. 157 PAGE 158 Silveira,G.F.,Carvalho,J.R.H.,Madirid,M.K.,Rives,P.,andBueno,S.S.,AFastVision-BasedRoadFollowingStrategyAppliedtotheControlofAerialRobots,IEEEProceedingsofXIVBrazilianSymposiumonComputerGraphicsandImageProcessing,1530-1834/01,Florianopolis,Brazil,October2001,pp.226-231. [52] Soatto,S.,andPerona,P.,DynamicVisualMotionEstimationfromSubspaceConstraints,IEEE,0-8186-6950-0/94,1994,pp.333-337. [53] Soatto,S.,Frezza,R.,andPeronaP.,MotionEstimationviaDynamicVision,IEEETransactionsonAutomaticControl,Vol.41,No.3,March1996,pp.393-413. [54] Soatto,S.,andPerona,P.,Recursive3-DMotionEstimationUsingSubspaceConstraints,InternationalJournalofComputerVision,Vol.22,No.3,1997,pp.235-259. [55] Soatto,S.,andPerona,P.,Reducing'StructurefromMotion',IEEE,1063-6919/96,1996,pp.825-832. [56] Soatto,S.,andPerona,P.,VisualMotionEstimationfromPointFeatures:UniedView,IEEEInternationalConferenceonImageProcessing,Vol.3,October1995,pp.21-24. [57] Soatto,S.,andPerona,P.,Reducing'StructurefromMotion':AGeneralFrameworkforDynamicVisionPart1:Modeling,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.20,No.9,September1998,pp.933-942. [58] Soatto,S.,andPerona,P.,Reducing'StructurefromMotion':AGeneralFrameworkforDynamicVisionPart2:ImplementationandExperimentalAssessment,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.20,No.9,September1998,pp.943-960. [59] Erol,A.,Bebis,G.,Nicolescu,M.,Boyle,R.D.,andTwombly,X.,AReviewonVision-BasedFullDOFHandMotionEstimation,IEEEComputerSocietyInternationalConferenceonComputerVisionandPatternRecognition,SanDiego,CA,June2005. [60] Huang,T.S.,andNetravali,A.N.,MotionandStructurefromFeatureCorrespondences:AReview,ProceedingsoftheIEEE,Vol.82,No.2,February1994,pp.252-268. [61] Stewart,C.V.,RobustParameterEstimationinComputerVision,SocietyofIndustrialandAppliedMathematics,Vol.41,No.3,1999,pp.513-537. [62] Weng,J.,Huang,T.S.,andAhuja,N.,MotionandStructurefromTwoPerspectiveViews:Algorithms,ErrorAnalysis,andErrorEstimation,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.11,No.5,May1989,pp.451-476. [63] Jianchao,Y.,ANewMethodforPassiveLocationEstimationfromImageSequenceUsingAdaptiveExtendedKalmanFilter,InternationalConferenceonSignalProcessing,Beijing,China,October1998,pp.1002-1005. [64] Qian,G.,Kale,G.,andChellappa,R.,RobustEstimationofMotionandStructureusingaDiscreteH-innityFilter,IEEE0-7803-6297-7/00,2000,pp.616-619. 158 PAGE 159 Weng,J.,Ahuja,N.,andHuang,T.S.,OptimalMotionandStructureEstimation,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.15,No.9,September1993,pp.864-884. [66] Blostein,S.D.,Zhao,L.,Chann,andR.M.,Three-DimensionalTrajectoryEstimationfromImagePositionandVelocity,IEEETransactionsonAerospaceandElectronicSystems,Vol.36,No.4,October2000,pp.1075-1089. [67] Broida,T.J.,Chandrashekhar,S.,andChellappa,R.,Recursive3-DMotionEstimationfromaMonocularImageSequence,IEEETransactionsonAerospaceandElectronicSystems,Vol.26,No.4,1990,pp.639-656. [68] Aidala,V.J.,KalmanFilterBehaviorinBearings-OnlyTrackingApplications,IEEETransactionsonAerospaceandElectronicSystems,Vol.15,No.1,1979,pp.29-39. [69] Bolger,P.L.,TrackingaManeuveringTargetUsingInputEstimation,IEEETransac-tionsonAerospaceandElectronicSystems,Vol.23,No.3,1987,pp.298-310. [70] Gavish,M.,andFogel,E.,EffectofbiasonBearings-OnlyTargetLocation,IEEETransactionsonAerospaceandElectronicSystems,Vol.26,No.1,January1990,pp.22-25. [71] Peach,N.,Bearings-OnlyTrackingUsingaSetofRange-ParameterizedExtendedKalmanFilters,IEEProceedings-ControlTheoryandApplications,Vol.142,No.1,January1995,pp.73-80. [72] Taff,L.G.,TargetLocalizationfromBearings-OnlyObservations,IEEETransactionsonAerospaceandElectronicSystems,Vol.33,No.1,January1997,pp.2-9. [73] Guanghui,O.,Jixiang,S.,Hong,L.,andWenhui,W.,Estimating3DMotionandPositionofaPointTarget,ProceedingsofSPIE,Vol.3173,1997,pp.386-394. [74] Fang,Y.,Dawson,D.M.,Dixon,W.E.,andQueiroz,M.S.de,.5DVisualServoingofWheeledMobileRobots,IEEEConferenceonDecisionandControl,LasVegas,NV,December2002,pp.2866-2871. [75] Chen,J.,Dixon,W.E.,Dawson,D.M.,andChitrakaranV.K.,VisualServoTrackingControlofaWheeledMobileRobotwithaMonocularFixedCamera,IEEEConferenceonControlApplications,Taipei,Taiwan,September2004,pp.1061-1066. [76] Chen,J.,Dawson,D.M.,Dixon,W.E.,andBehal,A.,AdaptiveHomography-BasedVisualServoTrackingforFixedandCamera-in-HandCongurations,IEEETransactionsonControlSystemsTechnology,Vol.13,No.5,September2005,pp.814-825. [77] Mehta,S.S.,Dixon,W.E.,MacArthur,D.,andCrane,C.D.,VisualServoControlofanUnmannedGroundVehicleviaaMovingAirborneMonocularCamera,IEEEAmericanControlConference,Minneapolis,MN,June2006. 159 PAGE 160 Junkins,J.L.,andHughes,D.,Vision-BasedNavigationforRendezvous,DockingandProximityOperations,AASGuidanceandControlsConference,Breckenridge,CO,February1999. [79] Alonso,R.,Crassidis,J.L.,andJunkins,J.L.,Vision-BasedRelativeNavigationforFormationFlyingofSpacecraft,AIAAGuidance,NavigationandControlConferenceandExhibit,AIAA-2000-4439,Denver,CO,August2000. [80] Houshangi,N.,ControlofaRoboticManipulatortoGraspaMovingTargetusingVision,IEEEInternationalConferenceonRoboticsandAutomation,CH2876-1/90/0000/0604,Cincinnati,Ohio,1990,pp.604-609. [81] Hansen,J.L.,Murry,J.E.,andCampos,N.V.,TheNASADrydenAARProject:AFlightTestApproachtoanAerailRefuelingSystem,AIAAAtmosphericFlightMechanicsConferenceandExhibit,Providence,RhodeIsland,August2004. [82] Chang,P.,andHebert,P.,RobustTrackingandStructurefromMotionthroughSamplingBasedUncertaintyRepresentation,InternationalConferenceonRoboticsandAutoma-tion,WashingtonD.C.,May2002. [83] Oliensis,J.,ExactTwo-ImageStructureFromMotion,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.24,No.12,December2002,pp.1618-1633. [84] Svoboda,T.,andSturm,P.,BadlyCalibratedCamerainEgo-MotionEstimation-PropagationofUncertainty,InternationalConferenceComputerAnalysisofImageandPatterns,Kiel,Germany,September1997,pp.183-190. [85] Zhang,Z.,DeterminetheEpipolarGeometryanditsUncertainty:AReview,Interna-tionalJournalofComputerVision,Vol.27,No.2,1998,pp.161-195. [86] Qian,G.,andChellappa,R.,StructureFromMotionUsingSequentialMonteCarloMethods,InternationalConferenceOnComputerVision,Vancouver,Canada,July2001,pp.614-621. [87] Franke,U.,andHeinrich,S.,FastObstacleDetectionforUrbanTrafcSituations,IEEETransactionsonIntelligentTransportationSystems,Vol.3,No.3,September2002,pp.173-181. [88] Bhanu,B.,Das,S.,Roberts,B.,andDuncan,D.,ASystemforObstacleDetectionDuringRotorcraftLowAltitudeFlight,IEEETransactionsonAerospaceandElectronicSystems,Vol.32,No.3,July1996,pp.875-897. [89] Huster,A.,Fleischer,S.D.,andRock,S.M.,DemonstrationofaVision-BasedDead-ReckoningSystemforNavigationofanUnderwaterVehicle,OCEANS'98ConferenceProceedings,0-7803-5045-6/98,Vol.1,September1998,pp.326-330. [90] Roderick,A.,Kehoe,J.,andLind,R.,Vision-BasedNavigationusingMulti-RateFeedbackfromOpticFlowandSceneReconstruction,AIAAGuidance,Navigation,andControlConferenceandExhibit,SanFrancisco,CA,August2005. 160 PAGE 161 Papaikolopoulos,N.P.,Nelson,B.J.,andKhosla,P.K.,SixDegree-of-FreedomHand/EyeVisualTrackingwithUncertainParameters,IEEETransactionsonRoboticsandAutomation,Vol.11,No.5,October1995,pp.725-732. [92] Sznaier,M.,andCamps,O.I,ControlIssuesinActiveVision:OpenProblemsandSomeAnswers,IEEEConferenceonDecisionandControl,Tampa,FL,December1998,pp.3238-3244. [93] Frezza,R.,Picci,G.,andSoatto,S.,Non-holonomicModel-basedPredictiveOutputTrackingofanUnknownThree-dimensionalTrajectory,IEEEConferenceonDecisionandControl,Tampa,FL,December1998,pp.3731-3735. [94] Papaikolopoulos,N.P.,Khosla,P.K.,andKanade,T.,VisualTrackingofaMovingTargetbyaCameraMountedonaRobot:ACombinationofControlandVision,IEEETransactionsonRoboticsandAutomationVol.9,No.1,February1993,pp.14-35. [95] Papaikolopoulos,N.P.,andKhosla,P.K.,AdaptiveRoboticVisualTracking:TheoryandExperiments,IEEETransactionsonAutomaticControl,Vol.38,No.3,March1993,pp.429-445. [96] Zanne,P.,Morel,G.,andPlestan,F.,RobustVisionBased3DTrajectoryTrackingusingSlidingModeControl,IEEEInternationalConferenceonRoboticsandAutomation,SanFrancisco,CA,April2000,pp.2088-2093. [97] Zergeroglu,E.,Dawson,D.M.,deQueiroz,M.S.,andBehal,A.,Vision-BasedNonlinearTrackingControllerswithUncertainRobot-CameraParameters,IEEE/ASMEInternationalConferenceonAdvancedMechanics,Atlanta,GA,September1999,pp.854-859. [98] Valasek,J.,Kimmett,J.,Hughes,D.,Gunnam,K.,andJunkins,J.L.,VisionBasedSensorandNavigationSystemforAutonomousAerialRefueling,AIAA's1stTechnicalConferenceandWorkshoponUnmannedAerospaceVehicles,Portsmouth,Virginia,May2002. [99] Pollini,L.,Campa,G.,Giulietti,F.,andInnocenti,M.,VirtualSimulationSet-UpforUAVsAerialRefueling,AIAAModelingandSimulationTechnologiesConference,Austin,TX,August2003. [100] No,T.S.,andCochan,J.E.,DynamicsandCOntrolofaTeatheredFlightVehicle,JournalofGuidance,Control,andDynamics,Vol.18,No.1,January1995,pp.66-72. [101] Forsyth,D.A.,andPonce,J.,ComputerVision:AModernApproach,Prentice-HallPublishers,UpperSaddleRiver,NJ,2003. [102] Ma,Y.,Soatto,S.,Kosecka,andSastry,S.S.,AnInvitationto3-DVision:FromImagestoGeometricModels,Springer-VerlagPublishing,NewYork,NY,2004. [103] Faugeras,O.,Three-DimensionalComputerVision,TheMITPress,CambridgeMassachusetts,2001. 161 PAGE 162 Castro,G.J.,Nieto,J.,Gallego,L.M.,Pastor,L.,andCabello,E.,AnEffectiveCameraCalibrationMethod,IEEE0-7803-4484-7/98,1998,pp.171-174. [105] Tsai,R.Y.,AVersatileCameraCalibrationTechniqueforHight-Accuracy3DMachineVisionMetrologyUsingOff-theshelfTVCamerasandLens,IEEEJournalofRoboticsandAutomation,Vol.RA-3,No.4,August1987,pp.323-344. [106] Harris,C.,andStephens,M.,ACombinedCornerandEdgeDetector,ProceedingsoftheAlveyVisionConference,1988,pp.147-151. [107] Canny,J.F.,AComputationalApproachtoEdgeDetection,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.8,No.6,November1986,pp.679-698. [108] Etkin,B.,andReid,L.D.,DynamicsofFlight:StabilityandControl,JohnWiley&Sons,NewYork,1996. [109] Nelson,R.C.,FlightStabilityandAutomaticControl,McGraw-Hill,NewYork,1989. [110] Stevens,B.L.,andLewis,F.L.,AircraftControlandSimulation,JohnWiley&Sons,Inc.,NewYork,1992. [111] Kehoe,J.J.,Causey,R.S.,Arvai,A.,andLind,R.,PartialAircraftStateEstimationfromOpticalFlowusingNon-Model-BasedOptimization,IEEEAmericanControlConference,Minneapolis,MN,June2006. [112] KaminskiJ.Y.,andTeicher,M.,AGeneralFrameworkforTrajectoryTriangulation,JournalofMathematicalImagingandVision,Vol.21,2004,pp.27-41. [113] AvidanS.,andShashuaA.,TrajectoryTriangulation:3DReconstructionofMovingPointsfromaMonocularImageSequence,IEEETransactionsonPatternAnalysisandMachineIntelligence,Vol.22,No.4,2000,pp.348-357. [114] Fitzgibbon,A.W.,andZisserman,A.,MultibodyStructureandMotion:3DReconstructionofIndependentlyMovingObjects,EuropeanConferenceonComputerVision,Dublin,Ireland,July2000,Vol.1,pp.891-906. [115] Han,M.,andKanade,T.,ReconstructionofaScenewithMultipleLinearlyMovingObjects,InternationalJournalofComputerVision,Vol.59,No.3,2004,pp.285-300. [116] Ozden,K.E.,Cornelis,K.,VanEycken,L.,andVanGool,L.,Reconstructing3DTrajectoriesofIndependentlyMovingObjectsusingGenericConstraints,JournalofComputerVisionandImageUnderstanding,Vol.96,No.3,2004,pp.453-471. [117] YuanC.,andMedioni,G.,DReconstructionofBackgroundandObjectsMovingonGroundPlaneViewedfromaMovingCamera,IEEEConferenceonComputerVisionandPatternRecognition,NewYork,NY,June2006. 162 PAGE 163 Dobrokhodov,V.N.,Kaminer,I.I.,Jones,K.D.,andGhabcheloo,R.,Vision-BasedTrackingandMotionEstimationforMovingTargetsusingSmallUAVs,AmericanControlConference,Minneapolis,MN,June2006. [119] Faugeras,O.,andLustman,F.,MotionandStructureFromMotioninaPiecewisePlanarEnvironment,InternationalJournalofPatternRecognitionandArticialIntelligence,Vol.2,No.3,pp.485-508,1988. [120] Zhang,Z.,andHanson,A.R.,ScaledEuclidean3DReconstructionBasedonExternallyUncalibratedCameras,IEEESymp.onComputerVision,1995,pp.37-42. [121] Zhu,Q.,AStochasticAlgorithmforObstacleMotionPredictioninVisualGuidanceofRobotMotion,IEEEInternationalConferenceonSystemsEngineering,Pittsburgh,PA,August1990. 163 PAGE 164 RyanScottCauseywasborninMiami,Florida,onMay10,1978.Hegrewupinastablefamilywithonebrotherinatypicalsuburbanhome.Duringhisteenageyearsandintoearlyadolescence,Ryanbuiltandmaintainedasmallbusinessprovidinglawncaretothelocalneighborhood.Thetoolsacquiredfromthisworkcarriedoverintohiscollegecareer.AftergraduatingfromMiamiKillianSeniorHighSchoolin1996,RyanattendedMiamiDadeCommunityCollegeforthreeyearsandreceivedanAssociateinArtsdegree.AtransferstudenttotheUniversityofFlorida,Ryanwaspreparedtotacklethestressesofauniversityasidefromthepoorstatisticsontransferstudents.Afewyearslater,hereceivedaBachelorofScienceinAerospaceEngineeringwithhonorsin2002andwasconsideredinthetopthreeofhisclass.RyansoonafterchosetoattendgraduateschoolbackattheUniversityofFloridaunderDr.RickLindintheDynamicsandControlsLaboratory.Duringthesummertime,RyaninternedtwiceatHoneywellSpaceSystemsasaSystemsEngineerinClearwater,FLandonceatTheAirForceResearchLaboratoryinDayton,OH.Vision-basedcontrolofautonomousairvehiclesbecamehisinterestandheisnowpursuingadoctoratedegreeonthistopic.RyanwasawardedaNASAGraduateStudentResearchProgram(GSRP)fellowshipin2004forhisproposedinvestigationonthisresearch. 164 |