<%BANNER%>

Vector-Based Ground Surface and Object Representation Using Cameras

Permanent Link: http://ufdc.ufl.edu/UFE0024966/00001

Material Information

Title: Vector-Based Ground Surface and Object Representation Using Cameras
Physical Description: 1 online resource (153 p.)
Language: english
Creator: Lee, Jae
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2009

Subjects

Subjects / Keywords: autonomous, computer, finder, lane, model, pattern, recognition, road, vector, vehicle, vision
Mechanical and Aerospace Engineering -- Dissertations, Academic -- UF
Genre: Mechanical Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Computer vision plays an important role in many fields these days. From robotics to bio-medical equipment to the car industry to the semi-conductor industry, many applications have been developed for solving problems using visual information. One computer vision application in robotics is a camera-based sensor mounted on a mobile robot vehicle. Since the late 1960s, this system has been utilized in various fields, such as automated warehouses, unmanned ground vehicles, space robots, and driver assistance systems. Each system has a different mission, like terrain analysis and evaluation, visual odometers, lane departure warning systems, and identification of such moving object as other cars and pedestrians. Thus, various features and methods have been applied and tested to solve different computer vision tasks. A main goal of this vision sensor for an autonomous ground vehicle is to provide such continuous and precise perception information as traversable paths, future trajectory estimations, and lateral position error corrections with small data size. To accomplish these objectives, multi- camera-based Path Finder and Lane Finder Smart Sensors were developed and utilized on an autonomous vehicle at the University of Florida s Center for Intelligent Machines and Robotics (CIMAR). These systems create traversable area information for both an unstructured road environment and an urban environment in real time. Extracted traversable information is provided to the robot s intelligent system and control system in vector data form through the Joint Architecture for Unmanned Systems (JAUS) protocol. Moreover, a small data size is used to represent the real world and its properties. Since vector data are small enough for storing, retrieving, and communication, traversability data and its properties are stored at the World Model Vector Knowledge Store for future reference.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Jae Lee.
Thesis: Thesis (Ph.D.)--University of Florida, 2009.
Local: Adviser: Crane, Carl D.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2009
System ID: UFE0024966:00001

Permanent Link: http://ufdc.ufl.edu/UFE0024966/00001

Material Information

Title: Vector-Based Ground Surface and Object Representation Using Cameras
Physical Description: 1 online resource (153 p.)
Language: english
Creator: Lee, Jae
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2009

Subjects

Subjects / Keywords: autonomous, computer, finder, lane, model, pattern, recognition, road, vector, vehicle, vision
Mechanical and Aerospace Engineering -- Dissertations, Academic -- UF
Genre: Mechanical Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Computer vision plays an important role in many fields these days. From robotics to bio-medical equipment to the car industry to the semi-conductor industry, many applications have been developed for solving problems using visual information. One computer vision application in robotics is a camera-based sensor mounted on a mobile robot vehicle. Since the late 1960s, this system has been utilized in various fields, such as automated warehouses, unmanned ground vehicles, space robots, and driver assistance systems. Each system has a different mission, like terrain analysis and evaluation, visual odometers, lane departure warning systems, and identification of such moving object as other cars and pedestrians. Thus, various features and methods have been applied and tested to solve different computer vision tasks. A main goal of this vision sensor for an autonomous ground vehicle is to provide such continuous and precise perception information as traversable paths, future trajectory estimations, and lateral position error corrections with small data size. To accomplish these objectives, multi- camera-based Path Finder and Lane Finder Smart Sensors were developed and utilized on an autonomous vehicle at the University of Florida s Center for Intelligent Machines and Robotics (CIMAR). These systems create traversable area information for both an unstructured road environment and an urban environment in real time. Extracted traversable information is provided to the robot s intelligent system and control system in vector data form through the Joint Architecture for Unmanned Systems (JAUS) protocol. Moreover, a small data size is used to represent the real world and its properties. Since vector data are small enough for storing, retrieving, and communication, traversability data and its properties are stored at the World Model Vector Knowledge Store for future reference.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Jae Lee.
Thesis: Thesis (Ph.D.)--University of Florida, 2009.
Local: Adviser: Crane, Carl D.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2009
System ID: UFE0024966:00001


This item has the following downloads:


Full Text

PAGE 1

1 VECTORBASED GROUND SURFACE AND OBJECT REPRESENTATION USING CAMERAS By JAESANG LEE A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2009

PAGE 2

2 2009 Jaesang Lee

PAGE 3

3 To my dear parents, whose continuous love, pra y er and support made my journey possible and T o my lovely wife and daughter Youngah Kim and Jeanne

PAGE 4

4 ACKNOWLEDGMENTS I thank my advisor, Dr. Carl Crane for his kind support and advice throughout my whole graduate education. I also thank my committee, Dr. Warren Dixon, Dr. John K. Schueller, Dr. Antonio Arroyo, and Dr. Douglas Dankel, for their support and guidance. I also thank Center for Intelligent Machine s and Robotics ( CIMAR ) manager, Dave Armstrong, and friend and leader Shannon Ridgeway for their valuable discussion s This work was made possible by the Air Force Research Laboratory (AFRL) at Tyndall Air Force Base, Florida. Thank you to all the AFRL staff. Finally I thank all my colleagues and friends at CIMAR. T he Defense Advanced Research Projects Agency ( DARPA) Grand Challenge 2005 team members, Tomas G alluzzo, Danny Kent, Bob Touchton, and Sanjay Solanki with helped and guide d my first step s of the research journey And thanks to t he DARPA Urban Challenge 2007 team members Antoin Baker, Greg Garcia, Nicholas Johnson, JeanFrancois Kamath, Eric Thorn, Steve Velat, and Jihyun Yoon, who all work ed well and ha d a n exciting time together a nd Ryan Chilton for helping with both testing and discussion.

PAGE 5

5 TABLE OF CONTENTS page ACKNOWLEDGMENTS ...............................................................................................................4 LIST OF TABLES ...........................................................................................................................8 LIST OF FIGURES .........................................................................................................................9 LIST OF ABBREVIATIONS ........................................................................................................12 ABSTRACT ...................................................................................................................................13 C H A P T E R 1 INTRODUCTION ..................................................................................................................15 Motivation ...............................................................................................................................15 Literature Review ...................................................................................................................15 Terrain A nalysis ..............................................................................................................16 Feature selection .......................................................................................................16 Classifier ...................................................................................................................18 Monocular vision, stereo vision, LADAR, or sensor fusion ....................................18 Lane Tracking ..................................................................................................................19 Feature selection .......................................................................................................19 Lane extraction .........................................................................................................19 Lane model ...............................................................................................................20 Tracking method (estimator) ....................................................................................21 2 RESEARCH GOAL ...............................................................................................................22 Problem Statement ..................................................................................................................22 Development ...........................................................................................................................22 Further Assumptions ...............................................................................................................23 3 PATH FINDER SMART SENSOR .......................................................................................24 Introduction .............................................................................................................................24 Feature Space ..........................................................................................................................24 RGB Color Space ............................................................................................................25 Normalized RGB Color Space ........................................................................................25 Training Area ..........................................................................................................................26 Classifier .................................................................................................................................27 Maximum Likelihood ......................................................................................................27

PAGE 6

6 Mixture of Gaussians .......................................................................................................28 Expectation and Maximization ( EM ) A lgorithm ............................................................30 Sub Sampl ing Method ............................................................................................................34 Pixel B ased S egmentation ...............................................................................................34 Block B ased S egmentation .............................................................................................34 C oordinate Transformation .....................................................................................................35 4 LANE FINDER SMART SENSOR .......................................................................................53 Introduction .............................................................................................................................53 Camera Field of View .............................................................................................................54 Canny Edge Detector ..............................................................................................................54 First Order Line Decision .......................................................................................................55 Hough Transform ............................................................................................................56 Lane Line Search .............................................................................................................58 Polynomial Line Decision ......................................................................................................58 Cubic Splines ...................................................................................................................59 Control Points ..................................................................................................................61 Lane Model .............................................................................................................................62 Lane Estimation ......................................................................................................................63 Lane Center Correction ...........................................................................................................65 Lane Correction by Two Cameras ...................................................................................65 Lane Correction by One Camera .....................................................................................68 Lane Property ..........................................................................................................................68 Uncertainty Management ........................................................................................................70 5 VECTORBASED GROUND AND OBJECT REPRESENTATION ...................................97 Introduction .............................................................................................................................97 Approach .................................................................................................................................97 Ground Surface R epresentation ..............................................................................................99 Static Object Representation .................................................................................................100 World Model Vector Knowledge Store ................................................................................101 6 EXPERIMENTAL RESULTS AND CONCLUSIONS .......................................................111 Platform ................................................................................................................................111 Hardware ...............................................................................................................................111 Software ................................................................................................................................112 Results ...................................................................................................................................114 LFSSWing test results ...................................................................................................115 The PFSS test result .......................................................................................................117 Building vector based map ............................................................................................118 Conclusions ...........................................................................................................................118 Future Work ..........................................................................................................................119

PAGE 7

7 LIST OF REFERENCES .............................................................................................................148 BIOGRAPHICAL SKETCH .......................................................................................................153

PAGE 8

8 LIST OF T ABLES Table page 31 Reference point locations in image domain and grid map domain. Grid map size is 60 x 60 meters and resolution is 0.25 meter. .....................................................................36 41 Vehicle travel distance per camera frame rate ...................................................................59 42 Two side cameras horizontal pixel resolution on 640 x 380 image. ................................66 43 Side cameras pixel location by real distance on 640 380 resolution image. ..................67 44 Lane center correction experimental JAUS message definition ........................................67 45 The LFSS center cameras hori zontal pixel resolution on 640 x 218. ...............................68 46 Lane color lookup table ....................................................................................................69 51 Raster based traversability grid map data size ...................................................................98 52 Vectorbased traversability data size .................................................................................99 54 The PFSS ground surface DB table. ................................................................................102 61 Camera and lens specification. ........................................................................................112

PAGE 9

9 LIST OF FIGURES Figure page 31 CIMAR Navigator II Path Finder system for D ARPA Grand Challenge 2005. ................39 32 Sample u nstructured environment. ....................................................................................40 33 Sample structured road environment.. ...............................................................................41 34 RGB ( r ed, green blue) c olor s pace ...................................................................................42 35 Training area selection. ......................................................................................................43 36 RGB distribution of road training area and background training area ...............................44 37 Classified road image s .......................................................................................................45 38 Classifier e rror for Citra and DARPA Grand C hallenge 2005 s cene with varying number s of mixture of Gaussian distributions .. ................................................................46 39 Two Gaussian distribution absolute mean values over the iteration step for the DARPA Grand C hallenge 2005 image. .............................................................................47 310 Classification result ............................................................................................................48 311 Traversability grid map [ Solanki 2006] .............................................................................49 312 Coordinate systems. ...........................................................................................................50 313 Perspective transformation reference points. .....................................................................51 314 Transformed image ...........................................................................................................52 41 Camera field of view diagram ............................................................................................71 42 C amera field of view .........................................................................................................72 43 Canny filtered image samples ............................................................................................73 44 Two camera Canny filtered image in various situations ...................................................76 45 Hough space parameters in image coordinate system. ......................................................77 46 Hough space. ......................................................................................................................78 47 Hough line transform results ..............................................................................................82

PAGE 10

10 48 Lane line checking parameters, angle ( L dR). .....................................83 49 Center camera r esults of lane finding with estimated center line .. ....................................84 410 Diagram of Cubic spline and control points. .....................................................................85 411 Two camera overlay image of Hough line in curved road. ................................................86 412 Hough line + curve control points for spline model. .........................................................87 413 Curved lane. .......................................................................................................................88 414 Lane model checking view. A) Straight line, B) curved line. ............................................89 415 Line parameter estimation flowchart. ................................................................................90 416 Two sequence source image and detected (blue) and estimated (orange) line. .................91 417 Least squares angle parameter estimation result. ...............................................................92 418 Lane correction distance definition.. ..................................................................................93 419 Real world and image distance relationship .....................................................................94 420 The LFSS and PFSS calibration image. .............................................................................95 51 Various grid map sizes. ....................................................................................................103 52 T he triangulation map ......................................................................................................104 53 Vector representations. ....................................................................................................105 54 Irregular vector points. .....................................................................................................107 55 The TIN representation of traversability map. .................................................................108 56 Road boundary polygon and stored polygon points (red points). ....................................109 57 Lane objects and stored polygon points (red points). ......................................................110 61 CIMAR Navigator III, Urban NaviGator. .......................................................................123 62 NaviGator III camera sensor systems. ............................................................................124 63 The PFSS software. ..........................................................................................................128 64 The LFSSWing software ..................................................................................................131

PAGE 11

11 65 The LFSS Arbiter and the RN screen. .............................................................................132 66 Urban Navi Gator 2009 system architecture. ....................................................................133 67 The LFSSWing test result at the Gainesville Raceway ..................................................136 68 The LFSSWing test results at the University of Florida campus .....................................139 69 The LFSSWing test results in the urban road. .................................................................140 610 The LFSSWing test results at the University of Florida with nighttime setting. .............141 611 The PFSS test results in the Gainesville Raceway. Source, segmented and the TIN control points images, respectively. ...............................................................................144 612 The PFSS test results in the University of Florida campus. Source, segmented and the TIN control points images, respectively. ..................................................................145 613 The LFSSWing vector based representation. ..................................................................146 614 The PFSS vector based representation ............................................................................147

PAGE 12

12 LIST OF ABBREVIATIONS DARPA Defense Advanced Research Projects Agency DGC 2005 DARPA Grand Challenge 2005 DUC 2007 DARPA Urban Challenge 2007 GPS Global Positioning System IMU Inertial Measurement Unit INS Inertial Navigation System JAUS Joint Archite c ture for Unmanned Systems LADAR Light Detection and Ranging LFSS Lane Finder Smart Sensor component software LFSSWing Lane Finder Smart Sensor Wing component software LFSS Arbiter Lane Finder Arbiter Smart Sensor component software NFM North Finding Module PFSS ( Vision based ) Path Finder Smart Sensor component software RN Roadway Navigation component software TSS ( L ADARbased ) Terrain Smart Sensor component software WMVKS World Model Vector Knowledge Store Smart Sensor component software

PAGE 13

13 Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy V ECTORBASED GROUND SURFACE AND OBJECT REPRESENTATION USING CAMERAS By Jaesang Lee December 2009 Chair: Carl D. Crane III Major: Mechanical Engineering C omputer vision plays an important role in many fields these days From r obotics to bio medical equipment to the car industry to the semi conductor industry, many applications have been developed for solving problems using visual information. One computer vision application in robotics is a camerabased sensor mounted on a mobile robot vehicle S inc e the late 1960s t his system has been utilized in various field s such as automated warehouses unmanned ground vehicle s space robots, and driver assistance system s Each system has a different mission like terrain analysis and evaluation, visual odomet er s lane departure warning system s and identification of such moving object as other car s and pedestrians. Thus, various features and methods have been applied and tested to solve different computer vision tasks. A main goal of this vision sensor for an autonomous ground vehicle is to provide such continuous and precis e perception information as traversable paths future trajectory estimation s, and lateral position error correction s with small data size To accomplish these objectives, multicamerabased Path Finder and Lane Finder Smart Sensor s were developed and utilized on an autonomous vehicle at the University of Florida s Center for Intelligent Machines and Robotics

PAGE 14

14 ( CIMAR) Th ese systems create traversable area information for both an unstruct ured road environment and an urban environment in real time Extracted traversable information is provided to the robot s intelligent system and control system in vector data form through the Joint Architecture for Unmanned Systems ( JAUS ) protocol. Moreover, a small data size is used to represent the real world and its properties Since vector data are small enough for storing, retrieving and communication, traversability data and its propert ies are stored at the World Model Vector Knowle dge Store for future reference.

PAGE 15

15 CHAPTER 1 INTRODUCTION Motivation C omputer vision plays a n important role in many field s these days From r obotics to bio medical equipment to the car industry to the semi conductor industry, many applications have been developed for problem s using visual information Since the price of camera sensors is falling and they are now less expensive and gather more information than light detection and ranging ( LADAR) sensors, their use for applicable problem solving seems assured One computer vision application in r obotics is a c amerabased sensor mounted on a mobile robot vehicle. S ince the late 1960s, t his system has been utilized in various field s such as automated warehouse s unmanned ground vehicle s space robots and driver assistance system s [ Gage 1995, McCall 2006, Matthies 2007] E ach system has a different mission F or example, systems provide terrain analysis and evaluation, visual odomet er s lane departure warning system s and moving object like other cars and pedestrians identification Thus, various features and methods have been applied and tested for solving different computer vision task s A main goal of the vision sensor for autonomous ground vehicle sensor development is to provid e continuous and precis e perception information, such as traversable path, future trajectory estimation, lateral position error correction and moving and static object classification or identification The prior work associated with the tasks of terrain analysis and the identification of roadway lanes is discussed in the literature review s ections. L iterature R eview This chapter describes the various methods and algorithms that were developed to accomplish terrain analysis, lane extraction and tracking. A terrain analysis covers feature

PAGE 16

16 selection, classifier selection and sensor fusion. A lane tracking covers feature selection, lane extraction, lane model assumption, and tracking method. Terrain A nalysis F eature selection T errain analysis and estimation is an essential and basic goal of autonomous vehicle development For offroad situations a lack of environmental structure plus hazardous situation s and difficulty of prediction inhibit an accurate and constant path evaluation The key step of terrain estimation starts with proper feature selection which is applie d to the classifier. I n addition to Light Detection and Ranging ( LADAR) distance information t he visual information is a good source for analyzing traversable terrain Consequently, i mage i ntensity, various color models texture information and edge s have been suggested and utilized for the main input for vision based terrain estimation One primary feature, a n intensity image is used for terrain estimation [ Sukthankar 1993, Pomerleau 1995] The intensity image is easy to understand and requires only a small processing time but lacks information M a ny edge based path finding systems [ Behringer 2004] and stereo vision system s have used gray image s to find disparit ies between two images [ Bertozzi 1998, Kelly 1998] The next and most common ly used feature is the red blue green (RGB) color space The RGB color space is the standard representation in computer and digital cameras; therefore it is widely known and easily analy zed. The Carnegie Mellon Navlab vehicle used a color video camera and the RGB color space as a feature for following a road [ Thorpe 1987]. R oad and nonroad RGB color model s are generated and applied to color classification algorithm s

PAGE 17

17 Another color vision system is t he Supervised Classification Applied to Road Following (SCARF ) system that detects unstructured roads and intersections for intelligent mobile robot s [ Crisman 1993] This system can navigate a road that has no lane s or edge lines, or has degraded road edge s and road surface scars. The most common and challenging camera problem is that data are affected by changes in illumination from one time to another, which results in an inconstant color source This is the biggest challenge to an outdoor visionbased robot system. N ormalized RGB is one method used to overcom e lighting effects in the color based face recognition system [ Vezhnevets 2003] and in many other fields [ Brunl 2008] Normalization of the RGB color space makes the system less sensitive to certain color channel s In terms of hardware attaching a polariz ing filter on the camera has been consider ed O ther research in vehicle sensor development introduces different color spaces like hue and s aturation as additional feature s with in the RGB color space [ Bergquist 1999]. The RGB color space is also used as a primary feature for object detection and identification as well as terrain estimation Another approach to road segmentation is a texture based system Zhang [ 1994] utilized road texture orientation using a gray image as one road segmentation feature, as well as image coordinate information. Chan dler [ 2003] applied texture information to an autonomous lawn mowing machine. The discrete cosine transform ( DCT) and discrete wavelet transformation were applied to distinguishing tall grass areas and mowed grass area.

PAGE 18

18 Classifier A n a utonomous vehicle is a real time outdoor application, and its color camera input size is relatively large F or th ese reason s a simple and strong algorithm is demanded when selecting a classifier The Bayesian algorithm used with t he RGB color space is applied as a classification algorithm in the Navlab vehicle [ Thorpe 1988, Crisman 1993] R oad and nonroad Gaussian model s were generated using color pixel s and applied to whole image pixel s for road classification Davis [ 1995] implemented two different algorithms : the Fisher L inear D iscriminant (FLD) applied to two dimensional RG color feature space and the Backpropagation Neural Network was applied to a three dimensional RGB color feature space. Monocular vision s tereo vision LADAR, or sensor fusion D ifferent type s of perception system s ha ve been developed using different type sensors, such as a monocular vision camera, stereo vision camera, Light Detection and Ranging ( LADAR ) sensor and cameraLADAR fusion sensor [ Rasmussen 2002] The monocular vision system is found in the Carnegie Mellon Navlab and SCARF system [ Thorpe 1988, Davis 1995] A camera is mounted on the front in the middle of the car and faces the ground. The source image is resized for computation efficiency. Sukthankar [ 1993] used a steerable camera to improv e the camera field of view in sharp turn situations Unlike the monocular vision system, a stereo vision system can detect not only terrain area but also obstacle distance The Real time Autonomous Navigator with a Geometric Engine (RANGER) uses stereo vision for determining the traversable area of the terrain [ Kelly 1997]

PAGE 19

19 The Generic Obstacle and Lane Detection (GOLD) stereo vision system detects obstacle s and estimates obstacle distance at a rate of 10Hz [ Bertozzi 1998] Lane T racking Feature selection Ur ban areas ha ve artificial structures for example road lane markings traffic signal s and information signals T he first step for a vision based roadfollowing system in an urban area i s road line marking extraction. T o meet this goal, many systems use different features F or instance, the edge extraction filter method [ Broggi 1995a ] morphological filtering [ Beucher 1994, Yu 1992] template matching [ Broggi 1995b] or frequencybased methods using a gray image or certain single channel image are utilized For the edge extraction method, different spatial edge filters are applied to extract lane markings The Yet Another Road Follower ( YARF ) system and POSTECH research vehicle ( PRV) II used a S obel s patial filter [ Schneiderman 1994, Kluge 1995, Y im 2003] and the CIMAR NaviGator III vision system used a C anny filter for extracting edge information [ Apostoloff 2003, Wang 2004, Velat 2007] The lane finding in another domain ( LANA) system applied a frequency domain feature to a lane extraction algorithm [ Kreucher 1999] The Videobased Lane Estimation and Tracking (VioLet) system used a steerable filter [ McCall 2006] Lane extraction After a n edge based filtered image is generated two steps are required to extract the lane. One is grouping lane pixels among the edge data which contains many noise pixels and the other is computing lan e geometry from the grouped lane pixels T he Hough transform is applied to overcom e imperfect line segment detection cause d by noise a natural property of a road. The Hough line transform is a nonlinear transform from the

PAGE 20

20 image pixel (X Y ) into a parameter space ( rho, z eta) It searches for the local maximum to find the most dominant line segment. Yu [ 1997] Taylor [ 1999] Lee [ 2002] and Velat [ 2007] all applied the Hough transform for dominant line detection of an image. Chapter 4 further explains the Hough transform T he RANdom SAmple Consensus ( RANSAC) algorithm is another good tool for lane extraction in a challenging situation [ Davies 2004] The RANSAC is an iterative method to estimate mathematical model parameters from given data set that contains many outliers. It is a very robust algorithm with many outlier pixels for most hypothes ized line models although it need s many iterative steps to reach the hypothes ized lane model. Kim [ 2006, 2008] uses the RANSAC as a real time lane marking classifier. Lane m odel A l ane model is created using an assumption based on the nature of the structured road. This lane model assumes that the road lane is a linear or parabol ic curve on a flat plane and that road lane width does not change dramatically. The l inear lane model is satisfied in most case s for automated vehicle control system s and lane departure warning system s in both highway and rural environment s [McCall 2005]. The trajectory estimator for autonomous ground vehicles needs a parabolic curve model for accurate results [ Schneiderman 1994, Kluge 1995, and Yu 1997] Many spline curve models are utilized to represent a curved lane for example the C ubic spline line B s pline, and C atmullrom spline Originally, the spline method was developed by the computer graphics field for efficiently representing curve s Thus, e ach spline model ha s its own character and advantage. For instance, different initial assumption s control point locations number of control points and knots are suggested. The spline is also a good tool for representing

PAGE 21

21 a curved lane geometry model. Wang [ 1998, 2000] use d t he C atmullrom spline then the B spline [2004] and [Kim 2006, Kim 2008, Wu2009] uses the C ubic spline for lane structures. Tracking m ethod ( e stimator ) A s tructured road environment does not always provide humanmade line information When a vehicle passes an intersection area, or a blurred road line area or when other vehicle s obstruct part of the line segment the camera cannot see the road line segment. Therefore a lane tracking system is required to overcom e this limitation. The YARK s ystem uses a l east m edian s quare s (LMS ) f ilter for estimating road model parameters [ Kluge 1995]. A Kalman f ilter is applied for estimating the curvature of the highway [ Taylor 1999, Suttorp 2006] The Kalman filter estimates the state of a linear system from a noisy source. A Particle filter is also applied for tracking the road lane. The Particle filter is also known as the Condensation algorithm and it is an abbreviation of Con ditional Dens ity Propagation [Isard 1998] The Particle filter is a model estimation technique using probability distribution over the state space with given information. The a dvantage of the Particle filter is we can appl y it to a nonlinear model unlike the Kalman filter. Apostoloff [ 2003] and Southall [ 2001] used the Particle filter for estimating their lane model.

PAGE 22

22 CHAPTER 2 RESEARCH GOAL Problem S tatement There is a need for a vision sensor that will enable autonomous ground vehicles to provide continuous and precise information such as traversable paths, future trajectory estimation, and lateral position error corrections by the GPS drift combined with small data size. The purpose of this research is to construct a vision sensor that meets th e se needs, yet requires minimal data. F ollowing are the g iven item s or assumptions relevant to reaching that goal First, a vehicle move s manually or autonomously. Second, The position and orientation of the vehicle is measured with respect to a global coordinate system by a Global Positioning System ( GPS) and Inertial Navigation System (INS). Global position information is used to convert local information in to global information. Third, a n a utonomous vehicle mode is provided by behavior special ist software [ Touchton 2006] Therefore the vision sensor software will work differently based on the current vehicle behavior as for example traveling a road, passing an intersection, negotiating an N point turn, and the like Fourth, t hree cameras are use d to captur e different field s of the view source image s The s ource image s quality is reasonably clear to see the environment. Last, the t est area is an o utdoor environment that can include an urban environment and an unstructured environment for example a desert road Development Two computers are used for different range s and different field s of view One computer executes a lane finder system and the other executes a path finder s ystem A l ongrange camera, which is mounted at the center of the s ensor bridge shares its source image with the land finder

PAGE 23

23 and path finder. A system based on two short range cameras is designed for the high resolution lane tracking system. A lane finder computer vision system can detect a road lane in an urban environment using a s hort range field of view camera and a longrange field of view camera. It also identif ies lane propert ies, for example lane width and lane color to better understand the surroundings Different field of view cameras essentially calculate and perform the same task Since the confidence and resolution of each camera result are differ ent each generates a confidence value. A path finder computer vision system can detect the traversable area both in an unstructured road environment and a structured environment such as an urban road. The system use s a n RGB color as a feature and builds probabilistic model s for road and nonroad area s A segmented image is converted to a global coordinate view for assisting the robot s path planning. The p ath finder component software can create both vector and raster output. These vision systems are applied to a real robotic vehicle at update rates of at least 10 Hz The y use a Joint Architecture for Unmanned Systems (JAUS ) compatible software, so these vision components can communicate with any JAUS compatible system or subsystem Further Assumption s The r oad is relatively flat. A c amera captured source image is reasonabl y clear ; therefore a human also can see the road lane from the source image To meet this assumption, auto exposure control functionality is used to obtain clear source image s for various illumination conditions.

PAGE 24

24 CHAPTER 3 PATH FINDER SMART SENSOR Introduction The Path F inder Smart Sensor (PFSS) is a perception element and consists of a single color c amera at the front of a vehicle that is oriented facing the terrain ahead. Its purpose is to classify the area in the cameras range for terrain that is similar to that on wh ich the vehicle is currently traveling and then translate that scene information into a global coordinate system It uses a probabilistic model of a training road area by using color pixel data. Also, i t computes properties of the traversable area for example road type like asphalt, grass, or unpaved road. The PFSS works for both unstructured roads and structured road s Figure 31 ( A ) shows team CIMAR s DARPA Grand Challenge 2005 vehicle called NaviGator II. The NaviGator II was designed for the dese rt autonomous vehicle racing competition, t he DGC 2005. The Path Finder camera of the Navi G ator II is show n in F igure 31 ( B ) and its computer hous ing is show n in F igure 31 ( C ). Figure 3 2 (A) and (B) show sample source image s of the unstructured road environment and f igure 33 (A) and (B) show sample source image s of structured road environment s The PFSS output support s different type s of perception elements, such as L A DAR. The PFSS output is fused by intelligent elements of the robot system for outdoor environment autonomous vehicle driving. Feature Space When a human drives down a road, even if the road does not have any artificial information such as lane marks or road signs the human perception system naturally tr ies to find the best traversable area using its visual sense and any other sense or input information. In

PAGE 25

25 addition, when a human use s visual information, previous experience is added to increase the estimation ability Even though a computer vision system does not have a human s complicated and precise perception system, it can judge the traversable area using a limited amount of information. Like human visual system s most visionbased system s use three major visual features: color, shape and texture [ Apostoloff 2003, Rand 2003]. The P FSS is designed for both unstructured and structured road environments which means shape information is not available all the time. The t exture of a road also differ s from nonroad areas Even if texture is a good feature in this application, texture requires high computation power and it works best on a clear focused image. For the se reason s t he primary feature used for analytical processing is RGB color space or a vari ant version of RGB color space RGB C olor S pace The RGB c olor system is the standard in the world of computers and digital cameras and is a natural choice for color representation Furthermore, RGB is the standard output from CCD/CMOS cameras Therefore it is easily applied to a computer system. Fig ure 34 shows the RGB c olor space cube [ Gonzalez 2004] The RGB color space based system provide s fairly successful result s in most instance s but this feature is not robust enough for the real world outdoor environment. Normalized RGB C olor S pace Since the RGB c olor space does no t have illumination associated color element s selecting the RGB color system has a disadvantage with respect to illumination variation such as in outdoor environment applications The s aturation and hue in the hue, s aturation and value (HSV)

PAGE 26

26 color space are relatively strong color element s but th ese color elements also need additional element intensity for overall classification in the PFSS. The Normalized RGB is a vari ant version of the RGB color space that shows less sensitiv ity to various light conditions [ Vladimir 2003, Brunl 2008] It is insensitive to surface orientation, illumination direction and illumination intensity The Normalized RGB is only dependent on the sensor characteristics and surface albedo [ Luka c 2006] Eq ( 31) and Eq. (3 2) show two different versions of the normalized RGB equation. In this research, Eq. (3 1) is utilized. 222 222 222NormalizedRed NormalizedGreen NormalizedBlue Red RedGreenBlue Green RedGreenBlue Blue RedGreenBlue ( 31) or NormalizedRed ,NormalizedGreen NormalizedBlue Red RedGreenBlue Green RedGreenBlue Blue RedGreenBlue ( 32) Training Area The PFSS classification algorithm uses one or more subimage s as a training area for building a probabilistic model These training area s are selected on the assumption that the vehicle drives on the road. In an unstructured environment like the desert, since traversable road

PAGE 27

27 width and area are relatively narrow er than in an urban environment the camera can see both road and nonroad area s Therefore obtaining both traversable area s and nontraversable areas helps to increase the classification rate. In the structured road environment many background areas were observed to be similar to the drivable road. For example, when a vehicle is undergoing a sh arp turn at crossroads or a vehicle drives in a more than two lane road environment like a highway or an other vehicle that has an asphalt like body color drive s by the PFSS training area the non road model will be similar to the road model. In such cases the classification algorithm only relies on the drivable subimage s probabilistic model Figure 35 (A) shows a training area in a desert environment and F igure 3 5 (B) shows a training area in an urban road environment. Classifier Maximum L ikelihood The Maximum L ikelihood (ML) algorithm which is fundamental ly a probabilistic approach to the problem of pattern classification, is selected for this application. It makes the assumption that the decision problem is posed in probabilistic terms, and that all the relevant probability values are known. While the basic idea underlying the M aximum L ikelihood theory is very simple this is the optimal decision theory und er the Gaussian distribution assumption Therefore, most pixel classification is done using the M aximum Likelihood approach In unstructured road environment the decision boundary that was use d is given by 1 111 /21/2 1 1 222 /21/2 211 exp()() (2)||2 11 exp()(), (2)||2T d T d xx xx ( 33)

PAGE 28

28 where 1[ ],t RedGreenBlue ( 34) 222 RRRGRB 222 1RGGGGB 222 RBGBBB ( 35) Eq ( 34) and Eq ( 35) are the mean vector and covariance matrix respectively of the drivable area RGB pixel s in the training data and 2 and 2 are those of the background pixel s The decision boundary is simplified as follows : 11 11112222()()ln()()lnTT x x x x ( 36) In most pixel classification problems, the logarithm component s in Eq. ( 3 6) are not a dominant factor for classification T herefore to sav e time these two values are not computed since this application requires a real time implementation S o RGB pixels of X which belong to a class, are computed based on the power of the Mahalanobis distance 1 111()()T x x [ Charles 1988]. Mixture of Gaussian s The Maximum Likelihoodbased classification algorithm used for the 2005 D ARPA G rand C hallenge was limited in many situations because its basic assumption was that training areas ha ve only one Gaussian distribution. However i n most cases, the properties of the road training area do not change rapidly and its distribution is Gaussian background sub images do change at every scene and it is inappropriate to assume the data distribution is Gaussian Also, e ven if the road tr aining area model has a Gaussian distribution, the overall road scene is not always a Gaussian distribution because light conditions change and the road image can be contaminated

PAGE 29

29 by leaked oil and so on. For this situation the Expectation Maximization ( EM ) algorithm is implemented for road classification The RGB distribution of the unstructured scene shown in Fig ure 32 ( A ) is portrayed as a three dimensional distribution plot in Fig ure 36 (A), (C), and (E) Fig ure 32 ( A ) is a test area scene at Citra, FL, and Figure 32 ( B ) is a sample scene from the 2005 D ARPA G rand C hallenge course in Nevada. Like most of the outdoor image systems, the NaviGator vision system is susceptible to extreme changes in lighting conditions. Such changes can create shadows in captured images that can result in changes in the image color distribution. Such shadows can be seen in Fig ure 32 ( B ). From the 3D RGB plot s in Figure 3 6 (A) and (B) it is clear that most of the road train ingarea distribution is well cluster ed and can be evaluated with a single Gaussian distribution. However, f or the background, there is no common distribution in the data. F igure s 36 (C ) and (D) show 3 D RGB plot of background area s Thus, if a single Gaussian distribution is assumed for the se areas, a large number of classification errors will be introduced. For this reason the Gaussian based classifiers possess limited performance ability in real world scenes. This argument is further evidenced by the statistical model distribution for the background case in which the distribution is poorly defined. Therefore it is clear that a more sophisticated modeling approach is needed namely a mixture of Gaussian model s In the mixture model, a single statistical model is composed of the weighted sum of multiple Gaussian models Consequently, a mixture modeling classifier represents more complex decision boundaries between the road training subimage and background training subimages. However computing a complex mixture model requires more proce ssing time than computing a single Gaussian model choosing the proper number of mixture Gaussian model s for a real time application is

PAGE 30

30 critical. In this project, a single Gaussian model for the road training sub image and two mixture s of Gaussian model s for the background subimage were selected empirically. E xpectation and M aximization ( EM ) A lgorithm1 The ExpectationMaximization ( EM ) algorithm consist s of three steps The first step is to decid e on the initial value of {,,()}iiiiP ( the mean vector, covariance matrix and probability for ith Gaussian distribution respectively). T he second step is the e xpectation step, which calculates the expected value [|]ijEy for the hidden variable ijy given the current estimate of the parameter T he third step calculat e s a new maximumlikelihood estimate for the parameters k assuming that the value taken on by each hidden variable ijy is its expected value [|]ijEy The process then continues to iterate the second and third steps until the conve rgence condition is satisfied. 1argmax(,),kkQ ( 37) where 11(,)[log(,|)|,]kkQEpxyx ( 38) I t is given that jx is the known image pixel RGB vector and the labeling of the Gaussian distribution is given in the hidden variable iy By completing the data set for jz one can let {,}jjjzxy ( 39) where {1,2,...,} jn and n is the number of background data pixel s 1 This section is referred in [Lee 2006]

PAGE 31

31 In the mixture modeling, it has to compute the value of the hidden variable vector iy where {1,2} i The value iy can be decided by two simple binary random variables 1ijy when jx belongs to Gaussian ,i 0ijy otherwise s o that 12{,}jjjyyy ( 310) The vector jy can only take two sets of distinct values: {1, 0}, {0, 1}. By applying the K mean clustering method for two Gaussian distribution s initial {,,()}iiiP the algorithm clusters the RGB pixels based on attributes into k partitions. Since it w as decide d to employ a twomixture Gaussian model for the two background subimages, the clustering uses two means to compute the covariance and each Gaussian value s probability. The principal difficulty in estimating the maximum likelihood parameters of a mixture model is that it is hard to know the labeling iy of each data pixel. From the value by the kmeans clustering algorithm, one can compute [|]ijEy where the value iy is given by 2 1(|) (|)i i ii iPx y Px ( 311) where 1 1/2 3/211 (|)(|,) exp()(). 2 (2)T i ii iiipxpx xx i ( 312)

PAGE 32

32 N ext one can compute the new {,,()}iiiP in terms of the complete data set {,}jjjzxy Finally, a mixture of Gaussian model is computed from the new mean vector 1 1.n ijj j i n ij jyx y ( 313) Similarly, a new covariance matrix is obtained from 11()()/.nn T i jijiij jjyxx y ( 314) Likewise, the probability of a single Gaussian distribution is obtained from 1() /.n i i ij jn Pyn n ( 315) Finally, the solution to the mixture of Gaussian is found as 2 1122 1(|)(|)()(|)()(|)().ii iPxpxPpxPpxP ( 316) The EM algorithm was simulated on test images to gauge its performance in classifying images. For the purpose of the simulation, a single image was empirically chose n from the test site at Citra as a representative easy case and a second image from the 2005 DARPA Grand C hallenge as a representative hard case. T he Bayesian based classification result and the EMbased classification result are shown in Figure 3 7. In considering the D ARPA G rand C hallenge 2005 image (shown in Figure 33 ( B )) it is clear that there is a considerable affect of shadow/illumination on the road surface leaving the left portion of the road many shades darker than the right. Since slight ly more of the road is covered in shade, the resulting sample

PAGE 33

33 contained in the training segment is biased to the left half of the image as shown in Figure 33 ( B ) Fig ure 38 shows the EM error of the scene s at Citra and the DARPA G rand C hallenge 2005 wh ere the X axis shows the number of Gaussian distribution s for the road training region and background training region. For the DARPA Grand C hallenge 2005 scene with only one Gaussian distribution an error of 24.02% is obtained. However if one Gaussian model for the road training region and two Gaussian model s for the background training region are used an error of 6.12% is obtained. For the C itra case, the RGB distribution of the road training region and the background training region show s that the two distributions do not overlap or intermix. As a result, a single Gaussian distribution for both the road and background training regions yields an error of 9.31%. Similarly, by applying one Gaussian for the road training region and two G aussians for the background training regions, the error is 6.42%. From these results, it is clear that the EM classification algorithm provides better classification performance. Furthermore, it is clear that the EM algorithm can dramatically reduce the c lassification error over the Maximum L ikelihood, in particular in cases of images obscured by shadow, adverse lighting, or vibrationi nduced motion blur. Since the EM algorithm relies on an iterative approach, setting the correct iteration condition is critical to reducing processing time In this application, the mean RGB value is used to control the iterative process wherein the algorithm will continue to iterate until the difference between the previous mean RGB and the current mean RGB of the image i s less than a predefined thresh old value N T he N is determined heuristically and a value of 0.1 wa s used in this research

PAGE 34

34 1 kk iiN ( 317) It should be noted that the variable k in Eq. ( 317 ) represents the current step of the iteration Figure 39 shows the two Gaussian distributions absolute mean value s over the iteration step for the D GC 2005 image SubS ampling Method Pixel B ased S egmentation After building the probability model of the training area s each pixel in a source image is classified as drivable road or nondrivable background. Even if this is a simple procedure, applying each pixel to the classifier requires high computation power and is time consuming. Also it can lead to results that have much less particle noise. Figure 310 (A) is the source image at the Gainesville Raceway and Figure 310 (B) shows a pixel based classification result. Therefore a blockbased sub sampling procedure is suggested to both increase processing speed and reduce noise pixel s Block B ased S egmentation A blockbased segmentation method is used to reduce the segmentation processing time. R egions of N N pixel s are clustered together and each is replaced by its RGB mean value : (,) (,) 2 111NN LL xy ij ijP N ( 318) where is the new pixel mean value for the N N block, P is raw pixel data, (i, j) is raw pixel orientation (x, y) is new block orientation, {1,2,3} L for RGB, and N is block size. Th ese new (x, y) blocks are computed from top left to bottom right of the image and t he cluster or blocks are then classified This result has less noise and it decreases processing time

PAGE 35

35 dramatically since fewer number s of pixel data are applied to the classifier For example, if image size is 320 108, 34,560 pixels are processed to a classifier H owever if 4 4 blocks are computed and then applied to a classifier, only 2 ,160 blocks are applied to the classifier. Therefore a 4 4 blockbased method means 16 times less computation time is demanded. In the current computation environment, t he pixel based segmentation process ing time is 15 millisecond s. The block based method spends less than 1 millisecond which is 15 times faster than a pixel based classification with a 320 108 source im age This processing time depends on computer CPU speed, image size, and block s ize but it is clear that block based sub pixel classification method is faster than the pixel based classification. One disadvantage is that edges are blurred and are not as distinct. F igure 310 (C) and (D) show the 4 4 block and 9 9 blockbased classification results. I n the NaviGator II vision system, a 1 pixel offset corresponds to 1.1 cm in the bottom part of the image and in the NaviGator III vision system, a 1 pixel offset corresponds to 3.73 cm at a distance 10 m eter ahead of the North Finding Module ( NFM ) C oordinate T ransformation After c lassification of the image, the areas denoted as drivable road are converted by a perspective transformation into global coordinates used for the rast er based t raversability grid map T he raster based t raversability grid map tessellates the world around the vehicle into a 2 D grid. The grid is always oriented in a North East direction with the vehicle position ed in the center of the grid [ Solanki 2006] Figure 311 illustrates the t raversability grid map definition In a computer vision system, there is a similarity between a traversability grid map and a single channel image except for the coordination system An image uses a local image

PAGE 36

36 coordination system and a traversability grid map uses a north heading and a global coordinate system which is the same as a GPS coordinate system. Therefore a perspective transformed image is generated by using reference points that match the same points both in the traversability grid map and the image and then appl ying the current GPS position and rotating it by the inertial navigation system ( INS ) yaw data Figure 312 (A) shows the camera view and world view, and Figure 312 (B) shows the relationship between a camera coordinate system, vehicle coordinate system, and world coordinate system. Figure 313 (A) shows reference points in a calibration image F igure 31 3 (B) shows reference points in a 40 40 meter, 1 meter resolution t raversability g rid map. A perspective transformation is applied to convert image domain pixels to traversability grid map pixels. The perspective transformation matrix is calculated based on camera calibration parameters [ Hartley 2004] Table 3 1 shows the location of four reference point s in a n image and a 60 60 meter, 0.25 meter resolution grid map. Table 3 1. Reference point location s in image domain and grid map domain. G rid map size is 60 x 60 meter s and resolution is 0.25 meter. Reference Point # Image coordinate (x,y) G rid map 0 (20, 49) (101 = 121 20, 161 = 121+40) 1 (84,80) (101 = 121 20, 201 = 121+80) 2 (227,85) (141= 121+20, 201 = 121+80) 3 (303,53) (141 = 121+20, 161 = 121+40) This relationship can be described as XHx ( 319)

PAGE 37

37 where X is a vector of traversability grid map coordinates, x is a vector of image plane coordinates and H is a transform ation matrix. In a 2D plane, Eq (319) can be re presented in linear form by 11112131 22122232 33132333Xhhhx Xhhhx Xhhhx ( 320) Eq (3 20) can be rewritten in inhomogeneous form, 111213 1 3313233, hxhyh X X Xhxhyh ( 321) and 212223 2 3313233hxhyh X Y Xhxhyh ( 322) Since there are eight independent element s i n E q (320), only 4 reference points are needed to solve for the H matrix 11 11 1111 1 12 11 1111 1 13 22 2222 2 21 22 2222 2 22 23 44 4444 4 31 44 4444 4 321000 0001 1000 0001 ... ... 1000 0001 h xy XxXyX h xyYxYyY h xy XxXyX h xyYxYyY h h xy XxXyX h xyYxYyY h ( 323) Eq (3 23) is in Ab form. For solving Eq (3 23) equation, a pseudo inverse method is applied: 1().TTAAAB ( 324) Finally, the H transform ation matrix is calculated using Eq (3 24) and it is used to convert the segmented image to a traversability grid map image. Fig ure 314 ( A B ) shows the classified

PAGE 38

38 image, Fig ure 314 ( C D ) shows the transformed image without pixel interpolation, and Figure 314 ( E F ) shows the transformed image with pixel interpolation In each subfigure ( Figure s 314 C D E F ), the vehicle is located at the center of the image (blue square) with its direction indicated by a thin black line. Since a wideangle lens is used in the camera assembly, a broad swath of the road is captured. However, as a result of that wide angle, there is an appreciable distortion in distant regions of the image This distortion results in only a small amount of pixels representing most of the distant portion of the image This fact results in the transformation generating a mapped image with holes in the distant regions of the map. These holes can then be filled by linear interpolation with respect to the row number of each pixel (see Figure 314 ( C E )). After creating a traversability grid map, the GPS and INS yaw data are applied to convert local coordinate s into global coordinate s

PAGE 39

39 A B C Fig ure 31. CIMAR Navigator II P ath Finder system for D ARPA Grand Challenge 2005. A ) c amera a ssembly B ) c omputer and e lectronics e nclosure and C ) computer housing.

PAGE 40

40 A B Fig ure 32. Sample u nstructured environment A) Citra, FL ., B) DARPA Grand C hallenge 2005 course, N V

PAGE 41

41 A B Fig ure 33. Sample structured road environment A) Gainesville Raceway, Gainesville, FL., B) DARPA Urban C hallenge 2007 Area C course, CA.

PAGE 42

42 Fig ure 34. RGB ( r ed, green blue) c olor s pace

PAGE 43

43 A B Fig ure 35. Training area selection A) Unstructured r oad, B) s tructured road.

PAGE 44

44 A B C D E F Fig ure 36. RGB distribution of road training area and background training area A) Road at Citra B) DARPA Grand Challenge 2005 course r oad, C) background at Citra D) DARPA Grand C hallenge 2005 course background, E) r oad and background at Citra and F) DARPA Grand C hallenge 2005 course r oad and background

PAGE 45

45 Fig ure 37. Classified road image s A) Bayesian classification result of Citra, B) Bayesian classification result of DARPA Grand C hallenge 2005 course, C) EM classification result of Citra and D) EM classification result of DARPA Grand C hallenge 2005 course

PAGE 46

46 Fig ure 38. Classifier e rror for C itra and DARPA Grand C hallenge 2005 s cene with varying number s of mixture of Gaussian distributions The X axis shows the number of Gaussian distribution s for the road training region and background training region.

PAGE 47

47 A B Fig ure 39. Two Gaussian distribution absolute mean values over the iteration step for the DARPA Grand C hallenge 2005 image A) First Gaussian mean, B) Second Gaussian mean.

PAGE 48

48 A B C D F ig ure 310. Classification result A) Source image, B) pixel based classification result C ) 4x4 blockbased classification result and D) 9x9 blockbased classification result.

PAGE 49

49 Fig ure 311. Traversability grid map [ Solanki 2006]

PAGE 50

50 A B Fig ure 312. Coordinate systems A) Relationship between camera view and world view and B) r elationship between camera coordinate system, vehicle coordinate system and earth coordinate system.

PAGE 51

51 A B Fig ure 313. Perspective transformation reference points. A) Green dots are r eference points in 320 108 size image at F l avet F ield, University of Florida and B) Green squares are r eference points in a 40 40 m eter t raversability grid map image with 1 meter resolution. Red square is a vehicle

PAGE 52

52 A B C D E F Fig ure 314. Transformed i mage A) Classified image of Citra, B) c lassified image of D ARPA G rand C hallenge 2005 cour s e C) t raversability g rid map image without interpolation of Citra D) t raversability grid map image without interpolation of D ARPA G rand C hallenge 2005 cour s e E) t raversability grid map image with interpolation and F) t raversability grid map image with interpolation

PAGE 53

53 CHAPTER 4 LANE FINDER SMART SENSOR Introduction The first period of autonomous vehicle development involved operat ing in an off road environment where there were no lane demarcations for example the DARPA Grand C hallenge 2005 system and the Mars E xplorer robot vehicle. However at present, most vehicle s drive in an urban environment that is generally paved with lanes defined by painted lines Also, m any autonomous vehicles depend on a Global Positioning System (GPS) to comput e current location and project routes to desired location s Unfortunately, GPS provides less accurate positioning solution s in the urban environment than in an open area environment, since urban infrastructure can blocks satellite signals or cause m ultipath signal s An urban traffic area provide s more traffic facilities than off road or highway. These facilities include bike lane s curbs sidewalk s crossroads, and various traffic signals These urban environment facilities help human driver s understand their surroundings In other word s from the point of view of robot perception, it also increases the complexity of the surroundings On the highway, the lane lines are usually well marked, with no sharp curvature s and no oncoming traffic. Therefore with a driver assistant system, it turns out that highway driving is simpler than innercity driving [ Brunl 2008]. T he outdoor environment also presents an array of difficulties including dynamic lighting conditions, poor road conditions, and road networks that are not consistent from region to region. Because of these limitation s an a uto nomous vehicle that is designed for urban driv ing need s a more adaptive lane tracking system In this chapter, the lane and its property extraction and tracki ng system for an autonomous vehicle in an urban environment are described.

PAGE 54

54 Camera F ield of V iew Two different field o f view camera systems are applied to the Lane Finder Smart Sensor. Each vision system uses a different camera field of view and range to improv e overall lane tracking system output. Figure 41 is a diagram of the camera field of view diagram. Figur e 4 1 (A) shows the longrange field of view camera, mounted at the center of the vehicle and Figure 41 (B) shows the short range but wide field of view of those two cameras Tw o short range cameras capture the source image from the vehicle front, so it provide s enough high resolution and clear road lane source to calculat e not only lane tracking but also lane propert ies Also the two camerabased system provide s a clear lane image even if an other vehicle st ands or travels in front of the robot vehicle Figure 42 (A) and (B) show a two camerabased sample source image. A l ongrange camera is a good source for future trajectory estimation. Its resolution is less than a close view camera, but it can see the area further down the road. Figure 4 2 (C) and (D) shows a long range center camera sample image. Canny E dge D etector A road is defined by several characteristics. These may include color, shape, texture, edges, corners and other features. In particular the road lane is a human made artificial boundary line that is mark ed with color and type information. Consequently, road lane lines contain dominant edge information and this cue is the most important feature for extracting road lane information The edge of an image is created by several factors for example different 3D object depth s on a 2D image plane different reflection rates on a surface, various illumination condition s and sudden object orientation variation Edge detection is accomplished by use of the Canny edge filter [Nix on 2008] T he Canny edge filter utilizes a pre noise removing step and then computes omni directional edge

PAGE 55

55 information Finally it use s two threshold values that requires the detector to utilize much tuning and yield s a sufficiently segmented image. Because of this multistep approach, the Canny edge detector is widely use d in many field s The following four steps comprise the Canny edge filter algorithm: Apply a derivative of a Gaussian noise filter. Compute x, y gradients which are Sobel edge detection and gradient magnitude respectively Apply nonmaximum suppression. o Thin multi pixel wide ridges down to a single pixel width Add l inking and thresholding. o Use l ow, high edge strength thresholds o Accept all edges over low threshold that are connected to edges over high threshold To further enhance the edge detectors performance, only the red channel of the source image is processed. This channel is used because it has the greatest content in both yellow and white and thus can provide the greatest contrast between yellow/white regions and asphalt background. Figure 43 depicts the results of the C anny edge filter with two sets of threshold values and Figure 44 shows Canny filter results in various situations First O rder L ine D ecision T he l ane finder software has two main functions O ne is to establish lane departure warning and tracking, and the other is future trajectory estimation. For the first goal, there is no need for distance to detect a curve d line. If the camera sees a local area, curve d lines looks like straight line s Therefore a firstorder line solution is applied for a lane departure warning system It provides a lane center position with respect to the current driving vehicle position. This

PAGE 56

56 solution is more robus t and faster than a high order line solution; therefore processing update rate s can be increased For trajectory estimation, the camera has to see as far as it can because f a rther sight information provides greater environment al understanding for the vehicle. This information is used by the control element and consequently the control element can manage the vehicle at higher speed However if the camera has f a rther sight, just by the nature of the road, it can see many curved lines. Therefore a highe r order line solution is necessary for future trajectory estimation. Hough T ransform T he Hough transform is a technique that locates a certain shape in an image. T he Hough transform was fi r st implemented to find lines in images [Duda 1972] and it has been extended to further application s It is a robust tool for extracting lines, circles and ellipse s One advantage of the Hough transform in the lane extracting application is that it works well with many noise edge s and/or partial lin e edge s From this point of view the Hough transform can provide the same result as the template matching technique, but it use s many fewer computational resource s [ Nixon 2008]. Two disadvantage s of the Hough transform is that it requires a large storage space and high processing power and it produces as many lines as it can detect from the source image. Therefore searching for road lane lines among all the detected Hough lines is necessary The Hough transform algorithm is as follows : If one consider s a line in an image domain, its equation can be written as ymxc ( 41) or

PAGE 57

57 cossin, xy ( 42) w here is a distance from the image domain origin to the line and is the orientation of (the line from the origin perpendicular to the modeled line) with respect to the X axis as illustrate d in Figure 45. Based on E q ( 42), one can generate a Hough parameter space that plots possible ( ) values which are defined by (x, y) points in the image Finally, strong lines can be selected by searching maximum value s in the Hough parameter space ( ) Figure 46 ( B ) shows the Hough space diagram when using the Canny filtered edge image in Figure 46 (A) The maximum ( ) values in the Hough space is the str ongest line in the image domain. However this method cannot guar antee to extract a road lane line since an edge extracted image contains various noise pixels for many reasons. For example, an old tire track can register as a line in the road f igure 44 (H) and different reflection s from the road can appear to be a straight line like the edge in f igure 44 (B). Figure 47 shows the Hough transform line result in various situations Figure 47 (B) show when a vehicle passes a crossroad area, so stop lines are detected. A lso an other artificial line is easily detected and lane lines are blocked by other object s like grass. This case is shown in f igure 47 (C). Figure 4 7 (D) right image shows random noise edge pixels become a line object by coincidence D ue to the differ ing reflection rate s of the road surface, a different reflection boundary area can create strong edge s and can cause a false line object. Figure 47 (E ) and (F ) left image s illustrate this situation. Because of this Hough transform property and various real world situations two steps are required to correct it; a few lane candidate lines are extracted from the Hough space, then lane lines are searched for among the candidate Hough lines.

PAGE 58

58 Lane L ine S earch The Hough transform for line extraction finds many lines These include not just road lane lines but also other lines, like a crosswalk lane for example [ Hashimoto 2004]. Figure 47 shows sample results of this process. Consequently, among the candidate Hough lines, searching lane lines by using the properties of lane lines is a necessary step. Since road lane lines are parallel to each other and usually at the same angle, two parameters are used for searching the lane lines : angle with respect to vehicle axle and distance from the vehicl e center F igure 48 s hows this angle and distance. A binary search method is applied for detecting only lane lines among the many Hough lines and those line parameter s t hreshold values are selected by the heuristic method. Polynomial L ine D ecision The Hough transform based first order lane line solution is usually enough for lane departure or a lane tracking system However if an autonomous vehicle drives at high speed and/ or drives on a curved road, a n autonomous vehicle control system needs furth er traversable road information. For example, if a vehicle drives at 40 m iles per hour, it means that that vehicle drive s around 18 meter s per second Therefore the perception system has to provide at least an 18 meter traversable area per second from the vehicle location for safe driv ing. T able 41 shows vehicle travel distance per camera frame rate. T he m ain goal of a perception system is to construct as accurat e a representation of the world as possible Clearly, a ccurate and high resolution information helps an autonomous vehicle controller control a vehicle properly and safely Thus, a longrange camera and higher order lane line solution are necessary for future trajectory estimation for high speed driving on a curved

PAGE 59

59 road. Figure 49 (B) shows lane extraction and lane center trajectory error in far sight. This case can cause a future path estimation error. T able 41. Vehicle travel distance per camera frame rate mile/hour 5 10 15 17.5 20 25 30 40 50 60 70 kilometer/hour 8.05 16.09 24.14 28.16 32.19 40.23 48.28 64.37 80.47 96.56 112.65 meter/hour 8046.70 16093.40 24140.10 28163.45 32186.80 40233.50 48280.20 64373.60 80467.00 96560.40 112653.80 meter/sec 2.24 4.47 6.71 7.82 8.94 11.18 13.41 17.88 22.35 26.82 31.29 meter/frame 35 0.06 0.13 0.19 0.22 0.26 0.32 0.38 0.51 0.64 0.77 0.89 30 0.07 0.15 0.22 0.26 0.30 0.37 0.45 0.60 0.75 0.89 1.04 25 0.09 0.18 0.27 0.31 0.36 0.45 0.54 0.72 0.89 1.07 1.25 20 0.11 0.22 0.34 0.39 0.45 0.56 0.67 0.89 1.12 1.34 1.56 17 0.13 0.26 0.39 0.46 0.53 0.66 0.79 1.05 1.31 1.58 1.84 15 0.15 0.30 0.45 0.52 0.60 0.75 0.89 1.19 1.49 1.79 2.09 10 0.22 0.45 0.67 0.78 0.89 1.12 1.34 1.79 2.24 2.68 3.13 5 0.45 0.89 1.34 1.56 1.79 2.24 2.68 3.58 4.47 5.36 6.26 1 2.24 4.47 6.71 7.82 8.94 11.18 13.41 17.88 22.35 26.82 31.29 Cubic S pline s A s pline is a function that describes polynomials for formulating a curve The s pline has been developed and applied in many fields, for example computer aided design ( CAD), computer aided manufacturing ( CAM ) computer graphics (CG), and computer vision (CV) A number of variants have been developed to control the shape of a curve including Bezier curves, B s pline, nonuniform rational B spline (NURBS ) and others [ Sarfraz 2007] In this research, t he C ubic spline method is applied to the curve lane model Unlike other spline model s the C ubic spline passes a set of all N control points and it can use different boundary conditions for each application. The f ollowing is the Cubic spline condition : 1. Curve model is a third order polynomial:

PAGE 60

60 23()iiiiifxabxcxdx ( 43) a nd spacing between the successive data points is 1.iiihxx ( 44) 2. Curves pass through all points 1()().iiiiifxfxy ( 45) 3. The f irst order derivative, the slop e of curve, is equal on either side of a point 1()().iiiifxfx ( 46) 4. The s econd order derivative is equal on either side of a point 1()().iiiifxfx ( 47) 5. For a natural spline case, the second order derivative of the spline at the end points is zero : 10()()0.nnfxfx ( 48) In m atrix f orm one can write : 2 ( h1+ h2) h2 h22 ( h1+ h2) hn 2 2 ( hn 2+ hn 1) f2fi fn 1 = 6 y3 y2h2 y3 y2h2 yn yn 1hn 1 yn 1 yn 2hn 22 Finally, the C ubic spline parameters are calculated as follows: 1 11()/6 /2 2 6 .iiii ii iiiiii i i iiaffh bf yyhfhf c h dy ( 49) Figure 410 is a diagram of a Cubic spline curve.

PAGE 61

61 Even if a vehicle drives a curve d road area, it can be assumed that the vehicle drives in a straight lane from the local point of view. In many cases, a lane curve line start s from a straight line and gradually changes its shape to match the curve. Figure 411 (C) shows this case. Since the Canny edge + Hough transform based first order line solution provide s a fairly robust solution, the Hough transform based line geometry is a good initial source for finding higher order line geometry. Figure s 49 (B) and 411 (B) show the center camera and two camera lane line overlay image using the Hough transform for a curve d road and Figure 412 (B) and (C) shows straight and curve d line result s, respectively. While the C ubic spline is well behaved for a lane curve model representation [Kim 2006, 2009] it is possible to generate an overshot curve because of one or more false intermediate control point s from noise pixels Therefore s electing control points is a key step to creat ing a lane curve model, so it has to be carefully selected. C ontrol P oints Since the C ubic spline passes through all control points those points have to be selected precisely All control points have the same weight ; therefore an incorrectly selected control point or points can create an erroneous lane model. This problem occurs more at the far side of an image. In the real world environment, obtaining a clear lane edge filtered image is almost impossible. Nonlane edge pixels exist randomly by the nature of the world, so the far side of an image is easily wash ed out compared to the near side of an image. For this reason, a new control point selection method is proposed in this dissertation The f ollowing describes this method:

PAGE 62

62 Normally, a curve d line s start poin ts match the Hough transform line. Therefore a fter computing the Hough transform line, N distance pixels from the Hough line are selected for the curve d line candidates pixels. Figure 412 (A) shows the Canny filtered edge image and Figure 4 12 (B) shows the N pixel distance area from the Hough transform line. The r esult ing edge is shown in Figure 4 12 (C). The Figure 412 (C) image still has many outlier pixels and two lane side s The greatest size of connected edge pixels is extract ed as shown in Figure 412 (D). A t this step, only curvature is left to be determined, if a lane is a curve d line. Next, the normal vectors from the Hough transform line to the curvature pixels and the distance are computed. B ased on the image resolution and the real world distance, control points are selected In F igure 412 (F) the blue line shows the normal vectors from the Hough transform line to curvature pixels. Finally, the Cubic spline is computed using selected control points. Lane Model Because of a property of t he Hough transform many lines which include not just road lane lines but also other lines are detected but only lane lines need to be classified [ Hashimoto 1992]. Therefore a search method is applied to detect only the two lane lines among the many line candidates While l ine angle and distance parameters are used for this procedure, sometimes more than two lines meet these line angle and distance conditions. In those cases, the closest line is selected as the lane line. Those angle and distance parameters are also employed to verify the lane model assumption in which the road is modeled as a plane and the lane lines are parallel to each other in the global view. Also, this lane line model assumption can be applied not only as the vehicle drives a straight road but also as the vehicle drives along a curved road. Since the camera field of view is local and the update rate is around 20Hz the far area lane correction error caused by a first order line assumption can be ignored.

PAGE 63

63 A global coordinate view, also called a bird s eye view, is a good coordinate system for checking the lane model. Figure 414 (A) shows detected lines on a straight road and Figure 414 (B) shows detected lines on a curved road from a bird s eye view. Lane Estimation In the real world, road conditions may be such that for a moment, only one or no lane line s are visible. Even on roads in good repair with proper markings the problem of losing a lane reference may occur. This can happen when there is segmented line pain t ing, intersections, or lane merges. Also, it can happen there is a partial obstruction by an other vehicle or there is a strong shadow on the line on a bright day. For these instances, an estimation technique is employed to estimate the likely location of the missing lane boundary line This is accomplished by using a previous N number of line parameters that are slope and intersection in the first order line model : ymxc ( 410) Eq ( 410) defines a linear line with slope m and intersection c in a Cartesian coordinate system and (x,y) is the image pixel location T he l inear l east s quare s estimation technique is applied to estimate a first order lane line s angle and intersection parameters. Whenever a lane line is detected, N numbers of line angle and intersection parameters are stored in a buffer. Then when the vehicle pass es the segmented road line area or crossroad area, those stored parameters are employed for estimating the likely position of the line. Finally, estimated line parameter s quality is checked by the lane line model I f an estimated line meets a lane model, it is use d to compute lane correction data. However, even if the estimated line parameters are good enough to computes lane correction, the estimated parameter s use old data again and again without considering vehicle behavior Theref ore only N -

PAGE 64

64 number of the estimated data is processed, otherwise the confidence value is set to zero. Figure 415 depicts a line parameter estimation flowchart. This method can be applied without an accurate vehicle dynamic model [ Apolloni 2005] Let y be the observation vector and N the observation number. The observation vector can be written as [(1),...,()]tyyyN ( 411) The l east s quare estimate is the value of h that minimizes the square deviation: ()()(),TJhyXhyXh ( 412) where X is an N P matrix where P is the order of the polynomial model of the function. The solution can be written simply as 1().TThXXXy ( 413) Figure 416 depicts sample results of the lane estimation result Figure 416 (A) and (C) show two sequential source images from the left camera. Figures 416 (B) and (D) show the detected (blue) line and the estimated (orange) line, when a vehicle passes the segmented line area. From F igure 416, it is clear that the estimation process can effectivel y determine the location of the missing boundary and is useful when dealing with segmented lines Figure 417 shows line angle parameter estimation result s The X axis shows frame number of sequential image s and the Y axis show s the Hough transform line s angle parameter. When an autonomous vehicle passes the segmented line area, the lane show s and disappear s again and again. Detected line angle parameters displayed in blue points and estimated line angle parameter s are displayed in orange points.

PAGE 65

65 Lane C enter Correction Lane C orrection by T wo C ameras Information such as the estimated center of the current lane combined with lane width is used to determine the vehicle orientation within the lane. After converting data from the image coordinate system to the real world coordinate system, the distances between the detected / estimated lane lines and the vehicle side are computed ( F igure 4 18, purple arrows ). From these distance values, lane correction data (F igure 4 18 blue arrow ) and lane width ( F igure 418, red arrow) are easily compute d. Eq (4 14) shows the definition of the lane correction ,RLCorrectiondd ( 414) and Eq ( 416 ) shows how to compute lane width using two distances between the vehicle side and lane boundary, .laneLRWdd ( 415) where Wlane is lane width ande dL dR is distance between vehicle side and lane boundary. By the camera calibration, the relationship between the real world distance and the image pixel distance is measured. Figure 419 (A, B, C and D) shows two camerabased lane finder calibration images. A resolution at 4, 5, 6, 7, 8, 9, 10, 12 and 14 mete r s from the vehicle reference point are summarized in T able 4 2. T he resolution at the 5 meter position is around 0.66 centimeter s per pixel and at the 7 meter position is around 1 centimeter per pixel

PAGE 66

66 T able 42. Two side cameras horizontal pixel resolution on 640 x 380 image Y Distance X pixel location (Center) X pixel location (Left) Pixel distance Real distance (cm) Resolution (cm/pxl) on 640 x 380 Resolution (cm/pxl) on 320 x 190 5 470 150 320 212 0.6625 1.3250 6 452 192 260 212 0.815384615 1.6308 7 436 222 214 212 0.990654206 1.9813 8 426 240 186 212 1.139784946 2.2796 9 418 258 160 212 1.325 2.6500 10 412 268 144 212 1.472222222 2.9444 12 404 286 118 212 1.796610169 3.5932 14 398 296 102 212 2.078431373 4.1569 Since two wing cam eras can see both vehicle sides, and face the ground, and those cameras field of view start s from the vehicle s front axis the accuracy of the lane width and lane center in near view is better than a camera with a far view. Based on the perspective property of a camera or human eye c ertain distance points from the vehicle rear ax is are selected as a lane center estimation positions Figure 4 19 ( E ) and t able 4 3 shows that relationship between real world position and camera pixel position from the vehicle reference position. The image resolution from the 5 to 8 meter position is high but from 9 to 15 meters resolution is low. Therefore it is unnecessary to select a lane center estimation position every meter. Th e 5, 6, 7, 8, 9, 10 meter position s and the 12 and 14 meter position s from the vehicle reference points were selected. Each position s lane correction distance is computed and t hese values are collected by the LFSS Arbiter component through an experimental JAUS message along with other sensors lane correction data for estimating the future trajectory. Table 4 4 shows the lane center correction JAUS message data structure in C/C++. This message contains not just lane center correction data, but also lane properties, like lane color, type and width.

PAGE 67

67 T able 4 3. Side cameras pixel location by real distance on 640 380 resolution image Distance from Vehicle reference (meter s ) Y location from bottom (pixel s ) 5 125 6 184 7 220 8 245 9 261 10 276 11 288 12 298 13 305 14 313 15 321 T able 44. Lane center correction experimental JAUS message definition typedef struct LaneFinderCorrectionStruct { float rangeM ; // distance ahead of IMU float offsetM; // lane correction float offsetConfidence; JausUnsignedInteger offsetOrigin; // offset from center // or offset from curb //Road Width float roadWidthM; float roadWidthConfidence; //Lane Width float laneWidthM; float laneWidthConfidence; //Normally, these values are computed by vision sensor JausUnsignedInteger boundaryColor; float boundaryColorLeftConfidence; float boundaryColorRightConfidence; //Lane Type JausUnsignedInteger boundaryType; float boundaryTypeLeftConfidence; float boundaryTypeRightConfidence; struct LaneFinderCorrectionStruct *nextCorrection; } LaneFinderCorrectionStruct;

PAGE 68

68 Lane Correction by One C amera The c enter camera s field of view is larger than the two LFSSWing cameras field s of view. This is designed for longrange lane correction for farther future estimation. Points at 8, 10, 15, 20, 25, and 30 m eters from the vehicle reference are selected to compute longrange lane correction Table 4 5 summarizes horizontal pixel resolution by center camera and its resolution is lower than two camera bas ed lane corrections which are shown at Table 4 2. Table 4 5. The LFSS center camera s horizontal pixe l resolution on 640 x 218. Y d istance X pixel location ( c enter) X pixel location ( l eft) Pixel distance Real distance (cm) Resolution (cm/pxl) o n 640 x 218 Resolution (cm/pxl) o n 320 x 109 8 317 15 302 424 1.40397351 2.8079 10 316 72 244 424 1.737704918 3.4754 15 315 149 166 424 2.554216867 5.1084 20 315 191 124 424 3.419354839 6.8387 25 315 215 100 424 4.24 8.4800 30 315 231 84 424 5.047619048 10.0952 Except for resolution and field of view, the lane correction algorithm is the same as the LFSSWing algorithm. L ane correction values are computed by using Eq (414). These lane correction values are sent to the LFSS Arbiter component with LFSSWing component correction values Lane Property Every object color consists of two pieces of color information; real object color and lighting color. Because of this property, i t is not easy to obtain the exact object color in the outdoor environment in real time For a moving object like an autonomous vehicle lighting source direction changes over time so it depends on the surroundings and time of day. For this reason, color information is not selected as a primary feature in the lane tracking system.

PAGE 69

69 However people can acquire more information from not just line type but also line color For example, a vehicle can not cross the yellow line. Although i t is hard to indentify real object c olor in the real world. I t is not hard to categorize the line color even if a color source is acquired from an outdoor environment since normal p ainted l ane l ines are only yellow or white A histogram matching method using color pixel distance is proposed to identify lane color. First, lane line mask images are generated using the bottom part of the image B ecause this part includes the most vivid color high line pixel resolution and partial line s help it to reduc e computation power needs A part of the detected or estimated Hough line s are used to generat e the mask images. Figure 317 shows two camera field of view mask images. After generating lane line mask images, the color distance between the lane line color and each color of the lane color look up table is calculated using Eq (4 16). 222min()()(),d rrggbbCpLpLpL ( 416) w here Pr,g,b is the RGB value of lane pixels and Lr,g,b is the RGB value of the lookup table color. Table 4 6 shows lane color lookup table. Table 46. Lane color lookup table Color Red Green Blue Yellow 255 255 150 White 255 255 255 Black 0 0 0

PAGE 70

70 In this procedure, asphalt color distance is also calculated and then those pixels are ignored for classifying lane color Finally, color distance values are utilized for creating a histogram and deciding the lane color that has the minimum sum of color distances [ Krishnan 2007]. U ncertainty Management The h uman perception system consists of more than one sensing element for example visual, aural and tactile sens es. Those sens es merge together with past experience and using the brain to make judgments for proper action In a robotics perception system the same approach is demanded because t here is no sensor equipment for capturing all sensing information at one time. Therefore a sensor fusion process is required in a multiple sensor based robot ic system and each sensor s output data management has an important role in this process. When different type s of sensors are task ed with the same goal the system has to identif y each sensor s output quality. For example, two different field of view camera systems are utilized for the lane tracking system and a LADARbased lane tracking is also developed. Additionally, e ven though the vision based lane tracking system give s reliable output in most cases, there will be occasions when there is the risk of poor or even erroneous output from the system because of the environment machine failure and so on, and these cases need to be identified. I n addition to the lane tracking output s, c onfidence value s are provided for uncertainty management A root mean square deviation (RMSD) value is used to determin e the confidence value of the lane tracking system output, such as lane center corrections, lane width and lane color. The RMSD measures the difference between actual measurement values and predicted values. In this system, two th ings are assum ed; first, the previous data is the measurement value

PAGE 71

71 S econd, the current measurement or estimate data is the predicted value. From this assumption, the RMSD is calculated as the confidence value using Eq ( 417 ): 2 RMSD()MSE()E(()). ( 417) A B Fig ure 41. Camera field of view diagram A) The center camera s view B) t he two side cameras view

PAGE 72

72 A B C D Figure 42. C amera field of view A) T wo camera view in an open area B) two camera view in a traffic area C) center camera view at the Gainesville Raceway and D) center camera view when other vehicle blocks a lane

PAGE 73

73 A B C D Figure 43. Canny filtered image samples. A ) Original road Image B ) r ed c hannel i mage C) C anny filter image with 50/200 threshold value and D) C anny filter image with 130/200 threshold value

PAGE 74

74 Figure 4 4. Two camera Canny filtered image in various situations. A) Solid and segmented line, B) segmented line, C) stop line, D) partial block of line, E) curved line, F) noise on the road, G) wipe out line, and H) old tire track on the road.

PAGE 75

75 A B C D E

PAGE 76

76 F G H Figure 44. Continued.

PAGE 77

77 Figure 45. Hough space parameters in image coordinate system

PAGE 78

78 A B Figure 46. Hough space A) The Canny filtered image at image space, B) A s Hough space Hough Space -80 -60 -40 -20 0 20 40 60 80 -600 -400 -200 0 200 400 600

PAGE 79

79 Figure 47. H ough line transform results. A ) Straight road, B ) stop line, C) other lines on the middle of the road, D) other lines by noise edge pixel, E) other lines by illumination difference on the road, case I, and F) other lines by illumination difference on the road, case II.

PAGE 80

80 A B

PAGE 81

81 C D Figure 47. Continued.

PAGE 82

82 E F Figure 47. Continued.

PAGE 83

83 Figure 48. Lane line checking parameters angle ( ) and distance (dL dR).

PAGE 84

84 A B Figure 49. Center camera r esults of lane finding with estimated center line A) S traigh t road B) curve d road

PAGE 85

85 Figure 410. Diagram of Cubic spline and control points.

PAGE 86

86 A B C Figure 411. T w o camera overlay image of lane line s in curve d road. A) S traight road B) crossroad with stop line and C) curve d road.

PAGE 87

87 A B C D E F G Figure 412. Hough line + curve control points for s pline model

PAGE 88

88 A B C Figure 413. C urve d lane A) S ource image B) straight line by the Hough transform and C) curve d line by C ubic spline

PAGE 89

89 A B Figure 414. L ane model checking view A) S traight line B) curve d line

PAGE 90

90 Figure 415. L i ne parameter estimation f lowchart

PAGE 91

91 A B C D Figure 416. Two sequence source image and detected (blue) and estimated (orange) line

PAGE 92

92 Figure 417. L east squares angle parameter estimation result

PAGE 93

93 A B Fig ure 418. Lane correction distance definition A) When a vehicle drives on the left side of the lane B) w hen a vehicle drives on the right side of the lane

PAGE 94

94 A B C D E Figure 419. Real world and image distance relationship. A) LFSSWing calibration images, B) Relation between real world position and image pixel on Y axis. 5 6 7 8 9 10 11 12 13 14 15 100 150 200 250 300 350 Distance from vehicle reference position(meter)Y-pixel location on image(bottom is zero) Relation between real world and image

PAGE 95

95 Fig ure 420. The LFSS and PFSS calibration image

PAGE 96

96 A B C F igure 420. Mask images for lane line color. A) S ource image B) Hough transform based line image, and C ) mask image

PAGE 97

97 CHAPTER 5 V ECTORBASED GR OUND AND OBJECT REPRESENTATION Introduction O ne of the biggest issue s in computer vision systems is the large sizes of the source and result data These propert ies can result in out sized computation processing, bandwidth requirement problem s communication delay s and storage issue s for an autonomous robot vehicle especially systems consist ing of multiple components A recently developed technique using computer hardware like g eneral purpose computing on graphic processing units (GPGPU) helps to reduce processing time remarkably, but problems still remain. For example, if the robot system uses a raster based world r epresentation and sensor data fusion, it requires great storage space, long computation time and a large communication bandwidth I t will support only the specific resolution that is initially defined. However, a vector based world representation permits stor ing, searching and analyz ing certain types of objects using a small storage space. It can also represent multiple resolution world model s that help to improve processing, analyz ing and displaying the data. In addition, a vector based system can store much property information in addition to object information. With th ese vector based sensor data representation advantage s one can store previous traveling information in a database system similar to human memor y. Therefore one can use previous travel data to verify current travel safety. Approach W ith respect to computation and data storage efficiency, t he vector based representation of a ground plane is better suited for real time robot system component s A vector based representation can generate both 2 D and 3D object model s with the help of a 3D sensor like GPS and LADAR. Spe ci fically a raster based world model s memory size is highly dependent

PAGE 98

98 on grid map coverage and resolution If a world model covers a wide area and is utilize d with high resolution, it n eeds large computation power and a very large memory space and it causes bandwidth problem between components. Another issue with a raster based grid map is that it contains many areas of unknown data because the map s coverage area is fixed F igure 51 (C, D) shows two different size grid map s from F igure 51 (A, B) the source and classified image respectively Figure 51(C) is a 241 241 size, 0.25 meter resolution grid map with a data size 56k. A nd figure 5 1 (D) is a 121 121 size, 0.5 meter resolution grid map, with a data size 14k. Therefore raster based grid map data size depends mostly on the resolution and coverage area. Table 5 1 summarizes data size of a various format traversability map Table 5 1. Raster based traversabil ity grid map data size C overage (meter) G rid map size (pixel) G rid map resolution (meter) D ata size 60 60 121 121 0.5 14K byte 60 60 241 241 0.25 56 K byte 60 60 601 601 0.1 352 K byte 300 300 601 601 0.5 352 K byte 300 300 1201 1201 0.25 1408 k byte 300 300 3001 3001 0.1 8794 K byte Unlike this raster based world re presentation, a vector based method has almost no limitation s to cover a ge and resolution A vector based coverage only depends on sensor coverage and sensor resolution. It also needs dramatically less memory for storing sending, and receiving data and easily buil ds a multi resolution world environment. For example, the PFSS s vector output needs around 4 to 20 points to represent traversable road area in an urban environment. T able 5 2 summari z es a vector based traversability data size based on the number

PAGE 99

99 of vector points Compared the raster based ground representation to t able 5 1, the vector based representation can reduce data size at least 60 times on a 60 60 grid map. Table 5 2. Vector based t raversability data size Points 2 D map 3 D map 4 2 4 x 4 = 32 byte 3 4 x 4 = 38 byte 12 2 12 x 4 = 96 byte 3 12 x 4 = 144 byte 20 2 20 x 4 = 160 byte 3 20 x 4 = 240 byte Ground S urface R epresentation In the PFSS, the T riangulated I rregular N etwork (TIN) [ Witzgall 2004] is selected to represent a world surface Fewer point s are needed to represent the same sized ground surface than with raster based representation s and i t is a digital data structure used for the representation of a ground surface in geographical i nformation s ystem s (GIS ) Figure 5 2 shows a sample TIN using LAD A R data From a road and nonroad segmented image as in F igure 310 (B, C, D) a high resolution bird s eye view image is generated as shown in F igure 53 (A) which is applied with a 10 centimeter per pixel resolution Unlike the grid map image, the sight of the bird s eye view image covers only the front area of a vehicle to reduce useless information. Because of this property, a bird s eye view image can be generated with 10 centimeter per pixel resolution. Next, the road boundary and candidate control points are extracted. Figure 53 (B ) and ( C) show boundary image and candidate control points image respectively. Finally, fewer candidates points are selected for consideration of vector resolution storage efficiency and accuracy For example, if a vehicle drives in a straight line the road looks like a rectangle T herefore fewer points are selected for represent ing the road. However, if a vehicle drives over a curve d

PAGE 100

100 road, the road shape looks like a curvilinear polygon. Consequently, many more points are required to represent a curve d road. These selected points are new candidate points for a TIN ground representation. Figure 54 shows irregular vector point s in various situation s for instance irregular vector points for a straight road, curve d road and T intersection case, respectively. Fig ure 55 ( A B, C, D, and E ) summarizes how to extract a boundary of traversable area and select candidate points for TIN representation From a bird s eye view image, noise pixels are eliminated as shown in Figure 55 (B). Extracting the road boundary is shown in Figure 55 (C). Selected TIN control points ar e shown in Figure 55 (D) A TIN map is shown in Figure 55 (E) F igure 55 ( F ) shows a 3D vector based ground representation with zero height using a TIN algorithm. After selecting the TIN control points of a traversable ground boundary in figure 5 3 (C) those points are used to build a road model and are store d. Since center camera s vertical field of view start s from the 8 meter for the reference position, r anges of stored points are 10 meter s to 20 meter s from the vehicle reference position. The figure 5 6 illustrates diagram of road boundary polygon area and stored points Static O bject R epresentation There are two main goals for the LFSS and the LFSSWing component software. First is a future trajectory e stimation and lane departure correction. Second is building and generating a lane model and lane properties by using GPS data. To solve these two problems t he LFSS and the LFSSWing detect static road objects that are road lane lines and direct ly compute the mathematical lane center as an output However the LFSS and the LFSSWing cannot detect both

PAGE 101

101 lane lines when a vehicle travels in an area with segmented lines. Therefore, in this case, an estimated line using the parameter estimator is stored using pre vious ly detected line information. After detecting and computing the lane center and its properties, vector points of the road lane are selected to build and stor e the lane model. T wo points from the vehicle reference points are selected at each side lane line For example 5 meter and 10 meter lane line points from the vehicle reference, which is located in inertial measurement unit ( IMU ) are selected since the point area image resolution is high enough. Figure 57 shows a diagram of lane polygon area. These four points create a polygon for notifying and storing traversable area to the World Model Vector Knowledge Store From t hese stored lane polygon data, vector based lane objects can be illustrated. World M odel Vector Knowledge Store The g enerated ground surface and lane object data will be stored in a database system that is called the World Model Vector Knowledge Store (WMVKS). T he WMVKS can store and retrieve ground surface characteristics, static and moving object and moving objects images from various type sensor s For the visionbased components, t he PFSS send s ground surface polygon points and the LFSSWing sends lane object polygon points to the WMVKS The WMVKS enable s an autonomous vehicle to drive the same place it has already traveled using archived ground surface and ground surface static object information T able s 53 and 54 show the LFSS lane object DB table and the PFSS ground surfa ce DB table.

PAGE 102

102 Table 53. The LFSS Lane object DB table. Lane Lane c onfidence Lane p roperty Lane w idth Width c onfidence Date/Time Run time Component ID Double I nt eger I nt eger double I nt eger YYYY/MM/DD HH/MM/SS I nt eger I nt eger {P1,P2,P3,P4} P1= {Latitude, Longitude, Altitude} 0 1 1 White 2 Yellow 10 Solid 20 Segmented Meter 1 high confidence 0 low confidence 1,2, 21 LFSSWing 22 LFSS Ex)P1= {29.756850064, 82.267883420, 0} Ex) 1 high confidence 0 low confidence Ex) 11 white solid line Ex) 4.3 Ex) 0.9 2009/10/19 /13/23/01 Ex) 21 Table 5 4. The P FSS ground surface DB table. Surface ID Polygon Surface p roperty Date/Time Run time Component ID I nt eger LLA points (double, double, double) I nt eger YYYY/MM/D D HH/MM/SS I nt eger I nt eger 1,2,3 {Pt1, pt2, ptN, Pt1} 1 Asphalt 2 Unstructured road 3 Grass 4 Unknown 1,2, 21 Ladar TSS 22 Vision TSS Ex) 1 Ex) {(29.75692231169602, 82.2675184406604, 0), (29.75691959879387, 82.26765801719262, 0) (29.75692231169602, 82.2675184406604, 0)} Ex) 1 2009/10/19 /13/23/01 Ex) 22

PAGE 103

103 A B C D Figure 51. Various grid map sizes. A) Source image at University of Florida campus B) classified image, C) 241 241with 0.25 meter resolution grid map image and D) 121 121 with 0.5 meter resolution grid map image.

PAGE 104

104 Fig ure 52. T he triangulation map1 1 This TIN image is a LADAR based ground surface image by Jihyun Yoon.

PAGE 105

105 A c B C Figure 53. Vector representation s. A) 10 centimeter resolution bird s eye view image B) boundary Image, and C) boundary image with control points

PAGE 106

106 A B C

PAGE 107

107 D F Figure 54. Irregular v ector points ( A B ) straight road (C ) curve d road and ( D E ) T intersection road

PAGE 108

108 A B C D E F Fig ure 55. The TIN representation of traversability map A) Bird s eye view image B) noise illumination C) road boundary extraction, D) control point selection, E ) 2D TIN map and F) 3D TIN map.

PAGE 109

109 Fig ure 56. Road boundary polygon and stored polygon points (red points)

PAGE 110

110 Fig ure 57. Lane objects and stored polygon points (red points)

PAGE 111

111 CHAPTER 6 EXPERIMENTAL RESULT S AND CONCLUSIONS Platform The target implementation for th is research is the University of Florida DARPA Urban Challenge 2007 team vehicle, called t he Urban NaviGator It is a modified 2006 Toyota Highlander Hybrid SUV equipped with numerous laser measurement systems, cameras, a differential GPS, and an inertial me asurement unit (IMU). A total of 8 Linux computers and 3 Windows computers are located in the vehicle to process, compute and control the vehicle. Figure 61 shows the Urban NaviGator vehicle and sensor mount Hardware The Urban NaviGator has one BlueFox highspeed USB 2.0 camera [Matrix Vision 2009] for the PFSS/LFSS and two BlueFox highspeed USB 2.0 cameras for the LFSSWing. The PFSS and LFSS share the source image s from the camera mounted in the center and the LFSSWing uses the two side cam eras. T he wing mounted cameras and the center camera use different focal length lenses and have different field s of view ; therefore they generate lane correction information at different distances and will increase the certainty of the overall prediction of the future trajectory. T he BlueFox USB 2.0 camera provides up to 100 Hz color frame grabbing rates with 640 480 resolution Because the vision components area s of interest are smaller than the camera s field of view the source image resolutions are reduced to 640 216 pixels for the LFSS, 320 108 pixels for the PFSS and 320 240 pixels for both cameras used by the LFSSWing. The PFSS/LFSS camera was mount ed in the middle of the sensor bridge and faces the ground, and the LFSSWing cameras were mount ed on either side of the sensor bridge Figure 62 ( A ) shows

PAGE 112

112 the PFSS/LFSS and the LFSSWing camera s mount ing location s The c enter camera uses the TAMRON var i focal lens with 412 mm focal length and the two si de cameras use 4 mm fixed wide focal lenses. Table 6 1 summarizes the camera and lens specification s and F igure 41 shows the PFSS/LFSS and LFSS Wing cameras angle s of view which depends on lens specification and the orientation. Each vision component uses an AMD dual core computer with 1 G B of RAM At the DARPA Urban Challenge 2007 competition, the PFSS could run at 1518 Hz update rates and the LFSSWing c ould operates with 1 017 H z update rate s with this hardware. Table 6 1. Camera and lens specification. PFSS LFSS LFSSWing Location C enter C enter E ach side of sensor bridge S ource image size 320 108 640 216 320 190 CCD size 1/3 1/3 1/3 Lens TAMRON var i focal lens TAMRON var i focal lens COMPUTAR fixed focal lens Focal lens 4 12 mm 4 12 mm 4mm Horizontal angle of view 31.2 93.7 31.2 93.7 63.9 Vertical angle of view 23.4 68.9 23.4 68.9 49.1 Software The PFSS, the LFSSWing and the LFSS software programs were written in C++ for the Windows environment. Additional functions, algorithm s, and GUI are constructed using the Matrix Vision API library and the OpenCV library Also, t he Posix thread library was utilized to quickly capture the source image s from camera s Both components s upport the J oint

PAGE 113

113 A rchitecture for U nmanned S ystems (JAUS) functionality [JAUS 2009]. JAUS is a communication protocol that serves to provide a high level of interoperability between various hardware and software components for unmanned systems. T he PFSS, the LFSSWing and the LFSS outputs are processed to JAUS messages and sent to the other customer components, for example the LFSS Arbiter [Osteen 2008] Each component provide s various intermediate pro cessing results that can be helpful in running the software parameters or for troubleshooting For example, the PFSS component program can display two of the following images: the source image, canny filtered image, source image without lane lines noise filtered image or training area image. For the PFSS output, a raster based grid map, raster based grid map without yaw adjustment road boundary point s or road polygon image can be selected by the user Figure 63 shows various s creenshot s of the PFSS software. For t he LFSSWing and the LFSS, t he source image, edge filtered image, Hough line image or detected lane line overlay image can be selected. A n information window displays each distance lane center correction s lane color and lane width values along with their associated confidence value s Figure 64 shows screenshots of the LFSSWing implementation ( the LFSS software is similar) T he LFSS Arbiter component fuses local roadway data from the TSS, the L FSS, the LFSSWing and the Curb Finder smart sensor component [Osteen 2008] The data consist of offsets from the centerline of the vehicle to the center of the road lane estimated at varying distances ahead of the vehicle. Finally, the LFSS Arbiter generates a curve fit from the different sensor data Th ese data are used to adjust for GPS measurement errors th at are supplied to the Roadway Navigation ( RN ) [ Galluzzo 2006] component which navigate s the vehicle within a

PAGE 114

114 lane using an A* search algorithm Figure 65 (A B ) shows the Gainesville Raceway test area from the LFSSWing cameras and from a vantage point off of the vehi cle respectively. Figure 65 (C) shows the LFSS Arbiter s curve fit screen when the vehicle drive s along a curve d road. Figure 65 (D) shows the Roadway Navigation component s screenshot of its path searching. Brown represents the A* search candidate branch es and white points are the intermediate goal point s provided from LFSS Arbiter. Figure 66 shows the Urban NaviGator 2009 system architecture. T he PFSS, the LFSSWing, and the World Model Vector Knowledge Store are highlighted. The PFSS stores a vectorized representation of the road in the WMVKS. T he LFSSWing also stores its output as a vector area that describe lane Results The following section describes the test results pertaining to this research for both t he LFSSWing and the PFSS components Since t he LFSS long range component s output is very similar to the LFSSWing component output, the LFSS output is not described in this section. Based on chapter 2 assumption s test areas are flat, and the camera source images are clear enough to see the environment. The a utoexpos ure control option, which is provided by the Matrix vision s camera setting software, helps to capture a clear source image from various illumination conditions. Tes ts are divided into four categor ies : The Gainesville R aceway The University of Florida c ampus NW 13th street, and NE 53rd avenue, Gainesville, F lorida and Night time setting

PAGE 115

115 The Gainesville Raceway is the only the place to perform an autonomous drive due to a safety reason s Since the Gainesville Raceway is a race track it provides a wide open area and it does not have standard necessary facilities, such as curbs pedestrian crossing marks, and so on. The U niversity of Florida campus provides various urban environment facilities such as bike lane s curb s pedestrian crossings sidewalk s more than two lanes, merg ing or divid ing lane s and shadow s from trees. NW 13th street and NE 53rd avenue are selected to test the software with traffic and/or at high speeds The PFSS component was tested only at the Gainesville Raceway and the University of Florida campus. F inally, the LFSSWing components were tested at night time with illumination p rovided from the head light s For the autonomous driving test s vehicle speed was approximately 10 mph. F or the real world tests vehicle travel s peeds were from 20 mph to 60 mph (driven manually) Vectorbased map s were built while traveling approximately 1020 mph. LFSSWing test result s T he Gainesville Raceway is pictured in the F igure 6 7 (A) The outer loop is approximately 1 km and the smaller half loop is 650 meters This course sequence includes a straight lane, a curve d lane, segmented painted lane line a narrow lane width area T intersection areas and cross road areas Since this location is an open area, cameras can receive different direction s light in a short time. Figure 67 ( B ) shows a straight lane at the starting point. Figure 67 ( C ) shows a curve d lane with the top part of the right camera source image washed out due to the light direction F igure 6 7 ( D ) shows that a short length of segmented line can be detected as a lane line. W hen the Hough candidate lines are extracted from an edge image ( F igure 44) the line length threshold is decided by the source image height In this research, 20% of source image height, 48 pixel s is selected as the Hough line minimum length parameter. F igure 67 ( E )

PAGE 116

116 shows a narrow lane compared to a wide lane area in the Gainesville Raceway. The l ane width threshold is defined by a roadway designplan s preparation manual [ Florida Department of Transportation 2009] 12 feet (3.65 meter) is a standard rural lane width In this research, a 2 f oot margin is applied, therefore a 10 f oot to 14 f oot (3.048 meter to 4.267 meter) lane gap is considered to be a proper ly detected lane width. The lane width in F igure 6 7 ( E ) is approximatel y 3.353.5 meter s and it is narrower than a standard lane width F igure 6 7 ( F ) shows a T intersection area. In this situation, the right lane line is detected and the left lane line is estimated using previous 20frames line parameters. F igure 67 ( G ) shows a vehicle travel ing on the right side line in an autonomous test run. Since vehicle controllers response is not always fast enough, it is a possi bility that the vehicle ventures on to the line momentarily However since lane correction values are be ing updated continuously, the vehicle can drive back to the middle of a lane The s econd test place was the University of Florida campus. This location has many artificial structures for example bike lane s curb s and pedestrian crosswalk and so on. The t est course is an approximately 4 km loop. F igure 68 (A) shows a satellite photo of the University of Florida campus. In F igure 6 8 (B), the LFSSWing operates with shadows in the image F igure 6 8 (C) shows a vehicle travel ing through a pedestrian crossing area, and figure 68 (D) shows the LFSSWing detect ing a curb The t hird test place was an urban area with real traffic In this environment, sample results include high speed conditions divided lane situations, and road s with more than two lane s Figure 69 shows some urban road test results. F igure 69 (A) shows multiple center lane lines, figure 69 (B) shows a divided lane area, and figure 6 9 (C) shows the LFSSWing output with real traffic on a four lane road.

PAGE 117

117 The fourth test pl ace was the University of Florida campus at night with a nighttime camera setting. Since illumination is too weak at nighttime the c amera exposure time was increased and the rest of the LFSSWing setting was the same as daytime Not just lack of illumination, but also other artificial lighting sources by other traveling car or streetlight are the big difference of this test. Figure 6 10 shows various situations outputs in a night time test. The PFSS test result The PFSS was tested at the Gainesville Raceway and the University of Florida campus. Since the PFSS is designed to characterize ground surface a rea, an urban environment is not suitable to get meaningful output. For example, if another vehicle travels in the camera s field of view the PFSS possibly considers a vehicle as a non traversable area. Therefore, the PFSS is designed and tested in a n open area only. F igure 611 shows the PFSS test results at the Gainesville Raceway (see satellite photo in the figure 6 7 ( A)) In each case, the source image segmented image and the TIN control points image are displayed F igure 6 11(A) shows a straight road and F igure 611 (B) shows a T intersection area on the right hand side. In the TIN control points image, the right intersection is identified F igure 6 11 (C) shows a curve d road with 10 points being used to describe it. Normally, a curved road area need s more point s to represent a ground surface than a straight road area When a vehicle turns at a T intersection a partial part of road is visible at the camera. F igure 611 (D) shows such a situation. F igure 612 shows the University of Florida campus test Straight road and curve d road cases are shown in F igure 612 (A ) and (B), respectively. In F igure 612 (C), part of the road is occluded by a bus traveling in the other lane. Therefore representation of the ground surface is incorrect

PAGE 118

118 Building vector based map Both the LFSSWing and the PFSS component s vector based lane object and ground surface map s are re constructed In the section, the Gainesville Raceway and the University of Florida campus are selected as a test environment F igure 67 (A) and F igure 6 8 (A) show each area s satellite image. To generate a lane object vector representation, the lane object is detected, converted from the local coordinate system to the global coordinate system, and then stored into the WMVKS. C hapter 4 describes the lane finder algorithm F igure 613 (A B ) shows the vector representation of a lane at the Gainesvi lle Raceway and at the University of Florida campus, respectively. For a ground surface vector representation, the ground surface is classifi ed and road boundary vector points extracted and convert ed from the local coordinate system to the global coordinate system before being stored into the WMVKS. Chapter 3 describes the path finder algorithm Figure 614 (A B ) shows the vector representation of the ground surface at the Gainesville Raceway and at the University of Florida campus, respectively. Con clusions The vector based ground surface and lane objects representation algorithms are development and implementation to extract and simplif y the traversable area by using a camera sensor. Unlike in simulation, algorithms and methods are engineered for outdoor real time application s with continuous and robust output. This approach allows a robot to have a humanlike cognitive system. People feel comfort able when they drive a known area because the human brain is able to store important features by experience This vision system s vector output is small enough to be stored and retrieve d like the human brain. Therefore, t he vector output can be utilized to rebuild road map s

PAGE 119

119 Also, properties of the road are calculated to better understand the world such as lane with and lane color. This information assists the vehicle s intelligen ce element in mak ing proper decisions All vector data can be stored in a database system in real time Confidence values of output data are also computed. These values play a key role when data are judged and fused with data from different type s of sensor output, such as fro m LADAR. It can be fused with visionbased sensor output since all confidence values are normalized The a uthor present s results from various test places, time, and conditions. The a utonomous run test verifies that this camerabased lane finder and path finder approach creates robust and accurate lane correction s road map, and lane map building. With a simple camera calibration, th is software can be easily deployed to any JAUS system. Future Work There are three main areas which could be improve d in this research. Fir st, a 3D model can be generated using GPS height information and pitch information. Currently, a 2D model is generated based on a flat road assumption. H owever if this system is used on a slope, hill, or mountain area, a 3D road or lane model would provide more a ccurate information. Second, the lane line estimator should consider vehicle dynamics. The c urrent estimator uses previously detected or estimated line parameter s to estimate future line parameters without considering the vehicle s movement I f vehicle s yaw information is added to the estimator in addition to the currently used parameters, the system could generate better estimat ions, especially when the vehicle travels through intersection or cross road area s Third a real time vector output verification procedure is su ggested. This system can store and build lane and road model s Therefore, if the vehicle re explore s the same area, the system

PAGE 120

120 can verify that its current position is within the lane or road by comparing current position with archived lane or road area information

PAGE 121

121 Fig ure 61. CIMAR Navigator III, Urban NaviGator. A) NaviGator III, B) The front view of NaviGator sensor location, and C) The rear view of NaviGator sensor location.

PAGE 122

122 A B

PAGE 123

123 C Figure 61. Continued.

PAGE 124

124 A B Fig ure 62. NaviGator III camera sensor systems. A) Cameras location. Center camera is shown in red circle and LFSSWing cameras are shown in blue circles, B) C omputer system in truck.

PAGE 125

125 Figure 63. The PFSS software. (A) JAUS service connection windows, (B) source image, (C) edge filtered image, (D) lane mask image, (E) source image over lane mask, (F) training area image, (G) classified image, (H) high resolution bird s eye view image, (I) boundary candidates point, (J) TIN control points image, (K) 0.25 meter resolution grid map image without heading, (L) 0.25 meter resolution gridmap image with heading, and (M) information GUI windows

PAGE 126

126 A B C D E F G

PAGE 127

127 H I J K L Figure 63. Continued.

PAGE 128

128 M Figure 63. Continued.

PAGE 129

129 Figure 64. The LFSSWing software A) JAUS service connection windows, B) source image, C) edge filtered image, D) Hough line image, E) detected lane line overlay image, and F) the LFSSWing information window

PAGE 130

130 A B C

PAGE 131

131 D E F Figure 64. Continued.

PAGE 132

132 A B C D Figure 65. The LFSS Arbiter and the RN screen A) Curved test area, B) different point of view of curved test area, C) the LFSS Arbiter curve fit2 and D) the RN s A* search result. B rown branches are A* search candidate branch and white points are intermediate travel point by the LFSS Arbiter. 2 This image is generated by Phil Osteen [Osteen 2008].

PAGE 133

133 Fig ure 66. Urban Navi Gator 2009 system architecture.

PAGE 134

134 Figure 67. The LFSSWing test result at the Gainesville Raceway. A) Straight lane, B) curved lane, C) segmented lane, D) narrow lane, E) T intersection lane, and F) when a vehicle drives on the lane line in autonomous run.

PAGE 135

135 A B C D

PAGE 136

136 E F G Figure 67. Continued.

PAGE 137

137 Figure 68. The LFSSWing test results at the University of Florida campus. A) Satellite photo, B) lane with bike lane in the shadow, C) pedestrian crossing area, and D) curb area.

PAGE 138

138 A B C

PAGE 139

139 D Figure 68. Continued.

PAGE 140

140 A B C Figure 69. The LFSSWing test result s in the urban road. A) Multiple center lines, B) divided lane, C) real traffic situation in four lane s road.

PAGE 141

141 A B C D E Figure 610. The LFSSWing test results at the University of Florida with nighttime setting A) S traight road, B) other vehicle passed at the other lane C) other vehicle travel in front of a NiviGator III, D ) under a streetlight and E) estimator

PAGE 142

142 Figure 611. The PFSS test results in the Gainesville Raceway. Source, segmented and the TIN control points images, respectively. A) Straight road, B) T intersection on right side C) curved road, and D) T intersection in front.

PAGE 143

143 A B C

PAGE 144

144 D Figure 611. Continued.

PAGE 145

145 A B C Figure 612. The PFSS test results in the University of Florida campus. Source, segmented and the TIN control points images, respectively. A) Straight road, B) curve d road, and C) block by other vehicle.

PAGE 146

146 A B Fig ure 613. The LFSSWing vector based representation. A ) T he Gainesville Raceway ( compare to figure 67(A) is satellite image ) B) the University of Florida campus ( compare to figure 68 (A) is satellite image )

PAGE 147

147 A B Fig ure 614. The PFSS vector based representation. A) The Gainesville Raceway (compare to figure 67(A) is satellite image), B) the University of Florida campus (compare to figure 68 (A) is satellite image).

PAGE 148

148 LIST OF REFERENCES B. Apolloni, A. Ghosh, F. Alpaslan, L. C. Jain, and S. Patnaik, Machine Learning and Robot Perception (Studies in Computational Intel ligence). Berline Heidelberg: Springer, 2005. R. Behringer, S. Sundareswaran, B. Gregory, R. Elsley, B. Addison, W. Guthmiller, R. Daily, and D. Bevly, "The DARPA grand challenge development of an autonomous vehicle," in Intelligent Vehicles Symposium, 2004 IEEE, 2004, pp. 226231. U. Bergquist, "Colour Vision and Hue for Autonomous Vehicle Guidance," Sweden: Linktping University, 1999. M. Bertozzi and A. Broggi, "GOLD: a parallel realtime stereo vision system for genericobstacle and lane detection," Im age Processing, IEEE Transactions on, vol. 7, pp. 6281, 1998. M. Bertozzi and A. Broggi, "Vision based vehicle guidance," Computer, vol. 30, pp. 4955, 1997. S. Beucher and M. Bilodeau, "Road segmentation and obstacle detection by a fast watershed transfo rmation," in Intelligent Vehicles '94 Symposium, Proceedings of the, 1994, pp. 296301. T. Brunl, Embedded Robotics: Mobile Robot Design and Applications with Embedded Systems 3rd ed. Berlin Heidelberg: Springer 2008. A. Broggi, "Robust real time lane and road detection in critical shadow conditions," in Computer Vision, 1995. Proceedings., International Symposium on, 1995, pp. 353358. A. Broggi and S. Berte, "Visionbased road detection in automotive systems: A real time expectation driven approach," Arxiv preprint cs.AI/9512102, 1995. R. C. Chandler, "A utonomous agent navigation based on textural analysis Ph.D. dissertation, U niversity of Florida, Gainesville, FL, U.S.A 2003. Charles, Martial H. Hebert, Takeo Kanade, 1998, Vision and Navigation for the Carnegie Mellon Navlab, IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 10, No. 3, May 1988 J. D. Crisman and C. E. Thorpe, "SCARF: a color vision system that tracks roads and intersections," Robotics and Automa tion, IEEE Transactions on, vol. 9, pp. 4958, 1993. E. R. Davies, Machine Vision : Theory, Algorithms, Practicalities (Signal Processing and its Applications). San Franscisco, CA: Morgan Kaufmann, 2004. I. L. Davis, A. Kelly, A. Stentz, and L. Matthies, "Terrain typing for real robots," in Intelligent Vehicles '95 Symposium., Proceedings of the, 1995, pp. 400405.

PAGE 149

149 R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, 2nd ed. New York: WillyInterscience, 2000. R. O. Duda and P. E. Hart, "Use of the Hough transformation to detect lines and curves in pictures," Commun. ACM, vol. 15, pp. 1115, 1972. F lorida D epartment of Transportation, Roadway design, Plans Preparation Manual Jan 2009 update, Volume 1 Design Criteria and Process January, 2 009, http://www.dot.state.fl.us/rddesign/PPMManual/2009/Volume1/2009Vol1.shtm,2009 D. W. Gage, "A brief history of unmanned ground vehicle (UGV) development efforts," Unmanned Systems, vol. 13, pp. 932, 1995. R. C. Gonzalez, R. E. Woods, and S. L. Eddins, Digital Image Processing Using MATLAB. New Jersey: Prentice Hall, 2004. T. Galluzzo, "Simultaneous Planning and Control for Autonomous Ground Vehicles." Ph.D. dissertation, University of Florida, Gainesville, FL., U.S.A, 2006. R. Hartley and A. Zisserman Multiple View Geometry in Computer Vision. Cambridge, UK: CAMBRIDGE UNIVERSITY PRESS, 2004. K. Hashimoto, S. Nakayama, T. Saito, S. Ishida, K. Unuora, J. Ishii, N. Oono, and Y. Okada, "An image processing architecture and a motion control method for an a utonomous vehicle," in Intelligent Vehicles '92 Symposium., Proceedings of the, 1992, pp. 213218. M. Isard and A. Blake, "CONDENSATIONConditional Density Propagation for Visual Tracking," International Journal of Computer Vision, vol. 29, pp. 528, 1998. J AUS W orking Group, "Joint Architecture for Unmanned Systems(JAUS)", http://www.jauswg.org/,2009 J. C. McCall and M. M. Trivedi, "Videobased lane estimation and tracking for driver assistance: survey, system, and evaluation," IEEE Transactions on Intel ligent Transportation Systems, vol. 7, pp. 2037, 2006. A. Kelly and A. Stentz, "An approach to rough terrain autonomous mobility," in International Conference on Mobile Planetary Robots, 1998, pp. 129198. Z. Kim, "Realtime lane tracking of curved local road," in Intelligent Transportation Systems Conference, 2006. ITSC '06. IEEE, 2006, pp. 11491155. Z. Kim, "Robust Lane Detection and Tracking in Challenging Scenarios," Intelligent Transportation Systems, IEEE Transactions on, vol. 9, pp. 1626, 2008. K. Kluge, "Extracting road curvature and orientation from image edge points without perceptual grouping into features," in Intelligent Vehicles' 94 Symposium, Proceedings of the 1994, pp. 109114.

PAGE 150

150 K. Kluge and S. Lakshmanan, "A deformable template approach to lane detection," in Intelligent Vehicles '95 Symposium., Proceedings of the 1995, pp. 5459. C. Kreucher and S. Lakshmanan, "LANA: a lane extraction algorithm that uses frequency domain features," Robotics and Automati on, IEEE Transactions on, vol. 15, pp. 343350, 1999. Krishnan, N., Banu, M. S. and Christiyana, C. C. Content Based Image Retrieval Using Dominant Color Identification Based on Foreground Objects City, 2007. J. Lee and C. D. Crane, "Road Following in a n Unstructured Desert Environment Based on the EM(Expectation Maximization) Algorithm," in SICE ICASE, 2006. International Joint Conference, 2006, pp. 29692974. J. W. Lee, "A machine vision system for lane departure detection," Comput. Vis. Image Underst., vol. 86, pp. 5278, 2002. Lukac, R. and K.N. Plataniotis, "Color Image Processing: Methods and Applications". 2006, Boca Raton, FL: CRC Press. M atrix Vision, The mvBlueFOX USB 2.0 camera" http://www.matrix vision.com/products/hardware/mvbluefox.php?lang=en,2009 L. Matthies, M. Maimone, A. Johnson, Y. Cheng, R. Willson, C. Villalpando, S. Goldberg, A. Huertas, A. Stein, and A. Angelova, "Computer Vision on Mars," International Journal of Com puter Vision, vol. 75, pp. 6792, 2007. J. C. McCall and M. M. Trivedi, "Videobased lane estimation and tracking for driver assistance: survey, system, and evaluation," IEEE Transactions on Intelligent Transportation Systems, vol. 7, pp. 2037, 2006. H. P Moravec, "The Stanford Cart and the CMU Rover," Proceedings of the IEEE, vol. 71, pp. 872884, 1983. M. Nixon and A. S. A. A. (Author), Feature Extraction in Computer Vision and Image Processing 2nd ed. Oxford, UK: Elsevier, 2008. P. Osteen, "Local Sens or Data Fusion Applied to Autonomous Navigation" M.S. thesis, University of Florida Gainesville, Fl., U.S.A 2008. D. Pomerleau, "RALPH: Rapidly adapting lateral position handler," in IEEE symposium on intelligent vehicles 1995, pp. 506511. D. Pomerl eau and T. Jochem, "Rapidly adapting machine vision for automated vehicle steering," IEEE Expert, vol. 11, pp. 1927, 1996.

PAGE 151

151 C. Rasmussen, "Combining laser range, color, and texture cues for autonomous road following," in Robotics and Automation, 2002. Proc eedings. ICRA'02. IEEE International Conference on, 2002. M. Sarfraz, Interactive Curve Modeling: with Applications to Computer Graphics, Vision and Image Processing London, UK: Springer Verlag, 2007. H. Schneiderman and M. Nashman, "A discriminating fea ture tracker for vision based autonomous driving," Robotics and Automation, IEEE Transactions on, vol. 10, pp. 769775, 1994. S. Solanki, "Development of Sensor Component for Terrain Evaluation and Obstacle Detection for an Unmanned Autonomous Vehicle" Ph .D d issertation, University of Florida, Gainesville, Fl., U.S.A, 2006. B. Southall and C. J. Taylor, "Stochastic road shape estimation," in International Conference on Computer Vision, 2001, pp. 205212. R. Sukthankar, D. Pomerleau, C. Thorpe, and I. Carnegie Mellon Univ Pittsburgh Pa Robotics, Panacea: An Active Sensor Controller for the ALVINN Autonomous Driving System : Carnegie Mellon University, The Robotics Institute, 1993. T. Suttorp and T. Bucher, "Learning of Kalman Filter Parameters for Lane Detec tion," in Intelligent Vehicles Symposium, 2006 IEEE 2006, pp. 552557. C. J. Taylor, J. Kosecka, R. Blasi, and J. Malik, "A comparative study of visionbased lateral control strategies for autonomous highway driving," The International Journal of Robotics Research, vol. 18, p. 442, 1999. C. Thorpe, M. Hebert, T. Kanade, and S. Shafer, "Vision and Navigation for the Carnegie Mellon Navlab," Annual Reviews in Computer Science, vol. 2, pp. 521556, 1987. S. Thrun, M. Monte merlo, H. Dahlkamp, D. Stavens, A. Aron, J. Diebel, P. Fong, J. Gale, M. Halpenny, and G. Hoffmann, "Stanley: The robot that won the DARPA Grand Challenge," JOURNAL OF FIELD ROBOTICS, vol. 23, p. 661, 2006. R. Touchton, "An Adaptive Planning Framework for Situation Assessment and Decision Making of an Autonomous Ground Vehicle." Ph.D. dissertation, University of Florida, Gainesville, FL., U.S.A, 2006. S. J. Velat, J. Lee, N. Johnson, and C. D. Crane, "Vision Based Vehicle Localization for Autonomous Naviga tion," in Computational Intelligence in Robotics and Automation, 2007. CIRA 2007. International Symposium on Jacksonville, FI 2007, pp. 528533 V. Vezhnevets, V. Sazonov, and A. Andreeva, "A Survey on Pixel Based Skin Color Detection Techniques," in Proc. Graphicon, 2003.

PAGE 152

152 Y. Wang, D. Shen, and E. K. Teoh, "Lane detection using catmull rom spline," in IEEE International Conference on Intelligent Vehicles 1998. Y. Wang, D. G. Shen, and E. K. Teoh, "Lane detection using spline model," Pattern Recogniti on Letters, vol. 21, pp. 677689, Jul 2000. Y. Wang, E. K. Teoh, and D. G. Shen, "Lane detection and tracking using B Snake," Image and Vision Computing, vol. 22, pp. 269280, Apr 1 2004. C. Witzgall, J. Bernal, and G. S. Cheok, "TIN Techniques for Data Analysis and Surface Construction," National Institute of Standards and Technology Internal Report 7078, 2004, 2004. B. F. Wu, C. T. Lin, and Y. L. Chen, "Dynamic Calibration and Occlusion Handling Algorithms for Lane Tracking," Ieee Transactions on Industrial Electronics, vol. 56, pp. 17571773, May 2009. Y. U. Yim and S. Y. Oh, "Three feature based automatic lane detection algorithm (TFALDA) for autonomous driving," IEEE Transactions on Intelligent Transportation Systems, vol. 4, pp. 219225, 2003. Yu, X., S. Beucher, and M. Bilodeau. "Road tracking, lane segmentation and obstacle recognition by mathematical morphology". in Intelligent Vehicles '92 Symposium., Proceedings of the. 1992. J. Zhang and H. H. Nagel, "Texture based seg mentation of road images," in Intelligent Vehicles' 94 Symposium, Proceedings of the, 1994, pp. 260265.

PAGE 153

153 BIOGRAPHICAL SKETCH Jaesang Lee was born and raised in Seoul, Korea. He completed his B achelor of Science degree in a n E lectrical E ngineering D epartment at the I nha University in Inchon, Korea, and began his M aster of S cience degree at I nha University. After finishing military duty at the Air Force, Jaesang decided to go back to school to finish his master s degree in the United States He finish ed his masters program in E lectrical and C omputer E ngineering D epartment at the University of Florida and then joined his Ph.D. program at the Center for Intelligent Machines and Robotics (CIMAR) in the Mechanical and Aerospace Engineering D epartment Jaesang has worked on various robotics projects as a graduate research assistant under the guidance of his advisor, Dr. C a rl D. Crane III. Among the projects, Jaesang was actively involved in the autonomous vehicle development for the Defense Advanced Research Projects Agency ( DARPA ) Grand Challenge 2005 and DARP A Urban Challenge 2007. His current research goal is computer vision system development for autonomous vehicle and drive assistant system After graduation, Jaesang will continue his career as an engineer w orking in the robotics industry. His field of interest includes computer vision, image processing, and pattern recognition system development for real time and real world