Cubesat Cloud

MISSING IMAGE

Material Information

Title:
Cubesat Cloud a Framework for Distributed Storage, Processing and Communication of Remote Sensing Data on Cubesat Clusters
Physical Description:
1 online resource (101 p.)
Language:
english
Creator:
Challa, Obulapathi Nayudu
Publisher:
University of Florida
Place of Publication:
Gainesville, Fla.
Publication Date:

Thesis/Dissertation Information

Degree:
Doctorate ( Ph.D.)
Degree Grantor:
University of Florida
Degree Disciplines:
Electrical and Computer Engineering
Committee Chair:
MCNAIR,JANISE Y
Committee Co-Chair:
LI,XIAOLIN
Committee Members:
LATCHMAN,HANIPH A
FITZ-COY,NORMAN G

Subjects

Subjects / Keywords:
bigdata -- cloud -- communication -- cubesat -- processing -- satellite -- storage
Electrical and Computer Engineering -- Dissertations, Academic -- UF
Genre:
Electrical and Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract:
CubeSat Cloud is a novel vision for a space based remote sensing network that includes a collection of small satellites (including CubeSats), ground stations, and a server, where a CubeSat is a miniaturized satellite with a volume of a 10x10x10 cm cube and has a weight of approximately 1 kg. The small form factor of CubeSats limits the processing and communication capabilities – implemented and deployed CubeSats have demonstrated about 1 GHz processing speed and 9.6 kbps communication speed. A CubeSat in its current state can take hours to process a 100 MB image and more than a day to downlink the same, which prohibits remote sensing, considering the limitations in ground station access time for a CubeSat. This dissertation designs an architecture and supporting networking protocols to create CubeSat Cloud, a distributed processing, storage and communication framework that will enable faster execution of remote sensing missions on CubeSat clusters. The core components of CubeSat Cloud are CubeSat Distributed File System, CubeSat MapMerge, and CubeSat Torrent. The CubeSat Distributed File System has been created for distributing of large amounts of data among the satellites in the cluster. Once the data is distributed, CubeSat MapReduce has been created to process the data in parallel, thereby reducing the processing load for each CubeSat. Finally, CubeSat Torrent has been created to downlink the data at each CubeSat to a distributed set of ground stations, enabling faster asynchronous downloads. Ground stations send the downlinked data to the server to reconstruct the original image and store it for later retrieval. Analysis of the proposed CubeSat Cloud architecture was performed using a custom-designed simulator, called CubeNet and an emulation test bed using Raspberry Pi devices. Results show that for cluster sizes ranging from 5 to 25 small satellites, faster download speeds – up to 4 to 22 times faster - can be achieved when using CubeSat Cloud, compared to a single CubeSat. These improvements are achieved at an almost negligible bandwidth and memory overhead (1%).
General Note:
In the series University of Florida Digital Collections.
General Note:
Includes vita.
Bibliography:
Includes bibliographical references.
Source of Description:
Description based on online resource; title from PDF title page.
Source of Description:
This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility:
by Obulapathi Nayudu Challa.
Thesis:
Thesis (Ph.D.)--University of Florida, 2013.
Local:
Adviser: MCNAIR,JANISE Y.
Local:
Co-adviser: LI,XIAOLIN.

Record Information

Source Institution:
UFRGP
Rights Management:
Applicable rights reserved.
Classification:
lcc - LD1780 2013
System ID:
UFE0046139:00001


This item is only available as the following downloads:


Full Text

PAGE 1

CUBESATCLOUD:AFRAMEWORKFORDISTRIBUTEDSTORAGE,PROCESSING ANDCOMMUNICATIONOFREMOTESENSINGDATAONCUBESATCLUSTERS By OBULAPATHINAYUDUCHALLA ADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOL OFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENT OFTHEREQUIREMENTSFORTHEDEGREEOF DOCTOROFPHILOSOPHY UNIVERSITYOFFLORIDA 2013

PAGE 2

2013ObulapathiNayuduChalla 2

PAGE 3

Idedicatethistomyfamily,mywifeSreevidyaInturi,mymoth erRangammaChalla,my fatherAnanthaiahChalla,mysisterSreelathaChowdaryLingu tla,mybrother-in-laws RameshNaiduLingutla,SreekanthChowdaryInturi,myfather -in-lawSreenivasulu ChowdaryInturi,mymother-in-lawVenkatalakshmiInturi, mybrothersAkshayKumar AnuguandDheerajKotaandmyuncleVenkatanarayanaPattipaa ti,foralltheirloveand support. 3

PAGE 4

ACKNOWLEDGMENTS IthasbeenagreatexperiencebeingastudentofDr.JaniseY. McNairforthelast veandhalfyears.TherewasneveratimethatIdidnotfeelca redfor,thankstoher constantsupportandguidance. IwouldliketothankmycommitteeDr.Xiaolin(Andy)Li,Dr.Nor manG.Fitz-Coy andDr.HaniphA.Latchmanforagreeingtoserveonmycommitte e.Iwouldliketo thankthemforprovidingvaluablefeedbackincompletingmy dissertation.Thanksto theprofessorsatUniversityofFloridaDr.PatrickOscarBoy kin,Ms.WenhsingWu,Dr. RamakantSrivastava,Dr.ErikSander,Dr.A.AntonioArroyo,Dr.Jo seA.B.Fortes,Dr. JohnM.Shea,Dr.GregStitt,Dr.SartajSahniandDr.ShigangChen, forteachingme whatallIknowtoday.ThankstostaffatUniversityofFlorid aRayE.McClureII,Jason Kawaja,ShannonMChillingworth,CherylRhodenandStephenieA. Sparkman,for theirpatiencewithmycountlessrequestsandadministrati vequestions.Iwouldliketo takethisopportunitytothankallmyWirelessandMobileGrou pcolleagues,pastand present,forbeingtherewithmeandhelpingmeallalonginon ewayorother.Iwould liketothankAlexanderVerbitskiforhismentorshipduringm yinternship. IwouldliketothankmyteachersSreedevi,UmaKantha,NaliniSr eenivasan,K. Ramakrishna,K.BhaskarNaidu,SambasivaReddy,AKoteswarRao ,A.K.Rama Rao,Dr.VijayKumarChakka,Dr.GautamDuttaandDr.PrabhatRa njanwhogreatly inuencedmylife.InternetandOpenSourcehavemadethiswor ldatrueVasudhaika Kutumbamforme.IwouldliketothankLinusTorvalds,creato rofLinux;Richard MatthewStallman,founderofGNU;VintCerf,fatherofInterne t;TimBerners-Lee, inventoroftheWorldWideWeb;GuidoRossum,creatorofPython programming language;SatoshiNakamoto,inventorofBitcoin;MasashiKish imoto,creatorofNaruto; MarkShuttleworth,founderofUbuntuandTimO'Reilly,thefo underofO'ReillyMedia. LifeatUniversityofFloridahasbeenalwaysfunandexcitin g,thankstothe wonderfulfriendsaroundhere:DanTrevino,DanteBuckley,G okulBhat,Hrishikesh 4

PAGE 5

Pendurkar,Jimmy(TzuYu)Lin,KarthikTalloju,KrishnaChait anya,KishoreYalamanchili, MadhulikaDandina,ManuRastogi,PaulMuri,RakeshChalasa ni,RaviShekhar,Seshu Pria,ShruthiVenkatesh,SubhashGuttikonda,UdayanKumar,Va ibhavGarg,Vijay BhaskarReddyandVivekAnand.IwouldliketothankMr.IqbalQai yumi,Dr.Shaheda Qaiyumi,Mr.JagatDesaiandMrs.VatsalaDesaifortakingca reofmeliketheirson. Thankstomylong-distancefriendsBhargaviVanga,PraveenKu mar,RadhaVummadi, UdayKumar,UzumakiNarutoandVijayKumar,whohavebeenclos eevenwhenthey werefar. Lastly,Iwouldliketothankmyfamily-mywifeSreevidyaIntu ri,mymother RangammaChalla,myfatherAnanthaiahChalla,mysisterSreel athaChowdary Lingutla,mybrother-in-lawsRameshNaiduLingutla,Sreeka nthChowdaryInturi,my father-in-lawSreenivasuluChowdaryInturi,mymother-inlawVenkatalakshmiInturi, mybrothersAkshayKumarAnuguandDheerajKotaandmyuncleVen katanarayana Pattipaati.Theirendlessloveandsupportthroughoutthey earshasmeantmoretome thanwordscanexpress.Iwouldliketodedicatemydissertat iontothem. 5

PAGE 6

TABLEOFCONTENTS page ACKNOWLEDGMENTS .................................. 4 LISTOFTABLES ...................................... 10 LISTOFFIGURES ..................................... 11 ABSTRACT ......................................... 13 CHAPTER 1INTRODUCTION ................................... 15 1.1CubeSatCloud ................................. 17 2BACKGROUND ................................... 20 2.1RemoteSensing ................................ 20 2.2EvolutionofCubeSatNetworks ........................ 21 2.2.1SummaryandLimitationsofCubeSatCommunications ...... 24 2.3DistributedSatelliteSystems ......................... 25 2.4ClassicationofDistributedSatelliteSystems ................ 27 2.5RelatedWork .................................. 27 2.5.1DistributedStorageSystems ..................... 27 2.5.2DistributedComputingTechniques .................. 30 3NETWORKARCHITECTUREOFCUBESATCLOUD .............. 34 3.1ComponentsoftheCubeSatNetwork .................... 35 3.1.1SpaceSegment ............................. 35 3.1.2GroundSegment ............................ 37 3.2SystemCommunication ............................ 38 3.2.1ClusterCommunication ........................ 38 3.2.2SpaceSegmenttoGroundSegmentCommunication ....... 39 3.2.3GroundSegmentNetworkCommunication .............. 40 3.3CubeSatCloud ................................. 40 3.3.1Storage,ProcessingandCommunicationofRemoteSensing Data onCubeSatClusters .......................... 40 3.3.2SourceCoding,StoringandDownlinkingofRemoteSensing Data onCubeSatClusters .......................... 41 4DISTRIBUTEDSTORAGEOFREMOTESENSINGIMAGESONCUBESAT CLUSTERS ...................................... 45 4.1KeyDesignPoints ............................... 45 4.1.1NeedforSimpleDesign ........................ 45 4.1.2LowBandwidthOperation ....................... 45 6

PAGE 7

4.1.3NetworkPartitionTolerant ....................... 45 4.1.4Autonomous ............................... 46 4.1.5DataIntegrity .............................. 46 4.2SharedGoalsBetweenCDFS,GFSandHDFS ............... 46 4.2.1ComponentFailuresareNorm .................... 46 4.2.2SmallNumberofLargeFiles ..................... 46 4.2.3ImmutableFilesandNon-existentRandomReadWrites ...... 47 4.3ArchitectureofCubeSatDistributedFileSystem .............. 47 4.3.1FileSystemNamespace ........................ 49 4.3.2Heartbeats ............................... 49 4.4FileOperations ................................. 50 4.4.1CreateaFile .............................. 50 4.4.2WritingtoaFile ............................. 51 4.4.3DeletingaFile .............................. 51 4.5EnhancementsandOptimizations ...................... 52 4.5.1BandwidthandEnergyEfcientReplication ............. 52 4.5.1.1Numberofnodesoncommunicationpath=replication factor ............................. 54 4.5.1.2Numberofnodesoncommunicationpath > replication factor ............................. 54 4.5.1.3Numberofnodesoncommunicationpath < replication factor ............................. 54 4.5.2LoadBalancing ............................. 55 4.5.3ChunkSizeandGranularity ...................... 56 4.5.4FaultTolerance ............................. 57 4.5.5MasterFailure .............................. 57 4.5.6WorkerFailure ............................. 57 4.5.7ChunkCorruption ............................ 58 4.5.8InterCubeSatLinkFailure ....................... 58 4.5.9NetworkPartitioning .......................... 58 4.6SimulationResults ............................... 59 4.7SummaryofCubeSatDistributedFileSystem ................ 59 5DISTRIBUTEDPROCESSINGOFREMOTESENSINGIMAGESONCUBESAT CLUSTERS ...................................... 60 5.1CubeSatMapMerge .............................. 60 5.2CommandandDataFlowduringaCubeSatMapMergeJob ........ 61 5.3FaultTolerance,Failures,GranularityandLoadBalanci ng ......... 63 5.3.1FaultTolerance ............................. 63 5.3.2MasterFailure .............................. 64 5.3.3WorkerFailure ............................. 64 5.3.4TaskGranularityandLoadBalancing ................. 64 5.4SimulationResults ............................... 64 5.5SummaryofCubeSatMapMerge ....................... 65 7

PAGE 8

6DISTRIBUTEDCOMMUNICATIONOFREMOTESENSINGIMAGESFROM CUBESATCLUSTERS ............................... 66 6.1CubeSatTorrent ................................ 66 6.2CommandandDataFlowDuringaTorrentSession ............. 67 6.3EnhancementsandOptimizations ...................... 67 6.3.1ImproveStorageReliabilityandDecreaseStorageOverh ead ... 67 6.3.2UsingSourceCodingtoImproveDownlinkTime .......... 69 6.3.3ImprovingtheQualityofServiceforReal-timeTrafcAp plications LikeVoIP ................................ 70 6.4FaultTolerance,Failures,GranularityandLoadBalanci ng ......... 71 6.4.1FaultTolerance ............................. 71 6.4.2MasterFailure .............................. 72 6.4.3WorkerFailure ............................. 72 6.4.4TaskGranularity ............................ 72 6.4.5TailEffectandBackupDownloads .................. 73 6.5SimulationResultsandSummaryofCubeSatTorrent ........... 73 7SIMULATOR,EMULATORANDPERFORMANCEANALYSIS .......... 75 7.1HardwareandSoftwareofMasterandWorkerCubeSatsforEmul ator .. 75 7.2HardwareandSoftwareofServerandGroundStationforEmulat or .... 77 7.3NetworkProgrammingFrameworks ..................... 77 7.3.1Twisted ................................. 77 7.3.2Eventlet ................................. 77 7.3.3PyEv ................................... 78 7.3.4Asyncore ................................ 78 7.3.5Tornado ................................. 78 7.3.6Concurrence .............................. 78 7.4TwistedFramework ............................... 78 7.5NetworkConguration ............................. 79 7.6CubeSatCloudEmulatorSetup ........................ 79 7.7CubeSatCloudSimulatorSetup ....................... 79 7.8CubeSatReliabilityModel ........................... 82 7.9SimulationandEmulationResults ...................... 82 7.9.1ProlingReadingandWritingofRemoteSensingDataChunk s onRaspberryPi ............................ 82 7.9.2Processing,CubeSattoCubeSatandCubeSattoGroundStatio n ChunkCommunicationTime ...................... 83 7.9.3StoringRemoteSensingImagesusingCubeSatCloud ....... 84 7.9.4ProcessingRemoteSensingImagesusingCubeSatCloud .... 85 7.9.5SpeedupandEfciencyofCubeSatMapMerge ........... 86 7.9.6DownlinkingRemoteSensingImagesUsingCubeSatCloud .... 87 7.9.7SpeedupandEfciencyofCubeSatTorrent ............. 88 7.9.8CopyOnTransmitOverhead ...................... 89 7.9.9SourceCodingOverhead ....................... 89 8

PAGE 9

7.9.10MetadataandControlTrafcOverhead ................ 90 7.9.11ComparisonofCDFSwithGFSandHDFS ............. 90 7.9.12SimulatorvsEmulator ......................... 91 7.10SummaryofSimulationResults ........................ 95 8SUMMARYANDFUTUREWORK ......................... 96 8.1Futurework ................................... 97 REFERENCES ....................................... 98 BIOGRAPHICALSKETCH ................................ 101 9

PAGE 10

LISTOFTABLES Table page 2-1CubeSatdataspeedsanddownloads ....................... 25 10

PAGE 11

LISTOFFIGURES Figure page 1-1CubeSat ........................................ 16 2-1GenerationsofCubeSatnetworks ......................... 23 2-2Genso ......................................... 24 2-3ArchitecturaloverviewoftheGoogleFileSystem ................. 29 2-4ArchitecturaloverviewoftheHadoopDistributedFileSys tem .......... 31 3-1ArchitectureofaCubeSatnetwork ......................... 34 3-2ArchitectureofaCubeSatcluster .......................... 35 3-3AblownuppictureofESTCube-ICubeSat,showingitssubsyst ems ...... 36 3-4Groundstation .................................... 38 3-5Groundstationantenna ............................... 39 3-6OverviewofCubeSatCloudanditscomponentframeworks ........... 42 3-7IntegrationofCubeSatDistributedFileSystemandCubeSat Torrent ...... 44 4-1ArchitectureofCubeSatDistributedFileSystem ................. 48 4-2Bandwidthandenergyefcientreplication ..................... 53 4-3Copyontransmit ................................... 55 5-1ExampleofCubeSatMapMerge .......................... 61 5-2OverviewofexecutionofCubeSatMapMergeonCubeSatclust er ....... 62 6-1OverviewofCubeSatTorrent ............................ 68 7-1RaspberryPiminicomputer ............................. 76 7-2CubeSatCloudemulator .............................. 80 7-3CubeSatCloudsimulator .............................. 81 7-4LifetimesofCubeSats ................................ 83 7-5Readandwritetimesofachunk .......................... 84 7-6CubeSattoCubeSatandCubeSattogroundstationchunkcommu nication proling ........................................ 85 7-7Filedistributiontimeforvariouslesizesandcluster sizes ............ 86 11

PAGE 12

7-8Fileprocessingtimeforvariouslesizesandclustersi zes ............ 86 7-9SpeedupofCubeSatMapMerge .......................... 87 7-10EfciencyofCubeSatMapMerge .......................... 88 7-11Filedownlinkingtimeforvariouslesizesandcluster sizes ........... 89 7-12SpeedupofCubeSatTorrent ............................ 90 7-13EfciencyofCubeSatTorrent ............................ 91 7-14Bandwidthoverheadduetoreplication ....................... 92 7-15Bandwidthoverheadduetosourcecoding ..................... 92 7-16Bandwidthandenergyoverhead .......................... 93 7-17BandwidthconsumptionofCDFSvsGFSandHDFS ............... 93 7-18WritetimeofCDFSvsGFSandHDFS ...................... 94 7-19EnergyconsumptionofCDFSvsGFSandHDFS ................. 94 7-20Simulatorvsemulator ................................ 95 12

PAGE 13

AbstractofDissertationPresentedtotheGraduateSchool oftheUniversityofFloridainPartialFulllmentofthe RequirementsfortheDegreeofDoctorofPhilosophy CUBESATCLOUD:AFRAMEWORKFORDISTRIBUTEDSTORAGE,PROCESSING ANDCOMMUNICATIONOFREMOTESENSINGDATAONCUBESATCLUSTERS By ObulapathiNayuduChalla December2013 Chair:JaniseY.McNairMajor:ElectricalandComputerEngineering CubeSatCloudisanovelvisionforaspacebasedremotesensin gnetworkthat includesacollectionofsmallsatellites(includingCubeSa ts),groundstations,anda server,whereaCubeSatisaminiaturizedsatellitewithavol umeofa10x10x10cm cubeandhasaweightofapproximately1kg.Thesmallformfac torofCubeSatslimits theprocessingandcommunicationcapabilities.Implement edanddeployedCubeSats havedemonstratedabout1GHzprocessingspeedand9.6kbpsc ommunicationspeed. ACubeSatinitscurrentstatecantakehourstoprocessa100MB imageandmorethan adaytodownlinkthesame,whichprohibitsremotesensing,c onsideringthelimitations ingroundstationaccesstimeforaCubeSat. Thisdissertationdesignsanarchitectureandsupportingn etworkingprotocolsto createCubeSatCloud,adistributedprocessing,storageand communicationframework thatwillenablefasterexecutionofremotesensingmission sonCubeSatclusters.The corecomponentsofCubeSatCloudareCubeSatDistributedFile System,CubeSat MapMerge,andCubeSatTorrent.TheCubeSatDistributedFileSy stemhasbeen createdfordistributingoflargeamountsofdataamongthes atellitesinthecluster.Once thedataisdistributed,CubeSatMapReducehasbeencreatedt oprocessthedata inparallel,therebyreducingtheprocessingloadforeachC ubeSat.Finally,CubeSat TorrenthasbeencreatedtodownlinkthedataateachCubeSatt oadistributedsetof groundstations,enablingfasterasynchronousdownloads. Groundstationssendthe 13

PAGE 14

downlinkeddatatotheservertoreconstructtheoriginalim ageandstoreitforlater retrieval. AnalysisoftheproposedCubeSatCloudarchitecturewasperfo rmedusinga custom-designedsimulator,calledCubeNetandanemulatio ntestbedusingRaspberry Pidevices.Resultsshowthatforclustersizesrangingfrom5 to25smallsatellites, fasterdownloadspeedsupto4to22timesfaster-canbeachie vedwhenusing CubeSatCloud,comparedtoasingleCubeSat.Theseimprovemen tsareachievedat analmostnegligiblebandwidthandmemoryoverhead(1%). 14

PAGE 15

CHAPTER1 INTRODUCTION ACubeSatisaminiaturizedsatelliteprimarilyusedforuniv ersityspaceresearch [ 1 ].Ithasavolumeofexactlyonelitre,weighsnomorethanone kilogramandisbuilt usingcommercialoff-the-shelfcomponents[ 2 ].Futuresatellitesystemsareenvisioned tobemadeupofaclusterorconstellationofsmallersatelli teslikeCubeSatsinsupport ofhugemonolithicsatellitestogetherformingadistribut edspacenetwork.However, weight,volume,powerandgeometryconstraintsofCubeSatsm ustbeovercomein ordertoproviderequiredprocessing,storageandcommunic ationcapabilities.Figure 1-1 showsthepictureofaCubeSat.ACubeSathasonlyabout1GHzpro cessor,1 GBofRAM,32-64GBofashmemoryand9.6kbpscommunicationca pability[ 2 ] [ 3 ].Onotherhand,remotesensingmissionslikeweathermonit oring,oodmonitoring andvolcanicactivitymonitoringrequireintensiveproces singordownlinkinglarge amountsofdata.Withitslimitedresources,aCubeSatcantake hourstoprocessone remotesensingimageanddaystodownlinkthesame[ 4 ][ 5 ].Thus,processingand communicationsystemshavebecomebottlenecksforemployi ngCubeSatsonremote sensingmissions. TheadvantagesofCubeSatsareitslowcost,lowroundtriptim eforcommunication ofgroundstationandtheyareeasytoexperimentwith.Thema nufacturingcostofa typicallargesatelliteweighingabout1000kgisontheorde rofhundredsofmillions ofdollars[ 6 ]becauseallthecomponentsarecustommadeandneedtobetes ted extensivelybeforelaunch.However,mostofthecomponents ofaCubeSatare commercialofftheshelfcomponents(COTS).Onlythepayload iscustomdesigned fromgroundup.Thus,CubeSatscanbeengineeredatapriceofa bouthalfamillionto afewmilliondollars.Thiscostisordersofmagnitudelesst hanthecostofatypicallarge satellite[ 7 ][ 3 ].LaunchescanbeachievedingroupsofCubeSatsatonetime. 15

PAGE 16

Figure1-1.CubeSatImagecourtesyofNASA.PicturebyPaulAdams. Largesatellitesarelaunchedintogeostationaryearthorb it(GEO)orhighlyelliptical orbitorhighearthorbit(HEO)orbitwhichare36000kmor5000 0km.Asaresultof thelongdistancebetweenearthandsatellite,signalpropa gationdelayisabout120 msandroundtriptimeapproximatestoabout250ms.CubeSatsa relaunchedintolow earthorbit(LEO)orbitwhichisabout600-800kmfromearth.As aresult,theRTTfor thesignalreducestoabout10ms,whichcouldbetterquality ofserviceforapplications likerealtimetrackingandvoiceapplications,whencompar edtothatofRTTforGEOor HEOsatellite. Finally,sinceaCubeSatmissioncostshalfamilliontoafewm illiondollarsandcan belaunchesinlargenumbersusingasinglerocket,missionf ailureisnotfatal.Since missionfailureisnotfatal,andcostsarelower,newtechno logiescanbeeasilyinserted intoanexistingspacenetworkviaCubeSats. CubeSatshaveverylimitedresourcestoaccomplishmeaningf ulremotesensing missions.TypicalprocessingpowerofCubeSatsisabout1GHz andhas1GBof RAM.Asaresult,computationalpowerofCubeSatsisnotsufcie ntforexecuting 16

PAGE 17

processingintensiveremotesensingmissions.CubeSatsuse structurallysimplelow gainantennaslikemonopoleordipoleandhavelimitedpower budgetofabout500mW forcommunicationsystem.Thetypicalcommunicationdatar atebetweenaCubeSat andagroundstationisabout9.6kbps[ 5 ].Asaresult,largeamountsofdatacannot bedownloadedtogroundstationsinareasonableamountofti me.CubeSatshavelow memory,processing,batterypowerandcommunicationcapab ilities.Timingconstraints aretootighttohavelongcommunicationwindows.EachCubeSat iscontrolled individually.Currently,thereisnomeaningfulwayofcont rollingmultipleCubeSats usingauniedcontrolmechanism.Asaresult,asingleCubeSat cannotperform processingandcommunicationintensiveremotesensingmis sionsinameaningfultime. 1.1CubeSatCloud InthisworkweproposeCubeSatCloud,aframeworkfordistrib utedstorage, processingandcommunicationofremotesensingdata.Wedem onstratethatCubeSat CloudcanstoreremotesensingdataonaCubeSatclusterinadi stributedfashionto allowthepossibilityofdistributedcomputationandcommu nication,speedingupremote sensingmissions.Fordistributingremotesensingdata,Cu beSatCloudusesCubeSat DistributedFileSystem.Fordistributedprocessingandcom munication,CubeSatCloud usesCubeSatMapMergeandCubeSatTorrentrespectively.Were ducethebandwidth andenergyconsumptionbyenergyefcientreplicationandl inerblocksourcecoding. Inchapter2weoutlinetheevolutionofCubeSatNetworkandpr esentrelevant backgroundindistributedsatellitesystems,storagesyst emsandcomputingtechniques. Inchapter3,wedescribethearchitectureofCubeSatNetwork ,whichconsistsoftwo segments,namelyaspacesegmentandagroundsegment.thesp acesegmentis designedtobeaCubeSatclusterwitharadiusofabout100km.I tconsistsofSensor nodesandWorkernodeswhichareinter-connectedusinghigh speedcommunication links.SensorCubeSathassensingsubsystemandactasMastero ftheclusterwhile executingremotesensingmissions.Workernodesaretypica l1UCubeSats(10x10x 17

PAGE 18

10cmcube)withstandardsubsystems.Groundsegmentismade upofagroundstation serverandseveralgroundstations.Groundstationsarecon nectedtotheserverviathe Internet.Groundstationsactasrelaysbetweengroundstat ionserverandCubeSats. Ontopofthedescribednetworkarchitecture,webuildtheCu beSatCloudplatform. InChapter4,wedescribethethreecorecomponentsofCubeSat Cloudnamely CubeSatDistributedFileSystem(CDFS),CubeSatMapMergeandCu beSatTorrent. CubeSatDistributedFileSystemisusedfordistributingther emotesensingdatatothe nodesintheCubeSatcluster.Oncetheremotesensingdataisd istributed,CubeSat MapMergeandCubeSatTorrentareusedforprocessinganddown linkingtheremote sensingdata. CDFSsplitslargesizedremotesensingdataintochunksandd istributesthechunks totheWorkernodesinthecluster.CDFSuses”Copy-On-Trans mit”forcreatingreplicas withverylowbandwidthandenergyoverhead.Sourcecodingis usedforreducing thestorageandbandwidthoverheadformissionswhichrequi reonlydownlinkingof remotesensingdata.WedemonstratethatCDFScanstoredata reliably,withoutloss ofanydata,evenifalimitednumberofCubeSatsgoofine.Dis tributingtheremote sensingdatatonodesintheclusterallowsthepossibilityo fdistributedprocessingand distributedcommunicationforspeedinguptheremotesensi ngmissions. Inchapter6,wedescribetheworkingofCubeSatMapMergeinde tail.CubeSat MapMergeisadistributedprocessingframeworkinspiredby GoogleMapReduce[ 8 ]. Workernodesprocessthechunksstoredwiththeminparallel .Failuresaredetected usingtheHeartbeatmechanismandfailedexecutionsarerescheduledonotherworker nodes.WedemonstratethatCubeSatMapMergecanspeedupproc essingoflarge sizesofremotesensingdataonCubeSatclustersbyafactorof thesizeofcluster(i.e., thenumberofCubeSatsinthecluster)andisresilienttowork erandcommunicationlink failures. 18

PAGE 19

InChapter7weexplainthedownlinkprocess.Multiplerawor processedchunks aredownlinkedinparalleltogroundstations.Onceachunki sdownlinkedtoaground station,itisforwardedtothegroundstationserver.Afterr eceivingallchunks,theServer usesthechunkstoreproducesensorimage.Wedemonstrateth atCubeSatTorrent canspeedupthedownlinkingoflargelesbyafactorofthecl ustersize(numberof CubeSatsinthecluster)andisresilienttoworkerandcommun icationlinkfailures. Totesttheperformanceofthesystem,webuiltaCubeSatCloud simulation frameworkandaCubeSatCloudtestbedforemulation.Wedescr ibethetestbed andsimulationsetupindetailinchapter8.CubeSatsareemul atedusingRaspberry Pimini-computers,whiletheterrestrialServerandgroundst ationsareemulated usingstandarddesktopcomputers.CubeSatCloudiswritteni nPythonprogramming languageusingTwisted,aneventbasedasynchronousnetwor kprogrammingframework. SimulationresultsindicatethatCubeSatMapMergeandCubeSat Torrent,withcluster sizesintherangeof5-25CubeSatstogetherenable4.75-23.1 5timesfaster (comparedtoasingleCubeSat)processinganddownlinkingof largesizedremote sensingdata.Thisspeedisachievedatanalmostnegligible bandwidthandmemory overhead(1%).EmulationresultsfromtheCubeSatCloudtestb edagreewithsimulation resultsand,indicatethatourproposedCubeSatCloudcanspe edupremotesensing missionsbyafactorofthesizeoftheCubeSatclusterwithmin imaloverhead,while achievingasynchronousdownloadwithshortcommunication windows. 19

PAGE 20

CHAPTER2 BACKGROUND TheCubeSatconceptwasinitiatedbyProfessorTwiggsasateac hingtoolto helpstudentslearntheprocessofdeveloping,launchingan doperatingsatellite. CubeSatsarecurrentlydesignedforlowEarthorbits.Theyare wellsuitedfordistributed sensingapplicationsandlowdataratecommunicationsappl ications.Unlikethelarge monolithicsatellites,CubeSatsarebuilttoalargedegreeu singcommercialofftheshelf components(COTS).EngineeringCubeSatsusingCOTSequipment andfollowing standardsindesignanddevelopmenthaveshortenedthedeve lopmentcyclesand reducedcosts.CubeSatsaretypicallylaunchedanddeployed usingamechanismcalled P-POD[ 9 ],developedandbuiltbyCalPoly.P-PODsaremountedtoalaunc hvehicle carryingCubeSatsanddeploythemoncethepropersignalisre ceivedfromthelaunch vehicle.TheP-PODMkIIIhasacapacityforthree1UCubeSats.APPODcandeploy three1Uorone1Uandone2Uorone3UCubeSat. CubeSatscarryoneortwoscienticpayloadslikemagnetice ldsensor,image sensororionconcentrationnder.Severalcompaniesandres earchinstitutesoffer regularlaunchopportunitiesinclustersofseveralcubes. CubeSatasaspecicationfor constructinganddeployingpicosatellitesaccomplishesf ollowinggoals: 1. Encapsulationoflauncher-payloadinterface:CubeSatstand ardeliminatesa signicantamountofmanagerialworkandmakesiteasytomat eapiggyback satellitewithitslauncher. 2. Unicationamongpayloadsandlaunchers:Satellitesadheri ngtoCubeSat standardcanbeinterchangedquicklyforoneanotherandthu senablesutilization oflaunchopportunitiesonshortnotice. 3. Simplicationofpicosatelliteinfrastructure:CubeSatsta ndarditpossibletodesign andproduceanoperationalsmallsatelliteataverylowcost 2.1RemoteSensing Acquiringinformationaboutanobjectwithoutmakingphysic alcontactwithitis calledRemotesensing.Usuallyitreferstogatheringinfor mationaboutatmosphereand earthsurfaceusingsatellites.Remotesensingcanbeperfo rmedineitherpassiveor 20

PAGE 21

activesensors.Passiveremotesensorsusenaturalradiati onreectedbytheobject underobservation.Filmphotography,infrared,charge-co upleddevicesandradiometers areexamplesofpassiveremotesensors.Activeremotesensor smakeuseofradiation sourceandobserveobjectsusingthescatteredorreectedr adiation.Examples ofactiveremotesensorsincludeRADARandLiDAR.Itiseasytoco llectthedata frominaccessibleanddangerousplacesusingremotesensin g.Weathermonitoring, deforestationmonitoring,glacialactivitymonitoring,v olcanomonitoring,oodandother disastermonitoringaresomeoftheexamplesofremotesensi ngapplications.Each datapointcollectedbyremotesensorsistypicallyanywher ebetween10MBto100 MB.Resolutionofaremotesensorvariesfrom1m-1000mperpi xeldependingon thesensor.remotesensingdataisimmutable.Itdoesnotcha ngeafteracquisition. Withitsgivenresources,asingleCubeSatcantakeabout10hou rsforprocessing aremotesensingimageand2daystodownlinkthesame[ 5 ].Executionofremote sensingmissionsusingCubeSatscanbespeededupbyparallel izingtheprocessing anddownlinkingoftheremotesensingimagesusingacluster ofCubeSats. 2.2EvolutionofCubeSatNetworks SincethelaunchoftherstCubeSatintospacein2003,CubeSatc ommunication networkshaveevolvedinseveralways.VeryearlyCubeSatsus edtocommunicate withtheirhomegroundstationonlyasshownintheFigure 2-1 .Thesenetworkscan beclassiedasgeneration1CubeSatnetworks.AtypicalCube Satin600-800km orbithasawindowofabout8minutesanditgetsincontactwit hgroundstationabout 4timesaday.Thislimitedthecommunicationwindowtoabout 25minutesperday. TherstgenerationCubeSatsoperatedatspeedsof1.2kbps.T hislimitsthetotal downlinkcapacityto1.8MB.HowevernoCubeSatachievedthis highdownlinkoruplink bandwidthduetovariouslimitationsincludinginefcient protocols,largeamountof beacondata,powerconstraints,unreliablecommunication systemsonboard.Asa 21

PAGE 22

result,mostmissionscollectedamodestamountofaboutafe wMBofdata( < 12MB) fortheirwholelifetime[ 10 ]. WiththeintroductionofMoreDBs(Massiveoperations,record ing,andexperimentation DatabaseSystem)[ 11 ],CubeSatnetworksmadethenextsignicantstepintheir evolution.MoreDBsisasystemtomanagealldatageneratedby CalPolysmall satellites.Itisanattempttoconsolidateallsatellitein formationintoasingle,readily accessiblelocationtomakedataanalysismoreefcient.Us ingnetworkslikeMoreDBs, missioncontrollerscancollectthebeaconsfromtheirsmal lsatellitesreceivedbyother amateurradiooperatorsasshownintheFigure 2-1 .Thissignicantlyincreasedthe amountofsmallsatellitehealthinformationavailableand alsoservedtotrackthe whereaboutsofsmallsatellite.TheseeffortsbroughtCube Satnetworksintogeneration 2. However,MoreDBhasfollowingtwosignicantlimitations. First,MoreDBarchitecture requiresmissionspecicsoftwaretobedevelopedanddistr ibutedtothegroundstation facilities.Second,anymodicationtothepacketformatreq uiresanupgradeofsoftware atallthegroundstations.Thisiscumbersomeanderrorpron e. Asaresultoftheselimitations,thesolutionisnotscalable toalargenumberof CubeSats.Inordertoovercometheselimitations,SpaceSystem sGroup(SSG)[ 12 ] andWirelessAndMobileLab[ 13 ]atUniversityofFloridadevelopedACloudComputing ArchitectureforSpacecraftTelemetryCollection(T-C3)[ 14 ],ascalableandexible meansofcollectingthetelemetrydata.T-C3isanefforttoi mproveMoreDBandmakeit auniversaltelemetrydecodingsolutionsolutionforCubeSa ts.Insteadofdecodingthe receivedbeaconattheamateurradiostationdirectly,T-C3 forwardsthebeacontothe T-C3centralserverwhichngerprintsthebeaconanddecode sthebeaconusingthat satellitestelemetryformat,makingitamuchmorescalable andexiblesolution[ 14 ]. GENSO(GlobalEducationalNetworkforSatelliteOperations)[ 15 ]isthenext signicantmilestoneintheevolutionofCubeSatnetworks.G ENSOwasfoundedto 22

PAGE 23

Figure2-1.Generation1(a)andGeneration2(b)CubeSatNetw orks ImagecourtesyofSpaceSystemsGroup.PicturebyTzuYu(Jimmy) Lin. createanetworkofamateurradiostationsaroundtheworldt osupportthesmallsatellite operationsofvariousuniversities.GENSOhasbeendesigneda sadistributedsystem connectedviatheInternetasshownintheFigure 2-2 .Thesatellitecancommunicate withthemainbasestationthroughanyarbitraryavailabler elaystation.Withasingle groundstation,auniversitycangatherabout25minutesofd atafromaCubeSatina day.UsingtheGENSOnetwork,missioncontrollerscangatherh oursofworthdata perdaybyreceivingdataviahundredsofnetworkedradiosta tionsaroundtheworld. Itwillalsoallowthemtocommandtheirspacecraftfromtheo thergroundstations. GENSOandothersimilareffortscanbeclassiedasgeneration 3CubeSatnetworks. GENSOplanstohaveabuilt-indatabaseofallthesatellites.T hisdatabasecanbe usedtopredictandautomatethetrackingofthesatellitest ocollectthetelemetryin anefcientway.Oncethedataisdownlinked,thedatawillbe providedtorespective missioncontrollers. 23

PAGE 24

Figure2-2.ArchitectureofGENSO 2.2.1SummaryandLimitationsofCubeSatCommunications Table 2-1 summarizesthedataspeedsanddatadownloadsofCubeSatcomm unication systems.TypicalcharacteristicsofaCubeSatcommunicatio nsubsystemcanbe summarizedasfollows.Datarateis9600baud,powerrating5 00mWwithanefciency ofabout25%andatotaldownloadof12MBhasbeenachievedsof arusing13 satellitesforaperiodof5years.Asonecansee,communicati onsistheprimary bottleneckforemergingremotesensingmissions.Inordert oimprovethedownlink speedwedevelopedCubeSatTorrent,adistributedcommunica tionsframeworkfor 24

PAGE 25

CubeSatclusters.Thisproposalenvisionsthenextgenerati onofCubeSatNetworks, whichisthedistributedsatellitesystem. Table2-1.CubeSatdataspeedsanddownloads ParameterMinMaxAverage Speed1200bps38.4kbps9600bps Power350mW1500mW500mW Frequency433MHz900MHzNA TotalDownload320KB6.77MB0.5-5MB 2.3DistributedSatelliteSystems Adistributedsystemisacollectionofindependentcompone ntsthatworktogether toperformadesiredtaskandappearstoenduserasasingleco herentsystem. ExamplesofdistributedSystemsincludetheWorldWideWeb(WWW),C lusters, NetworkofWorkstationsorEmbeddedSystems,Cellprocessore tc.,Thesedistributed systemsarefueledbytheavailabilityofpowerfulandlowco stmicroprocessorsand highspeedcommunicationtechnologieslikeLocalAreaNetwo rk(LAN).Astheprice toperformanceratioofmicroprocessorsdropandspeedofco mmunicationnetworks increase,distributedcomputingsystemshavemuchbetterp rice-performanceratiothan asinglelargecentralizedsystem. AsmoreandmoreCubeSatsarelaunched,itisbecomingapparent thatsome spaceresearchneedsmaybebettermetbyagroupofsmallsate llites,ratherthanby asinglelargesatellite.Thisisakintotheparadigmshiftt hathappenedinthecomputer industryafewdecadesago:shiftoffocusfromlarge,expens ivemainframestousing smaller,cheaper,moreadaptablesetsofdistributedcompu tersforsolvingchallenging problems[ 16 ]. Distributedsatellitesystemshavetheirownadvantagesan dchallenges.Duetothe advancesinmodernVLSItechnologythatcreateintegratedcir cuitswithlowerpower andsmallerinsize,andduetosubsystemslikeRelNAVSoftwar eDenedRadio[ 17 ] thathaveenabledhighspeedsatellitecommunication,dist ributedsatellitesystems 25

PAGE 26

potentiallyhaveamuchbetterpricetoperformanceratioth anasinglelargemonolithic satellite.Applicationslikeweathermonitoringandtracki ngareinherentlydistributed innatureandmaybebetterservedbyadistributedsystemtha nacentralizedsystem. Monolithicsatellitearchitecturerequiresthateachsate llitemusthaveallofthesensing, processing,storageandcommunicationperipheralsonboar d.Distributedsatellite systemscanshareresourceslikesensing,memory,processi ngandcommunications, aswellasinformation.Themultiplicityofsensors,storag edevices,processorsand communicationdevicesmeansthereisnosinglepointoffail ure.Criticalinformation canbeduplicatedallowingthesystemtocontinuetoworkeve nifsomecomponents fail.Similarly,distributedsatellitesystemsmayhavebet teravailabilitythancentralized satellitesystems,duetotheabilityofthesystemtoworkat reducedcapacitywhen componentsfail. Finally,distributedsatellitesystemsenjoytheadvantag eofincrementalgrowth. Thefunctionalityofadistributedsatellitesystemcanbeg raduallyincreasebyadding moresatellitesasandwhenneedarises.Distributedsmalls atellitesystemsrelyonwhat iscalledhorizontalscaling,whereoneemploysmoresatell itestoserveanincreased need. Onotherhand,distributedsatellitesystemsaremorecompl exanddifcultto buildthanmonolithicsatellitesystems.Severalchallenge ssuchasorbitplanning, resourcemanagement,communicationanddatamanagement,s ecuritymustbe addressed[ 16 ].Thereisverylittleornosupportfordistributeddatasto rage,processing orcommunicationsfordistributedsatellitesystems.Dist ributedsatellitesystemsneed fastandlowpowerbackbonenetworkfordataandcontrolinfo rmationexchange.The backbonenetworkmustbereliableandshouldpreventproble mssuchasmessage loss,overloadingandsaturation.Distributedsystemssto redataatseveralplaces.It providesmoreaccesspointsforcriticalinformation.Asare sult,additionalsecurity measuresneedtobetakentosafeguarddataandsystems.Fina lly,ndingoutproblems 26

PAGE 27

indistributedsatellitesystemsandtroubleshootingthem requiresdetailedanalysisof eachsatelliteandcommunicationbetweenthem. 2.4ClassicationofDistributedSatelliteSystems Constellation,FormationFlyingandSwarm/Clusterarethre emaintypesof distributedsatellitesystems[ 16 ].Agroupofsatellitesinsimilarorbitswithcoordinated groundcoveragecomplementingeachotheriscalledaConste llation.Satellitesina constellationdonothaveon-boardcontroloftheirrelativ epositionsandarecontrolled separatelyfromgroundcontrolstations.IridiumandTeled esicarewellknownexamples ofsatelliteconstellations.Agroupofsatelliteswithcoo rdinatedmotioncontrol,based ontheirrelativepositions,topreservethetopologyiscal ledaFlyingFormation.Position ofasatelliteinayingformationiscontrolledbyonboardc losed-loopmechanism. Satellitesofayingformationworktogethertoperformthef unctionofasingle,large, virtualinstrument.TICS,F6andOrbitalExpressarewellkno wnexamplesofying formations.Agroupofsatellites,withoutxedabsoluteor relativepositions,working togethertoachieveajointgoaliscalledaClusterorSwarm.M oreaboutsatellite clustersispresentedintheChapter3. 2.5RelatedWork 2.5.1DistributedStorageSystems Belowwepresentanoverviewofrelatedworkdoneintheeldso fdistributed storage,processingandcommunications.Wesurveyedsomew ellknowndistributedle systemssuchasTheGoogleFileSystem(GFS)[ 18 ],HadoopDistributedFileSystem (HDFS)[ 19 ],Coda[ 20 ]andLustre[ 21 ].Owingtosimplicity,faulttolerantdesignand scalability,architectureofGFSandHDFSsuitswellfordis tributedstorageonCubeSat clusters.Belowwepresentanoverviewofdistributedstorag esystems,GFSandHDFS. GoogleFileSystemisthemajorstorageengineforlargescale dataatGoogle.A briefsummaryofGFSisasfollows.ArchitectureofGoogleFil eSystemisshowninthe Figure 2-3 .GFSconsistsoftwocomponents:amasterandoneormorechun kservers. 27

PAGE 28

GFSfunctionssimilarlytoastandardPOSIXlelibrarybutisn otPOSIXcompatible. Eachleissplitintoxedsizeblockscalledchunks.Eachchun kis64MBinsize,by default.Chunksarestoredonchunkservers.Metadatainfor mationlikeconstituent chunksofale,letochunkmapping,chunktochunkserverma ppingisstoredwith master.Chunkserversstoretheactualdataintheformofchu nks.Whenclientsinteract withGooglelesystem,largeshareofcommunicationwillbe betweentheclientand chunkservers.Thisavoidsmasterasthebottleneckfortran sferringlargelesinandout ofGoogleFileSystem. ClientmachinecommunicatestotheGoogleFileSystemthroug htheclientclass library.Ittranslatestheopen,read,writeanddeleteles ystemcallsintoGoogleFile Systemcalls.Clientlibrarycommunicateswiththemasterfo rmetadataoperationsand chunkserversforactualdataoperations.TheinterfaceofG FSclientisverysimilarto thatofPOSIXlesystem.ToworkwithGFS,noknowledgeaboutdi stributedsystems isrequired.GFSclientabstractsalltherequireddistribu tedknowledge.However,some localisedchunkinformationisusedforschedulingMapRedu cejobsonnodestoimprove theefciency.Eachoftheopen,read,writeanddeleteoperat ionsareimplementedin thefollowingway.GFSclientrequestsmasterformetadatai nformationincludingleto chunkmapping,chunktochunkservermappingandcommunicat eswithchunkservers foractualdata.Oncetheoperationiscompletemetadataoft heMasterisupdatedto reectthenewstateofthelesystem. HadoopDistributedFileSystem(HDFS)isanopen-sourceimple mentationof GoogleFileSystem.HadoopiswritteninJavaprogramminglan guage.Itcanbe interfacedwithC++,Python,Rubyandmanyotherprogramming languagesusingits Thrift[ 22 ]interface.Itisdesignedforstoringhundredsofgigabyte sorevenpetabytes ofdataandforfaststreamingaccesstotheapplicationdata .SimilartoGFS,HDFS supportswrite-once-read-manysemanticsonles. 28

PAGE 29

Figure2-3.ArchitecturaloverviewoftheGoogleFileSystem29

PAGE 30

HDFSalsousesmaster/slavearchitecture.ArchitectureofH adoopDistributedFile SystemisshownintheFigure 2-4 .InHDFS,NameNodeplaystheroleofthemaster nodeofGFS.Itcontrolsthenamespaceandimplementstheacc esscontrolmechanism fordatastoredinHDFS.DataNodestakescareofmanagingthe localstoragehardware. Inordertobringthecostoftheimplementationdown,thesen odesrunopensource operatingsystem,typicallyaGNU/Linuxsystem.Whenaleco piedintoHDFS,itissplit intoblocksanddistributedtoDataNodes.Faulttoleranceo fthestoreddataisachieved throughreplication.Eachchunkisreplicated3times,bydef ault.HDFSdocumentation isavailableattheirwebsite[ 23 ]. ThereareseverallimitationsofGFSandHDFSforstoringrem otesensingdata onCubeSatclusters.Unlikewiredcommunicationchannels,w irelesscommunication channelsinspacearemoreunreliable.Communicationlinks breakoftenleadingto networkpartitions.GFSandHDFSarenotpartitiontolerant .GFSandHDFSare notoptimizedforpowerconsumption.ForCubeSatclusters,p owerisaveryscarce resource.Also,costofcommunicationissignicantlyhighf orwirelesslinks,compared towiredlinks.GFSandHDFSaredesignedasgenericdatastor ageplatforms.They arenottailoredforstoringremotesensingdata.Theirgene ricdesignistoocomplexfor CubeSatClusters.UsingGFSorHDFScausesalotofoverheadin termsofprocessing, memory,bandwidthandpower.WedesignedCubeSatDistribute dFileSystemto overcometheabovementionedproblemsandtailoreditforst oringremotesensingdata onCubeSatclustersbyusinglargechunksizesandloadbalanc ing. 2.5.2DistributedComputingTechniques Distributedcomputingisaformofcomputingwhereprocessi ngisperformed simultaneouslyonmanynodes.Thekeyprinciplebehinddist ributedcomputingisthat mostofthelargeproblemscanbedividedintosmallerproble ms,whichcanbesolved concurrently.Cheapcomputingnodesareconnectedusinghi ghspeedbackbone networktoformaclustertoexecutethesmallerproblems.We surveyeddistributed 30

PAGE 31

Figure2-4.ArchitecturaloverviewoftheHadoopDistribute dFileSystem31

PAGE 32

computingtechniquessuchasCommonObjectRequestBrokerArc hitecture(COBRA), Webservices,RemoteProcedureCall(RPC),RemoteMethodInvo cation(RMI)and MapReducethatareusedfordistributedprocessingoncompu tingmachines.Belowwe presentabriefoverviewofthem. Distributedobjectstechniqueinvolvesdistributedobjec tscommunicatingvia messages.CommonObjectRequestBrokerArchitecture(COBRA),J AVARemote MethodInvocation(RMI),IBMWebsphereMQ,Apple'sNSProxy,Gnu step,Microsoft's DistributedComponentObjectModel(DCOM)and.Netarewell knownexamplesof thismodel.Owingtoitsplatformindependenceandinterope rablenature,CORBA programscanworktogetherregardlessoftheprogrammingla nguagesused.JavaRMI, IBMWebsphereMQ,Apple'sNSProxy,Microsoft'sDCOMand.Netare proprietary technologies.Theyarenotindependentoftheprogrammingl anguageandarenotquite versatile. Webservicesisthewaythroughwhichwebbasedapplications operatevia theHTTPprotocol.WebservicesusesSimpleObjectAccessProto col(SOAP)for exchangingstructuredinformationbetweenthewebcompone nts;JavaScriptObject Notation(JSON)toexchangedatabetweenwebservicesinhuma nreadableformat, WebServicesDescriptionLanguage(WSDL)toprovideamachiner eadable(machine-readable) descriptionofawebserviceandUniversalDescription,Dis coveryandIntegration (UDDI)fordescribingwebservices. MessagePassingInterface(MPI),OpenMessagePassingInter face(OpenMPI), OpenMultiProcessing(OpenMP)andParallelVirtualMachine(PVM )aretheprevalent technologiesinmessagepassinginterfacecategory.These technologiesareusedin massivelyparallelapplicationsandsupercomputingwhend ataneedstobedistributed andcommunicatedefciently. Socketsisapopularoptionforclientserverbasedarchitect ures,likemailservers andwebservers.Theiravailabilityonanysystemequippedw ithTCP/IPstackmakesit 32

PAGE 33

anattractiveoption.Trafccaneasilybere-routedtodiff erentportsusingsecureshell (SSH),SecureSocketsLayer(SSL)orVirtualPrivateNetwork(VPN)conn ections. MapReduceistherecentdevelopmentintheeldofdistribut edcomputing. IntroducedbyGoogleInc.in2004[ 18 ],designsimplicity,faulttoleranceandeaseof implementationmakesitanattractivecandidateforlarges caledistributedprocessing. Itisabasedonthemapandreduceprimitivesoffunctionalla nguageslikeLisp. MapReduceprogramsarehighlyparallelizableandthuscanb eusedforlarge-scale dataprocessingbyemployinglargeclusterofcomputingnod es.CompanieslikeGoogle, Yahoo!andFacebookuseMapReducetoprocessmanyterabytes ofdataonalarge clustercontainingthousandsofcheapcomputingmachines. MapReduceperforms large-scaledistributedcomputationwhilehidingthecomp licationsofparallelization,data distribution,synchronization,locking,loadbalancinga ndfaulttolerance. Westudiedindetailabouttheadvantagesanddisadvantages oftheabove mentioneddistributedcomputingtechniques.Theydonotac countforsalientandunique featuresofCubeSatsandCubeSatclusterslikepower,memorya ndcommunications constraints,unreliablewirelesscommunication,highcos tofcommunicationandneed fortightlocalityoptimizationofdatastorageandoperati ons.Owingtoitssimplicity andfaulttolerantdesign,MapReducesuitswellforlargesc aledistributedprocessing ofremotesensingdataonCubeSatclusters.BasedonMapReduce ,wedesigned CubeSatMapMergetoservetheneedsofCubeSatcommunity.Toov ercometheabove mentionedlimitations,wedesignedCubeSatMapMergeandtai loreditforprocessing remotesensingdataonCubeSatclusters. 33

PAGE 34

CHAPTER3 NETWORKARCHITECTUREOFCUBESATCLOUD ArchitectureofCubeSatNetworkisshowninFigure 3-1 .CubeSatNetwork consistsofspaceandgroundsegments.SpacesegmentisaCube SatCluster. ArchitectureofCubeSatClusterisshowninFigure 3-2 .ACubeSatclusterhasa radiusofabout25km.ItconsistsofSensornodesandWorkerno desinter-connected usinghighspeedcommunicationlinks.WorkernodesareCube Satswithstorage, processing,communicationandotherstandardsubsystems. Inadditiontothestandard subsystems,SensorCubeSathassensingsubsystem.Sensornode sactasMasterof theclusterwhileorchestratingremotesensingmissions.G roundsegmentiscomposed ofServerandseveralgroundstations.CubeSattoCubeSatcommu nicationlinksare shortdistance,reliable,directional,lowpowerandhighs peed.CubeSatstoground stationcommunicationlinksarelongdistance,highpower, lowspeedandunreliable. EachCubeSatisconnectedtoagroundstation.Groundstations areconnectedtothe ServerviatheInternet.Groundstationsactasrelaysbetwee nServerandCubeSats. Figure3-1.ArchitectureofaCubeSatnetwork 34

PAGE 35

Figure3-2.ArchitectureofaCubeSatcluster 3.1ComponentsoftheCubeSatNetwork 3.1.1SpaceSegment AworkerCubeSatisatypical1UCubeSatthathasdimensionsof1 0cmx10cm x10cm,volumeofexactlyonelitre,weighsaboutonekilogra m.However,itdoesnot needtobea1UCubeSatonly.Itcanbe2Uor3Uoranyotherformfa ctor.Worker CubeSatsneedstohavestorage,processingandcommunicatio nsubsystems.Other standardsubsystemsincludeSatelliteBus,ElectricalPower, StructuralandThermal, AttitudeDeterminationandControl.WorkerCubeSathasabout 1GHzprocessor,1 GBofRAM,32-64GBmemory.CubeSattogroundstationcommunica tionspeedis about9.6kbps.SensorCubeSatshassensingmoduleinaddition toabovementioned subsystens.Figure 3-3 showsablownupESTCube-ICubeSatshowingitsvarious subsystems. Sensornodeisequippedwithsensinghardware.Itperformsse nsing(takean imageordoaradarscan).Whileorchestratingamission,aSens ornodeactsasthe MasternodeforCubeSatCluster.Whennotorchestratingamiss ion,Sensornode performstheroleofaworkernode.Masternodeistheprimary centerforreceiving 35

PAGE 36

Figure3-3.AblownuppictureofESTCube-ICubeSat,showingits subsystems ImagecourtesyofUniversityofTartu.PicturebyAndreasVald mann. 36

PAGE 37

commandsfromtheserverandissuingsubcommandstoworkerC ubeSatsinthe cluster.Itkeepstrackofallthemetadatarelatedtothemis sionincludingthelistof participatingnodes,theirresourcecapabilities,mapjob s,mergejobs,downlinkjobs andtheirstatus.Itkeepstrackofalltheresourcesavailab leinthecluster,theirstate andtracksavailableresourcesoneachnode.Itisalsorespo nsiblefortakingscheduling decisionslikewhichjobneedstobescheduledonwhichnodea ndwhen.Workernodes havelimitedroleofexecutingtheprocessinganddownlinki ngjobsassignedtothemby theMasternode.3.1.2GroundSegment Groundstationoramateurradiostationisaninstallationt hatenablescommunication withCubeSatsatellite.Figure 3-4 showsgroundstationcontrolequipmentandFigure 3-5 showshighdirectionalYagiantennausedforcommunicating withsatellites.It containshighgaindirectionalantennaslikeYagiorparabo licdishantenna,communication equipmentlikemodemsandcomputerstosend,captureandana lysethedatareceived. Thereareseveraltypesofamateurradiostationsincluding xedgroundstations, mobilestations,spacestations,andtemporaryeldstatio ns.Mostoftheradiostations areestablishedtoprovideaneducationalandrecreational purposesforproviding technicalexpertise,skillsandvolunteermanningtopromo teattendancebythepublic, communicationseducationforthepublic. Groundstationserverisadedicatedcomputersystemthatis connectedtoground stationsthroughInternet.ItreceivescommandsfromAdmini stratoranduplinksthe commandstoCubeSats.Oncethemissionisexecuted,resultin gdataisdownlinkedto theServerandisstoredoitslocalstoragedisk.Serveractsas thecommandcenterfor CubeSatnetwork.AdministratorissuescommandstotheServer, whichthenforwards thecommandstotheMasterCubeSatthroughagroundstation.Se rvernodestoresall thedownlinkedmissiondataandthusactsasthestoragenode forthedownlinkeddata fromCubeSatcluster.GroundStationsactsasarelaybetweenC ubeSatsandserver. 37

PAGE 38

Figure3-4.GroundstationImagecourtesyofGatorAmateurRadioClub.ImagebyTzuYu(Ji mmy)Lin. TheydownlinkthedatafromCubeSatandsendittoserver.They uploadcommands anddatafromservertoworkerCubeSatswhichforwardthemtoM asterCubeSat. 3.2SystemCommunication 3.2.1ClusterCommunication CubeSatsareconnectedtoeachotherthroughahighspeed( > 1Mbps)and lowpowerconsumingbackbonenetwork.Highgaindirectedan tennaslikepatch orLASER[ 24 ]areusedforinterclustercommunication.Vescentphotoni cs[ 25 ]is developingaextremelysmallandlowpoweropticalcommunic ationsmodulesfor CubeSats.Therehasbeenresearchonusingtethersforlowdis tance,highspeed communicationbetweensatellites.RelNavdemonstratedas pacecraftsubsystemcall 38

PAGE 39

Figure3-5.GroundstationantennaImagecourtesyofGatorAmateurRadioClub.ImagebyTzuYu(Ji mmy)Lin. SWIFTSoftwareDenedRadio(SDR)[ 26 ][ 17 ]thatwillenableaockofsatellites. SWIFTSDRsubsystemdemonstratedbyRelNavprovidesprovidefo llowingservices: • 1Mbpsinter-satellitecommunicationlinkfordataexchang ebetweenCubeSats. • Relativepositionandorientationforformationight. • Clustersynchronizationandtimingforcoordinatedoperat ionsandcoherent sensing. 3.2.2SpaceSegmenttoGroundSegmentCommunication CubeSatgeometryprohibitstheuseofcomplexantennas[ 27 ].Asaresult, CubeSatsareconnectedtogroundstationsthroughsimpleant ennaslikemonopole ordipole.Coupledwithstringentpowerconstraintsanddis tancesoforder600-800km, thisresultedinlowspeedlinksbetweenCubeSatsandgrounds tations.TypicalCubeSat togroundstationspeedisabout9.6kbps[ 10 ]. 39

PAGE 40

3.2.3GroundSegmentNetworkCommunication GroundstationsandServerareconnectedviatheInternet.In ternetprovidesahigh speed(10Mbps)andreliablewiredcommunicationmediumbet weentheServerand groundstations.PowerisnotaconstraintfortheServerandg roundstationsastheyare connectedtotheelectricalgrid. 3.3CubeSatCloud WeproposeCubeSatCloud,aframeworkfordistributedstorag e,processingand communicationofremotesensingdataonCubeSatClusters.Cu beSatClouduses CubeSatDistributedFileSystemfordistributedstorageofre motesensingdataon CubeSatClusters.CubeSatMapMergeisthedistributedproces singframeworkused forprocessingremotesensingdatastoredinCDFS.CubeSatTo rrentisthedistributed communicationsframeworkusedfordownlinkingraworparti allyprocessedremote sensingdatafromCubeSatClusters.Belowwedescribehowremo tesensingmissions areexecutedusingtheCubeSatCloudframework.3.3.1Storage,ProcessingandCommunicationofRemoteSensing Dataon CubeSatClusters CubeSatCloud,asagenericframework,canbeusedforstoring ,processing anddownlinkofremotesensingdatafromCubeSatClusters.On cearemotesensing operationisperformed,obtainedsensordataisstoredonth eCubeSatclusterusingthe CubeSatDistributedFileSystem.Afterstoringthedataonthec luster,itisprocessed usingCubeSatMapMergeandobtainedresultsaredownlinkedu singCubeSatTorrent. Figure 3-6 showstheoverviewofCubeSatCloudframeworkconsistingofC ubeSat DistributedFileSystem,CubeSatMapMergeandCubeSatTorrent .Belowisadetailed descriptionofhowaremotesensingmissionisexecutedusin gCubeSatCloud. 1. ServersendstheSENSEandSTOREcommandtotheMaster.Uponreceiv ingthe SENSEandSTOREcommand,Masterperformsremotesensingoperati onand storesthesensordataonthelocallesystem.ServerandMast erdoesnotneed tohaveadirectcommunicationlink.Thecommandwillberela yedthroughthe groundstationnetworkandspacesegment. 40

PAGE 41

2. MasternodethensplitstheleintochunksC1,C2,C3,...Cn. Sizeofchunks isabout64Kb.MasternodedistributestheChunkstotheworke rnodes.Fileto chunkmapping,Chunktoworkermappingandothermetadatais storedMaster. Splittingtheremotesensingdataintochunks,distributing themandstoringthem onworkernodesisachievedusingCubeSatDistributedFileSys tem.Distributing thedataacrosstheworkernodesintheclusterallowsthepos sibilityofprocessing anddownlinkingthedataindistributedfashion. 3. ServersendsthePROCESScommandtoMaster.MasterCubeSatcomman dsthe WorkerCubeSatstoprocessesthestoredchunksstoredtoprod ucepartialresults. Obtainedpartialresultsarestoredonthelocallesystemo fworkernodes. 4. ServersendstheDOWNLINKcommandtotheMaster,whichthencom mandsthe workernodesintheclustertodownlinktheprocessedchunks togroundstations. DownlinkingtheprocessedchunkstotheServerisachievedth roughCubeSat Torrent. 5. OncetheServerreceivesalltheprocessedchunks,itstitche sthemintofull solution.Processingofchunkstoproducepartialresultson Workernodesand stitchingofpartialresultsintothecompletesolutiononSe rverconstitutesthe CubeSatMapMerge. 3.3.2SourceCoding,StoringandDownlinkingofRemoteSensing Dataon CubeSatClusters Alargenumberofmissionsrequireonlydownlinkingofthere motesensingdata withoutprocessingit.Forthesemissions,workernodesdoe snotrequireaccesstothe rawdata.ThiscanbeusedasanopportunitytooptimizetheCu beSatTorrentmissions byimprovingthequalityofserviceandreducingthestorage overhead.Belowwe presentindetailabouthowweutilizesourcecodingforimpr ovingthequalityofservice andreducestorageoverhead.Figure 3-7 showstheoverviewofhowadownlinkonly remotesensingmissionsareexecutedusingCubeSatCloudand isdescribedbelowin detail. 1. MastersendstheSENSE,CODEandSTOREcommandtotheServer. 2. UponreceivingtheSENSE,CODEandSTOREcommandfromtheServer,Ma ster performsremotesensingoperationandstoresdatafromthes ensoronthelocal lesystem. 41

PAGE 42

Figure3-6.OverviewofCubeSatCloudanditscomponentframe works42

PAGE 43

3. MasternodesplitstheremotesensingdataleintochunksC1 ,C2,C3...Cn.Size ofeachchunkisabout64Kb.Then,basedontherequiredredund ancy,itcreates codedchunksC1',C2',C3'...Cm,wherem > n. 4. MasterdistributescodedchunksC1',C2',C3'...CmtotheWo rkernodes.Master storesmetadata,whichincludesletochunkmapping,chunk toWorkermapping andchunkstatus.Splittingtheremotesensingdataintochun ks,performing coding,distributingthemandstoringthemonWorkernodesi sperformedby CubeSatDistributedFileSystem. 5. ServerthensendstheDOWNLINKcommandtotheMaster,whichthe ncommands theWorkernodesintheclustertodownlinktheprocessedchu nkstoGround stations.DownlinkingtheprocessedchunkstotheServerisp erformedby CubeSatTorrent. 6. OncetheServerreceivesnoutofmchunks,itstitchesthemint otheoriginalle. Aslongasnoutmchunksareavailable,theoriginaldatacanst illberecovered. Detailsaboutperformanceanalysisofsourcecodingaredis cussedinChapter8. 43

PAGE 44

Figure3-7.CubeSatCloud:IntegrationofCubeSatDistribute dFileSystemandCubeSatTorrent44

PAGE 45

CHAPTER4 DISTRIBUTEDSTORAGEOFREMOTESENSINGIMAGESONCUBESATCLUSTERS TheCubeSatDistributedFileSystem(CDFS)isbuiltforstoring largesizedremote sensinglesonsmallsatelliteclustersinadistributedfa shion.Whilesatisfyingthe goalsofscalability,reliabilityandperformance,CDFSis designedforCubeSatclusters whichusewirelessbackbonenetwork,arepartitionpronean dhaveseverepower andbandwidthconstraints.CDFShassuccessfullymetthesc alability,performance andreliabilitygoalswhileadheringtotheconstraintspos edbytheharshenvironment andlimitedresources.Itisbeingusedasastoragelayerfor distributedprocessing anddistributedcommunicationonCubeSatclusters.Inthisc hapter,wepresentthe architecture,lesystemdesignandseveraloptimizations 4.1KeyDesignPoints 4.1.1NeedforSimpleDesign AtypicalCubeSathasabout1GHzprocessingcapability,1GBR AM,32GBof ashstorage,1Mbpsinterclustercommunicationspeed,9.6 kbpscommunication capabilityand2Wpowergenerationcapability[ 5 ].ForCubeSats,processing,bandwidth andbatterypowerarescarceresources.Sothesystemdesignn eedstobesimple. 4.1.2LowBandwidthOperation CubeSatnetworkisbuiltusinglongdistancewirelesslinks( 10kmforintercluster and600kmforCubeSattogroundstation).Asaresult,thecosto fcommunicationis veryhigh.Asaresult,dataandcontroltrafcneedstoberedu cedasmuchaspossible. 4.1.3NetworkPartitionTolerant Thebackbonemediumofcommunicationiswirelessandthespa ceenvironmentis harsh.Highvelocityofsatellites(relativetogroundstat ions)inLEOmakesthesatellite togroundstationlinkfailureverycommon.TopologyofCube Satclusterisalsovery dynamic,causingtheintersatellitelinkstokeepbreaking veryfrequently.Sometimes, nodesgointosleepmodetoconservepower.Alltheabovefacto rscancausefrequent 45

PAGE 46

breakingofcommunicationlinks.Asaresult,ifanodeistemp orarilyunreachable, systemshouldnottreatitasanodefailure.Systemshouldbet oleranttotemporary networkfailuresorpartitions.4.1.4Autonomous Mostofthetime,individualCubeSatsandthewholeCubeSatclu sterareinaccessible tohumanoperators.Sothesoftwaredesignshouldtakecareof allfailurescenarios. Aresetmechanism,atthenodeandnetworklevel,shouldbepr ovided.Incaseifall thefaulttolerancemechanismsfail,systemwillundergore setmechanismandstart workingagain.Asaresult,distributedlesystemshouldbea bletooperatecompletely autonomouslywithouthumanintervention.4.1.5DataIntegrity Memoryfailuresarefatalforsatellitemissions.Eventhoug hmemorychipsfor satellitesareradiationhardened,highenergycosmicrays cansometimescausetrouble. Forexample,theMarsroverCuriosityhadsufferedasignic antsetbackbecauseof damagetothememoryofitsprimarycomputercausedbyahighenergyparticle. Hence,dataintegrityshouldnotbeviolated. 4.2SharedGoalsBetweenCDFS,GFSandHDFS AlongwiththeabovedesignpointsthatCDFSsharesadditiona ldesignpointswith GFSandHDFSwhicharehighlightedbelow.4.2.1ComponentFailuresareNorm GivenalargenumberofCubeSatsandcommunicationlinks,fai luresarenorm ratherthantheexception.Therefore,constantmonitoring ,errordetection,fault tolerance,andautomaticrecoverymustbeintegraltothesy stem. 4.2.2SmallNumberofLargeFiles Filesarehugebytraditionalstandards.Imagesandremotes ensingdatagenerated bysatellitestendtobeintheorderofhundredsofmegabytes 46

PAGE 47

4.2.3ImmutableFilesandNon-existentRandomReadWrites Randomwriteswithinalearepracticallynon-existent.On cewritten,thelesare onlyread,andoftenonlysequentially.Thiskindofaccessp atternsarecommonfor imaging,remotesensingmissionsandprogramslikeMapRedu cethatprocessthisdata andgeneratenewdata. CDFSsharesgoalsofavailability,performance,scalabili tyandreliabilitywithGFS andHDFS.Owingtoitsradicallydifferentoperatingenviro nment,thesystemdesign pointsandconstraintsareverydifferentforCDFS.GFSandH DFSweredesignedfor non-powerconstrainedclusterofcomputersconnectedusin ghighspeedwiredmedia. CDFSismeantfordistributeddatastorageonCubeSatcluster swhichusewireless communicationmediumforexchangingdataandhaveseverepo werandbandwidth constraints.DesignofCDFSshouldbesimple,operatewithv erylessbandwidth consumptionandoperateautonomouslywithouttherequirem entforhumanintervention. Itshouldbetoleranttonetworkpartitions,temporarylink failures,nodefailuresand preservetheintegrityofthedatastored. 4.3ArchitectureofCubeSatDistributedFileSystem Figure 4-1 ShowsthearchitectureofCDFS.ACDFSclusterconsistsofSens or CubeSatsandworkerCubeSats.Sensornodesareequippedwithse nsingmoduleand thusperformssensing.Whileorchestratingamission,aSenso rnodeplaystheroleof theMaster(M)node.WorkernodesaidtheMasternodeinproce ssingordownlinking largeles.HereishowCDFSstoresaleonthecluster.Admini stratorwillissuea remotesensingcommandtothecentralserver(asshowninFig ure 4-1 ).Centralserver willtransmitsthecommandtoarelaygroundstationwhichup linksittotheMaster CubeSat.Uponreceivingthecommand,MasterCubeSatwillperf ormsensing,like takingimageordoingaradarscan.Sensingoperationwillgen eratelargeamountsof data(about100MB)whichisstoredintoalocalleonMasterno de. 47

PAGE 48

Figure4-1.ArchitectureofCubeSatDistributedFileSystem48

PAGE 49

Masternode(M)splitsthisleintoblockscalledchunksand storesthemonworker CubeSats.Eachchunkisidentiedusinguniquechunkid.Forre liability,eachchunkis replicatedonmultipleworkers.Bydefault,CDFScreatestwo replicas(aprimaryreplica andasecondaryreplica),alongwithanimplicitreplicasto redonMasternode.Soin effect,therearethreereplicas.Alongwiththeimplicitrep licas,theMasterCubeSat holdsallmetadataforthelesystem.Metadataincludesthe mappingfromlesto chunks,thelocationofthesechunksonvariousworkers,the namespaceandaccess controlinformation.Theworkersstoretheactualdata.Wor kernodesstorechunksas regularlesonlocalashmemory.Asshowninthegure,thecl usterisorganizedasa treewiththeMasternodeastherootnode.4.3.1FileSystemNamespace CDFSsupportsatraditionalhierarchicalleorganization inwhichauseror anapplicationcancreatedirectoriesandstorelesinside them.Thelesystem namespacehierarchyissimilartothatofLinuxlesystems[ 28 ].Therootdirectory is“/”andisemptybydefault.Onecancreate,rename,reloca te,andremoveles. CDFSsupportshiddenlesanddirectoriesconceptinawaysi milartothatofLinuxle systems.Hiddenleordirectoriesstartwith“.”(period)a ndcontainmetadata,error detectionandcorrectioninformation,congurationinfor mationandothermiscellaneous informationrequiredbyCDFS.Thesehiddenlesarestoreda sregularlesratherthan distributedlessincetheseareverysmallandareusedbyth esystemlocally.Onecan refertothelesstoredonaserverusingnotation“cdfs://s erver/lepath”,wherelepath lookslike“/directory1/directory2/.../lename”.4.3.2Heartbeats Severalproblemscancauselossofdataorconnectivitybetwe enMasterandworker nodes.ProblemsarediagnosedusingHeartbeatmessages.Onc eevery10minutes, workernodessendaHeartbeatmessagetoMasternode.Heartb eatmessagecontains workerscurrentstatusandproblems,ifany.Masterperiodi callydiagnosesthereceived 49

PAGE 50

Heartbeatmessagestodetectanyproblemsandrectifythemi fpossible.Ifaworker doesnotsendanyHeartbeatmessagewithin10minutes,Maste rwillmarkthenodeas temporaryfailure.IfthereisnoHeartbeatmessagefromwor kerwithin30minutesof time,Mastermarkstheworkeraspermanentfailure.Whenanod eismarkedtemporary failure,datachunksassignedtoworkerwillnotbereplicat edonothernodes,instead thesecondaryreplicaswillbemarkedasprimary.Afterperma nentfailure,Master nodemarksthenodeasdead.WhenanodecontactstheMasteraft errecoveringfrom permanentfailure,Masterrefreshesitmetadatatoreectt hechange. 4.4FileOperations CDFShasasimpleinterface.CDFSsupportsleoperationscr eate,write,readand delete.Nextsectionwedescribeindetailwhathappenswhen eachoftheseoperations isperformed.4.4.1CreateaFile OnceMasterperformsremotesensingoperation(takeanimag eordoaradar scan),itgenerateshugeamountsofsensordata.Initiallyt hisdatawillbestoredinto alocalleontheMasternode.Typicalsizeofthisleisabou t100MB.Masterstores thisleonCubeSatclusterusingCDFStoperformdistributed processingordistributed downlinking.Followingactionswillimplementedinsequen cewhenaCDFSleis createdbyMasternode.Itrequireslename,andchunksizea sparameters.Bydefault, chunksizeis64KBandisoptional. 1. Mastercalculatesthenumberofchunksbasedonthelesizea ndchunksize. (Numberofchunks=lesize/chunksize). 2. Mastergenerateschunkidentierscalledchunkidsandassi gnsonetoeach chunk.Chunkidisanimmutableid. 3. Masterassignsthechunkstoworkernodes.Eachchunkisassig nedtooneworker nodeinaroundrobinfashion.Acopyofthechunkstoredatthe selectednodeis calledprimaryreplicaofthechunk. 50

PAGE 51

4. Masterstorestheabovemetadata(lename,numberofchunks ,chunktochunk idmappingandchunkidtoworkernodemapping)initspermane ntstorageand communicatesthesametothebackupMaster. 4.4.2WritingtoaFile WriteoperationisperformedbyMasternodewhenitwantstoco pyalocalle(on theMaster)toCDFS.FilesinCDFSareimmutable.Theycanbew rittenonlyonce afterthatarecreated.Inputsforawritingalearesource lenameonMasterandthe destinationlenameCDFS.Followingactionshappeninsequ encewhentheMaster writesalocalletoaCDFSle. 1. ForeachchunkMasterperformstheactionsdescribedinstep s2,3,4,and5. 2. MasterlooksupthemetadataofthedestinationleonCDFSto ndouttheworker noderesponsibleforstoringthechunk. 3. Masterdeterminestransmissionpath(fromMastertothewor kernode)usingtree basedroutingalgorithm. 4. Fromthenodesonthetransmissionpath(excludingtheMaste randdestination workernode), 5. Masterrandomlypicksanodetobethesecondaryreplicaofth echunkand notiesit. 6. Mastertransmitsthechunktotheprimaryreplicanode.While thechunkisbeing transmittedtotheprimaryreplicanode,secondaryreplica nodecopiesthechunk andstoresinitslocalstorage. 7. Afterstoringallthechunksonthecluster,Mastercommitsth emetadatatoits memoryandcommunicatesthesametotheServer. 4.4.3DeletingaFile Thefollowingactionsareperformedinsequencewhenaleis deleted. 1. AdministratorissuesdeletelecommandtotheServer. 2. ServeruplinksthecommandtotheMasterCubeSatthrougharela yground station. 3. Masternodelooksupthemetadatafortheleandsendsthedel etechunk commandtoalltheprimaryandsecondaryreplicasnodes. 51

PAGE 52

4. Onceaworkerdeletesthechunks,itsendsACKtotheMaster. 5. OncetheACKsarereceivedfromallworkerCubeSats,Masterdel etesthe metadataforthele. 6. MasterCubeSatwillsendSUCCESSmessagetotheServerthroughrel ayground station. 4.5EnhancementsandOptimizations CDFSserveswellasdistributeddatastorageonCubeSatClust ers.However CubeSatshavestringentenergyconstraintsandCubeSatclust ershavesevere bandwidthconstraints.Sothereisadireneedtoreduceenerg yandbandwidth consumption.Belowwedescribethemethodsweemployforredu cingtheenergy andbandwidthconsumption.4.5.1BandwidthandEnergyEfcientReplication Toensurereliabilityofdatastored,CDFSusesredundancy. Eachchunkhasthree replicasstoredonthreedifferentnodes,calledreplicano des.But,creatingreplicas isbothenergyandbandwidthconsuming.ForaCubeSatcluster ,bothenergyand bandwidthareprecious.Inordertoreduceenergyandbandwi dth,Masternode(Source node)canbeusedastheSuperReplicaNode(SuperReplicaNode: Anodewhich storesthereplicasofallchunks).SincetheMasternodeperf ormssensingandhasall thedatainitially,implicitreplicasonMasternodearecre atedwithoutanyenergyand bandwidthconsumption.UsingMasternodeasaSuperReplican odeessentiallymeans thatCDFSneedstocreateonlytwoadditionalreplicas.This alsomeansthatMaster nodeshouldbeequippedwithsufcientlyhighstoragetosto reallchunks.Butthisisa smallcostcomparedenergyandbandwidthsaved.Thedatafro mthesourcenodeis accessedonlyifothertworeplicasarenotavailable,inord ertoconservethepowerof thesourcenode. Foradditionaltworeplicas,anyrandomselectionofworker nodeswilldoagood jobforachievingreliability.But,ifreplicanodesarecare fullyselected,energyand bandwidthconsumptioncanbesignicantlyreduced.Consid erthetwoscenarios 52

PAGE 53

AandBdepictedinFigure 4-2 .InscenarioA,thechunkisreplicatedonnodesM (theMasternode),AandB2.InscenarioB,thechunkisreplica tedonnodesM,B andB1.Thecostofcommunication(bandwidthandenergyconsu med)intherst scenariois3timestheaveragelinkcommunicationcost(fro mMtoAandfromMto BtoB2).Inthesecondcase,energyconsumptionisonly2times theaveragelink communicationcost(fromMtoBtoB1).Storingachunkonnodest hatareonthe samecommunicationpathoronnodeswhicharelocatedcloset oeachotheryields bestenergyandbandwidthefciency.Exploitingtheaboveob servation,wedesigneda novelmethodforprovidingreliabilitywithlowpowerandba ndwidthconsumption.This techniqueiscalledCopy-on-transmit. Figure4-2.Bandwidthandenergyefcientreplication Whenthesourcenodetransmitsthedatatoadestination,itgo esthroughmultiple hops.Selectednodes,onthecommunicationpath,copythedat awhileitisbeing transmitted.Thismethodisveryconvenientfordoingdatar eplicationinwireless networks,withoutincurringadditionalenergyorbandwidt hconsumption.Considerthe 53

PAGE 54

scenariosshowninFigure 4-3 .InallcasesthesourcenodeM,transmitsdatatothe destinationnodeZ.Belowwedescribehowwereplicatedataus ingcopy-on-transmitfor differentcommunicationpathlengthsforareplicationfac torof3(1implicitreplicaand2 explicitreplicas).4.5.1.1Numberofnodesoncommunicationpath=replication factor Inthiscase,wereplicatethechunkonallnodesalongthepat hincludingthesource anddestination.WhenthechunkisbeingtransmittedfromNod eMtoNodeZthrough NodeA,NodeAmakesacopyofthechunkandstoresinitsmemory. Nowthechunk hasthreereplicas,oneeachatM,AandZ.4.5.1.2Numberofnodesoncommunicationpath > replicationfactor Inthiscase,wereplicatethechunkonMasternodeM,destina tionnodeZand arandomonthepath.WhenthechunkisbeingtransmittedfromN odeMtoNodeZ throughNodesA,B,C,D,andE,NodeCmakesacopyofthechunkand storesitinits memory.NowthechunkhasthreereplicasoneeachatM,CandZ.4.5.1.3Numberofnodesoncommunicationpath < replicationfactor Inthiscase,wereplicatethedataonallnodesonthecommuni cationpath(Node MandNodeZ)andsomeadditionalnodes.Thisscenariocanhav etwodifferent sub-scenarios(a)whenthedestinationnodeisnotaleafnod e(haschildren)and (b)whenthedestinationisaleafnode(nochildren).Theset wosub-scenariosare discussedasCase3(a)andCase3(b)below. Case3(a)Destinationnodeisnotaleafnode: Inthiscase,rstwereplicatethedata onallnodes(NodeMandNodeZ),alongpath.Inordertomeetth ereplication requirement,thecommunicationpathisextendedbeyondthe destinationnodeZto storedataonNodeA.Thisensuresthattherearerequirednumb ersofreplicas. Case3(b)Destinationnodeisaleafnode: Inthiscase,rstwereplicatethedata onallnodes(NodeMandNodeZ),alongpath.Inordertomeetth ereplication requirement,onemorereplicaofchunkneedstobecreated.M asterrandomly 54

PAGE 55

selectsanothernodeAandstoreschunkonit,ensuringthatt herearerequired numberofreplicas. Figure4-3.Copyontransmit 4.5.2LoadBalancing Thegoalofloadbalancingistodistributedatatothenodesi ntheclusterinorder tobalanceoneorseveralofthecriterialikestorage,proce ssing,communication,power consumption.Whenaleiscreated,numberofchunksassigned toaworkernodeis proportionaltothevalueoftheLBF(node),whereLBFistheloa dbalancingfunction. Belowweexplainhowwedeterminetheloadbalancingfunction foruniformstorage, proportionalstorageandseveralothercriteria.Customlo adbalancingfunctioncan beusedtoperformloadbalancingaccordingtouserswish.Ho weveritsneedstobe notedthatdistributingdatainordertoperformuniformsto ragemightresultinuneven loadbalancingforprocessingorcommunicationandvicever sa.Nisthetotalnumberof 55

PAGE 56

workernodesintheclusterandLBFistheloadbalancingfunct ion.Thefollowingarethe availableloadbalancingfunctionsavailableinCDFS: • Uniformstorage/processing/communicationspernode:LBF( node)=1/N • Inproportiontostoragecapacityofnode:LBF(node)=Storage capacityofthe node/TotalstoragecapacityoftheCluster • Inproportiontoprocessingcapacityofnode:LBF(node)=Proc essingpowerof node/TotalprocessingpoweroftheCluster • Inproportiontocommunicationcapacityofnode:LBF(node)= Communication speedofnode/Totalcommunicationspeedofthecluster. • Inproportiontopowergenerationcapacityofnode:LBF(node )=Powergeneration capabilityofnode/Totalpowergenerationcapabilityofth ecluster • Hybrid:LBF(node)=a*LBF(node)forstorage+b*LBF(node)forp rocessing+c *LBF(node)forcommunication+d*LBF(node)forpower,wherea ,b,canddare normalizedproportioncoefcients,andsumofa,b,canddis 1. Formissionsthatareprocessingintensive,itisdesirable thatnumberofchunks storedonanodeisproportionaltothenodesprocessingpowe r.Forcommunication intensivemissions,itisdesirablethatnumberofchunksst oredonanodeisproportional tothecommunicationcapabilitiesofthenode.Formissions thatarebothprocessing andcommunication,hybridfunctioncanbeused.Additionall y,inordernottooverload nodes,acappingonthenumberofchunksstoredpernodeperl eissuggested. 4.5.3ChunkSizeandGranularity Bysplittinglesintoalargenumberofchunks,granularityw illbeimproved.Small chunksensurebetterstoragebalancing,especiallyforsma llles.However,asthe numberofchunksincreases,sotheamountofmetadata,metad ataoperationsand numberofcontrolmessageswhichdecreasesthesystemperfo rmance.Inorderto strikebalancebetweentheadvantagesoflargechunkswitha dvantagesofgranularity, weselectedchunksizetobeabout64KB. 56

PAGE 57

4.5.4FaultTolerance CDFSisdesignedtobetolerantfortemporaryandpermanentC ubeSatfailures anditsperformancedegradesgracefullywithcomponent,ma chineorlinkfailures.A CubeSatclustercancontainuptoaboutahundredCubeSatsanda reinterconnected withroughlysamenumberofhighspeedwirelesslinks.Becaus eofalargenumberof componentsandharshspaceenvironment,someCubeSatsorwir elesslinksmayface intermittentproblemsandsomemayfacefatalerrorsfromwh ichtheycannotrecover unlesshardresetbygroundstation.Sourceoftheproblemcan beinapplication, operatingsystem,memory,connectorsornetworking.Sofail uresshouldbetreatedasa normratherthananexception.Inordertoavoidsystemdownt imeorcorruptionofdata, systemshouldbedesignedtohandlethefailuresanditsperf ormanceshoulddegrade gracefullywithfailures.Belowwediscusshowwehandlethes eerrorswhentheycome up.4.5.5MasterFailure Masternodestoresmetadata,whichconsistsofmappingbetw eenthelesto chunksandchunkstoworkernodes.IftheMasternodefails,t hemissionwillfail.In ordertoavoidmissionfailureincaseofMasterfailure,met adataiswrittentoMasters non-volatilememory,likeash,andthesameiscommunicate dtotheServer.Ifthe Masterrebootsbecauseofatemporaryfailure,anewcopywil lbestartedfromthelast knownstatestoredinMastersnon-volatilememory.Incaseo ffailureofMaster,worker nodeswillwaituntilanewMasterresumes.4.5.6WorkerFailure WorkernodessendHeartbeatmessagestomasteronceevery10 minutes.Ifa workerreportsafatalerror,Mastermarkstheworkernodeas failed.Ifaworkerdoes notsendheartbeatmessagewithin10minutes,Masterwillma rkthenodeastemporary failure.IfMasterdoesnotreceiveheartbeatmessagewithi n30minutes,Mastermarks 57

PAGE 58

theworkerasfailed.Oncethefailednodecomesbackonline, Mastersmetadatawillbe refreshedtoaccountthechange.4.5.7ChunkCorruption Harshspaceenvironmentandthecosmicraysleadtofrequent memorycorruption. OneofthecomputersystemsoftheMarsroverCuriosityhadam emoryproblemdue tohighenergyparticlesandresultedinamajorsetbackform ission.Thus,ensuring theintegrityofdatastoredonCDFSisofparamountimportan t.CDFSuseschecksum ofdatafordetectingbaddata.Performingdataintegrityop erationsonentirechunkis inefcient.Ifachunkisfoundtobecorrupt,discardingthe wholechunkwillleadtoalot ofwastedIO.Italsorequiresalotoftimeandmemorytoreadt hewholechunk(64KB), toverifyitsintegrity.Thus,eachchunkissplitintoblock sof512bytes.CDFSstores CRCofeachblockofdataandperformschecksumvalidationat theblocklevel.Whena readoperationisperformedonachunk,blockbyblockisread andeachblockisveried fordataintegritybycomparingthestoredchecksumwithnew lycomputedchecksum. Thiswayifoneoftheblocksisfoundtobecorrupt,onlythatb lockismarkedbadand canbereadfromanotherhealthyreplicaofthechunk.Employi ngdataintegritycheckat theblocklevelensuresthatpartialIOordownlinkingthatw asdonebeforedetectingthe datacorruptionwillnotgowaste.Doingdataintegrityatbl ocklevelsalsoincreasesthe availabilityofdata.4.5.8InterCubeSatLinkFailure Owingtoharshspaceenvironment,communicationlinksfail often.IfaCubeSatto CubeSatlinkfails,thechildnodeintheroutingtreewillret ryconnecttoitsparent.Ifthe linkre-establishmentisnotsuccessfulorthelinkquality isbad,thechildnodewillping itsneighboursandsearchforanewparentnodeandjoinsthec luster. 4.5.9NetworkPartitioning SometimesasingleCubeSatorseveralCubeSatsmaygetseparate dfromthe CubeSatcluster.Thisphenomenoniscallednetworkpartitio ning.Ineithercase,the 58

PAGE 59

datastoredontheseparatednodeswillberetainedandwillb eavailablefordownlinking tothegroundstations.Usingthedownlinkedmetadata,sepa ratedCubeSatscanbe contactedbyServerviagroundstationsfordownlinkingthed ata. 4.6SimulationResults WesimulatedCubeSatDistributedFileSystemwithaCubeSatclu sterconsistingof onemasternodeand5-25workernodes.EachCubeSathasaproces singclockedat 1GHz,1GBRAM,32GBofashstoragememory,1Mbpsinter-clust ercommunication linkand9.6kbpsCubeSattogroundstationdatarate.Oursimu lationresultsindicate thatlestoringtimefor100MBleonclusterofsize10isabo ut12.96minutes.Since lestoringtimeisonlyfewminutes,itisnegligiblecompar edtoleprocessingandle downlinkingtime,whichareinhours. 4.7SummaryofCubeSatDistributedFileSystem WebuiltCubeSatDistributedFileSystemtostorelargelesin distributed fashionandthusenabledistributedapplicationslikeCube SatMapMerge[ 4 ]and CubeSatTorrent[ 5 ]onCubeSatClusters.Ittreatscomponentandsystemfailure sas anormratherthantheexceptionandisoptimizedforprocess ingsatelliteimagesand remotesensingwhicharehugebynature.CDFSprovidesfault tolerancebyconstant monitoring,replicatingcrucialdata,anddoesautomaticr ecovery. InCubeSatClusters,networkbandwidthandpowerarescarcer esources.A numberofoptimizationsinoursystemarethereforetargete datreducingtheamountof dataandcontrolmessagessentacrossthenetwork.Copy-ontransmitenablesmaking replicaswithoutanyadditionalorverylittlebandwidthor energyconsumption.Failures aredetectedusingHeartbeatmechanism.CDFShasbuiltinlo adbalancersforseveral usecaseslikeCubeSatMapMergeandCubeSatTorrentandallows useofuserdened customloadbalancers. 59

PAGE 60

CHAPTER5 DISTRIBUTEDPROCESSINGOFREMOTESENSINGIMAGESONCUBESAT CLUSTERS ProcessingpowerofCubeSatsisabout1GHz.Lackofavailablep owerandactive coolingofmicroprocessorsfurtherrestrictstheavailabl eprocessingpower.Asaresult, processingintensiveremotesensingapplicationscannotb eperformedonindividual CubeSatsinameaningfulamountoftime.Distributedcomputi ngoffersasolution tothisproblem.BypoolingprocessingpowerofindividualCu beSatsinacluster, processingoflargeremotesensinglescanbespeededup.Cu beSatClouduses CubeSatMapMergetoprocessremotesensingdataonCubeSatClu sters. 5.1CubeSatMapMerge CubeSatMapMergeisinspiredbyMapReduceandistailoredfor CubeSatclusters. MasternodeorchestratesCubeSatMapMerge.Masternodecomm andstheworker nodestoprocessthechunksstoredwiththem.Workernodespr ocessthechunksand produceintermediateresults.Assoonastheworkersprocess chunks,theydownlink thepartialsolutionstotheServer.OncetheServergetsallth eresults,itstitchesthe intermediatesolutionstoobtainthefullsolution.Master nodetakescareofscheduling maptasks,monitoringthemandre-executingthefailedtask s.Theworkernodes executethesubtasksasdirectedbythemaster.Figure 5-1 showsanoverviewof howanimagecanbeprocessedusingCubeSatMapMergeandisexp lainedinbriefin followingsteps. 1. Masternodesplitstheimageintochunksanddistributesthe mtotheworkernodes intheclusterusingCDFS. 2. Workernodesprocessthesplitsgiventothemtoproducepart ialsolutionsand downlinkthesolutionstoServer. 3. Serverstitchesthedownlinkedpartialsolutionsintofulls olution. 60

PAGE 61

Figure5-1.ExampleofCubeSatMapMerge 5.2CommandandDataFlowduringaCubeSatMapMergeJob Figure 5-2 showstheowofdataandcommandsduringaCubeSatMapMerge operation.WhentheAdministratorissuesaprocesscommandto theServer(Ex: processimage.jpg),thefollowingactionsoccurinthesequ encenoted. 1. Uplinkingthecommand:AdministratorissuesacommandtoServ er.Server forwardsthecommandtoGroundstation,whichuplinkstheco mmandtomaster CubeSat.(Ex:takeanimageofaparticularareaandprocessit) 2. Workassignment:Masternodecommandstheworkernodestopr ocessthe chunksstoredwiththemanddownlinktheresults. 61

PAGE 62

Figure5-2.OverviewofexecutionofCubeSatMapMergeonCube Satcluster62

PAGE 63

3. Mapphase:Workernodeprocessthechunksstoredwiththeman dstoresthe resultlocally. 4. Downlinkingtheresults:Asandwhenaworkernodeprocessesa chunk,it downlinksthesolutiontoagroundstation.Groundstationf orwardsthesolutionto Server.DownlinkingofresultsisachievedthroughCubeSatTo rrent. 5. Reducephase:OnceServerreceivesallthepartialsolutions ,itstitchestheminto fullsolution. MoredetailsaboutCubeSatMapMergearepresentedinthepape rCubeSat MapMerge[ 4 ]. 5.3FaultTolerance,Failures,GranularityandLoadBalanc ing CubeSatMapMergeistoleranttotemporaryandpermanentCube Satfailures.Its performancedegradesgracefullywithcomponent,machineo rlinkfailures.Metadatais replicatedtoavoidmissionfailureincaseoffailureofthe masternode.Workerfailures aredetectedusingHeartbeatmechanism.Ifaworkernodefai ls,thetasksassigned fortheworkernodearerescheduledonotherworkernodes.Da tachunksaresplit intoalargenumberofpiecestoimprovegranularityandload balancing.Chunksizeis selectedtobeabout64KBinordertobalancetheadvantagesof granularitywithcontrol trafcoverhead.5.3.1FaultTolerance CubeSatMapMergeisdesignedtobetoleranttotemporaryandp ermanent CubeSatfailuresanditsperformancedegradesgracefullywi thcomponent,machineor linkfailures.ACubeSatclustercancontainuptoaboutahund redCubeSatsandare interconnectedwithroughlysamenumberofhighspeedwirel esslinks.Becauseofa largenumberofcomponentsandharshspaceenvironment,som eCubeSatsorwireless linksmayfaceintermittentproblemsandsomemayfacefatal errorsfromwhichthey cannotrecoverunlesshardresetbygroundstation.Sofailur esshouldbetreatedasthe normratherthananexception.Inordertoavoidsystemdownt imeorcorruptionofdata, systemshouldbedesignedtohandlethefailuresanditsperf ormanceshoulddegrade 63

PAGE 64

gracefullywithfailures.Belowwediscusshowwehandlethes eerrorswhentheycome up.5.3.2MasterFailure Masternodestoresmetadata,whichconsistsofmappingbetw eenthemapjobs toworkernodesandthestateofmapjobs.Inordertoavoidmis sionfailureincase offailureofthemasternode,periodicallymetadataiswrit tentomastersnon-volatile memory,likeash,andthesameiscommunicatedtotheServer. Ifthemasterreboots becauseofatemporaryfailure,anewcopywillbestartedfro mthelastknownstate storedinmastersnon-volatilememory.Ifthemastercannot recoverfromerror,Map Reducemissionisabortedandrawdatacanbedownlinkedtoth eServer. 5.3.3WorkerFailure WorkernodesperiodicallysendHeartbeatmessagescontain ingtheirstatusand problems,ifany.Ifaworkerreportsfatalerror,mastermar kstheworkernodeasfailed. Processingtaskassignedtotheworkernodeisresetbacktoit sinitialidlestateandis scheduledonotherworkernodecontainingthereplicaofthe chunk. 5.3.4TaskGranularityandLoadBalancing Bysplittingthedataintoalargenumberofpieces,taskgranu laritywillbeimproved. CubeSatswithafasterprocessororspecialhardwarelikeGPU, DSPorFPGAcan processanorderofmagnitudelargenumberofmaptasksthana standardCubeSat. Finetaskgranularitywillensurebetterloadbalancing.Ho wever,asthenumberof chunksincreasesodoesthemetadataoperationsandcontrol messages,leadingto decreaseinthesystemperformance.Tobalancetheadvantag esofgranularitywiththe controltrafcoverhead,chunksizeisselectedtobeabout6 4KB. 5.4SimulationResults WesimulatedCubeSatMapMergewithaCubeSatclusterconsisti ngofonemaster nodeand5-25workernodes.EachCubeSathasaprocessingclock edat1GHz,1 GBRAM,32GBofashstoragememory,1Mbpsinter-clustercomm unicationlinkand 64

PAGE 65

9.6kbpsCubeSattogroundstationdatarate.Weprocessedima gesusingde-noise, entropy,peakdetection,segmentationandSobeledgedetect ionalgorithms.Weused ScikitPythonimageprocessinglibraryforprocessingtheima ges.Oursimulations indicatethatCubeSatMapMerge,withclustersizesintheran geof5-25CubeSats, canprocessimagesatabout4.8-23.4timesfasterthananind ividualCubeSat.These resultsindicatethatCubeSatMapMergecanspeedupprocessi ngintensiveremote sensingmissionsbyafactorofsizeofthecluster.Moredeta iledresultsarepresentedin Section8:Simulationresults. 5.5SummaryofCubeSatMapMerge CubeSatMapMergeisaverysimple,yetefcientdistributedp rocessingframework forprocessingofremotesensingimagesonCubeSatclusters. Ittreatsnodeandlink failuresasanormratherthananexceptionandisoptimizedf orprocessingremote sensingimages.Itprovidesfaulttolerancebyconstantmon itoring,replicatingcrucial data,andfastandautomaticrecovery.WithHeartbeatmechan ismtodetectfailuresand redundantexecutiontorecoverfromfailuresthisdesignis faulttolerant.Optimalchunk sizebalancestheadvantagesofgranularitywithcontroltr afcoverhead.Loadbalancing takesintoaccountofnodeswithmulticoreprocessors,grap hicprocessingunits, digitalsignalprocessorsandFPGAsintoaccountanddistribu tedthedataaccordingly. CubeSatMapMergecanspeedupprocessingintensiveremotese nsingmissionsbya factorofsizeofthecluster. 65

PAGE 66

CHAPTER6 DISTRIBUTEDCOMMUNICATIONOFREMOTESENSINGIMAGESFROMCUBESAT CLUSTERS Duetostringentspaceconstraints,CubeSatstypicallyusem onopole,dipoleand turnstileantennas.Asaresult,atypicalCubeSattogroundst ationlinkhasadatarate of9.6kbps.Lowspeeddatacommunicationisoneofthemajorb ottlenecksforremote sensingmissionsthatrequiredownlinkingoflargeamounts ofdata.Foremerging remotesensingmissions,communicationbottleneckposesa severethreatasthe connectivitywithgroundstationwillbeverylimited,inte rmittentandcomesatavery highprice.Asaresult,dataintensiveremotesensingapplic ationscannotbeperformed usingindividualCubeSatsinameaningfulamountoftime.Dis tributedcommunication offersasolutiontothisproblem.Bypoolingthecommunicati onresourcesofindividual CubeSatsinacluster,downlinkingoflargesizedremotesens ingimagescanbe speededup. WestudiedCubeSatCommunicationprotocolsincludingAX.25[ 29 ]andCubeSat SpaceProtocol(CSP)[ 30 ].Alltheseprotocolsarepoint-to-pointanddoesnotsuppor t anyformofdistributedcommunicationsforfasterdownload ingoflargedataleslike imagesorvideos.Currentlytherearenoprotocolsfordownl oadingdatafromCubeSat clustersinadistributedfashion.SowedesignedCubeSatTorr entbasedonTorrent communicationprotocoltospeedupremotesensingmissions requiringdownlinkingof largeamountsofdata.CubeSatCloudusesCubeSatTorrentford istributeddownlinking ofremotesensingdatafromCubeSatClusters. 6.1CubeSatTorrent CubeSatTorrent[ 5 ]isadistributedcommunicationsframeworkinspiredbyTor rent [ 31 ].CubeSatTorrentworksinthefollowingway.Masternodepla ystheroleoftracker. Itkeepstrackofalltheworkernodesintheclusterandtheir availabledownlinkcapacity. WhentheServerrequestsforaletobedownlinked,Masternode commandsthe workernodesintheclustertodownlinkthechunksorpartial solutionsstoredwith 66

PAGE 67

them.Workernodessimultaneouslydownlinkchunkstovario usgroundstations. GroundstationsforwardthechunkstotheServer.OnceServerr eceivesallthechunks, Serverstitchesthemtogeneratetheoriginalle.Figure 6-1 showsanoverviewofhow CubeSatTorrentworks. 6.2CommandandDataFlowDuringaTorrentSession 1. Uplinkingthecommand:Serversendsaledownlinkcommandto theground station,whichuplinksittotheMaster. 2. Distributingthesubcommands:Masterissuessubcommandst otheworkernodes storingthechunksoftheletodownlinkthem. 3. Downlinkingthechunks:Whenaworkergetsachunkdownlinkco mmand,itreads thechunkfromitslocallesystemandstartsdownlinkingit totheconnected groundstation. 4. Notication:Uponsuccessfuldownloadingofchunk,worker notiesmasterand continuestonextchunk.Thisprocessrepeatsuntilallchun ksaredownlinked. 5. Forwardingofchunks:Oncegroundstationreceiveschunk,i tforwardsthechunk toServer. 6. Reconstructingoriginalle:Onceallthechunksaredownli nkedtotheServer, Serverstitchesthechunksintotheoriginalimage. 6.3EnhancementsandOptimizations WemadeseveralenhancementsandoptimizationstoCubeSatCl oudtoimprove performance.Belowwepresenttheenhancements,particular lyforremotesensing missions,whichrequireonlydownlinkingofremotesensing data. 6.3.1ImproveStorageReliabilityandDecreaseStorageOverh ead CubeSatCloudusesredundancytoprovidereliability.Eachch unkisreplicated3 times,sothatevenifaCubeSatfailsorlosesachunk,thechun kisstillavailablewith twootherCubeSats.Replicationprovidesaccesstorawdataa teachworkernodeso thatdatacanbeprocessedbeforeitisdownlinkedtothegrou ndstation.Italsoleads toalotofcommunicationandstorageoverhead.Replicating eachchunk3times,leads to200%storageoverheadand10%-25%communicationandener gyconsumption 67

PAGE 68

Figure6-1.OverviewofCubeSatTorrent68

PAGE 69

overhead.Moredetailsaboutoverheadresultingduetorepl icationarediscussedin detailinthepaperDistributedDataStorageforCubeSatClust ers[ 32 ].Forremote sensingmissionswhichonlyneedtodownlinkthedata,there isnoadvantageofhaving accesstorawdataasworkernodesdoesnotprocessthedata.T hiscanbeusedas anopportunitytoreducethestorageandcommunicationover head.OncetheMaster performssensing,itcreateschunksC1,C2,C3...Cnofrawda ta.Then,basedonthe requiredredundancy,itcreatescodedchunksC1',C2',C3'. ..Cm,wherem > n.Master nodethendistributesthesechunkstotheworkernodes.Aslon gasnoutmchunks aredownlinked,theoriginalimagecanberecovered.Morede tailsaboutperformance analysisofsourcecodingarediscussedinChapter8.6.3.2UsingSourceCodingtoImproveDownlinkTime Someworkernodestakeunusuallylongtimetodownlinkachunk .Thesenodes arecalledstragglernodes.Therecanbeseveralreasonsfor thislikeabadantenna, cachefailures,schedulingofintensivebackgroundtasks, verylowspeedlink,etc.Ifraw dataisdownlinkeddirectly,downlinkisnotcompleteuntil thelastchunkisdownlinked totheServer.Ifastragglernodetakesverylongtimetodownl inkachunk,nomatter howfasttheothernodesdownlinktherestofthechunks,led ownlinkwillstillbeslowed downduetothedelayindownlinkingofchunkbythestraggler node.Tomitigatethe riskofslowdownofdownlinkingofalebystragglers,CubeSa tCloudperformsuses duplicatedownlinkingoflastfewchunksasexplainedinCub eSatTorrent.However, formissionsthatrequireonlydownlinkingofremotesensin gdata,efciencyofthis mitigationmechanismcanfurtherbeimprovedbyusedofsour cecoding.AfterMaster performssensing,itcreateschunksC1,C2,C3...Cnofrawda ta.Then,basedonthe requiredredundancy,itcreatescodedchunksC1',C2',C3'. ..Cm,wherem > n.Master nodethendistributesthesechunkstotheworkernodes.Whent heMasterreceivesthe DOWNLINKcommandfromServer,itstartsdownlinkingthechunk sC1',C2',...inusual wayuntilN-WchunksaredownlinkedtotheServer,whereNisth enumberofchunks 69

PAGE 70

andWisthenumberofWorkernodesintheCluster.Atthatpoint ,onlyWchunksneeds tobedownlinkedtotheServertocompletetheledownload.If anyoftheWchunk downloadstakeunusuallylong,thewholeledownloadwillt akemoretime.Inorderto preventslowdownofdownlinkingoflesduetostragglers,M asterschedulesmorethan Wchunkdownloads.Asaresult,evenifstragglernodesslowdo wndownlinkingofsome chunks,requirednumberofchunks(N)willbedownlinkedtot heServeratthehighest possiblespeed.OncetheServerreceivesNchunks,itundoest hesourcecodingto createtheoriginallefromthedownlinkedchunks.Moredet ailsaboutperformance analysisofsourcecodingarediscussedinSection8.6.3.3ImprovingtheQualityofServiceforReal-timeTrafcA pplicationsLikeVoIP Real-timetrafcapplicationslikeVoIPneedhighqualityo fservice.Traditional methodsprovidebetterqualityofservicethroughtheuseof forwarderrorcorrectionand orretransmission.GiventhatbandwidthispremiumforCube Satcommunications,large amountsofforwarderrorcorrectiondatameanshighoverhea dandthuslessbandwidth foractualdata.Retransmissionsleadtoincreaseddownlin ktimes.Othermethods forprovidingqualityofserviceincludetheuseofmultiple channelstosendcopiesof packetscreatingredundanttransmissions.Although,these methodsarecomputationally lessintensive,theydonotensureresiliencetothelossesa ndreduceoverallthroughput ofthesystem. ConsiderascenariowhereCubeSatTorrentisusedforstreami ngdatafromMaster (Sensor)node.Asexplainedbefore,Masternodesplitstheraw dataintochunks.Let's supposethatthedataframe(D)fortime t i issplitintochunksC1,C2,C3...Cn.Master usesthesechunkstocreatelinearcodedchunksC1',C2',C3' ,...Cm,wherem > n. Masterforwardsthecodedpacketstotheworkernodes,which downlinkthemtothe groundstations.Intheprocessofthisdownlinking,somepa cketsarelost.Restofthe packetsreachServer,whichthenstitchesthembackintoD,th eoriginaldataframe. ServercanobtainDbackfromcodedpacketsaslongasitreceiv esatleastnofthem 70

PAGE 71

orifamaximumofm-npacketsarelostintransmission.Ifthe Masternodenoticesthat morethanrpacketsarebeinglostontheirwaytoServer,itinc reasestheredundancy byincreasingmandthusincreasingr.Moredetailsandresul tsaboutoursourcecoding techniquearepresentedinthepaperRobustCommunications forCubeSatCluster usingNetworkCoding[ 33 ]. 6.4FaultTolerance,Failures,GranularityandLoadBalanc ing CubeSatTorrentincorporatesseveralmechanismstomakeits elftolerantto temporaryandpermanentCubeSatfailures.Itsperformanced egradesgracefully withcommunicationlinkfailures.Workernodefailuresare detectedusingHeartbeat mechanism.Ifaworkernodefails,thedownlinktasksassign edfortheworkernode arerescheduledonotherworkernodes.Sizeofdatachunksiss electedtobe64KB toimprovegranularityandloadbalancing.Chunksizeissel ectedtobeabout64KB inordertobalancetheadvantagesofgranularitywithmetad ataandcontroltrafc overhead.6.4.1FaultTolerance CubeSatTorrentisdesignedtobetoleranttotemporaryandpe rmanentCubeSat failuresanditsperformancedegradesgracefullywiththem achineorlinkfailures. Failuresarethenormratherthananexception.Aclustercan containuptohundred nodesandisconnected,withroughlyaboutthesamenumberof groundstations, throughlongdistancewirelesslinks.Thequantityandqual ityofthelinksvirtually guaranteethatsomelinksbreakintermittentlyandarenotf unctionalatanygiven time,andsomewillnotrecoverfromtheirfailures.Problems canbecausedbyhuman errors,CubeSatmobility,badantennas,communicationsyst embugs,memoryfailures, connectorsandothernetworkinghardware.Suchfailurescan resultinanunavailable communicationlinksorcanleadtodatacorruption.Therefo re,constantmonitoring, errordetection,faulttolerance,andautomaticrecoverym ustbeapartofthesystem. 71

PAGE 72

Belowwediscusshowwemeetthesechallengesandhowweresolv etheproblems whentheyoccur.6.4.2MasterFailure Masterwritesperiodiccheckpointsofallthemasterdatast ructures.Ifthemaster taskdies,anewcopywillbestartedfromthelastcheckpoint edstate.Masternode representsthesinglepointoffailurefortheCubeSatTorren t.Inordertoavoidmission failureincaseoffailureofthemasternode,periodicallym etadataiswrittentomasters nonvolatilememory,likeash,andthesameiscommunicated totheServer.Ifthe masterrebootsbecauseofatemporaryfailure,anewcopywil lbestartedfromthelast knownstatestoredinmastersnonvolatilememory.Incaseof failureofmaster,datacan bedownlinkedtofromtheworkernodestotheServer.6.4.3WorkerFailure PeriodicallyworkerssendHeartbeatmessagetoMasternode .Heartbeatmessage containsthestatusoftheworkerandproblems,ifany.Ifthe Masterdoesnotreceive HeartbeatmessagefromMasterwithin30minutes,mastermar kstheworkerasfailed. Downlinktaskassignedtotheworkerisresetbacktoitsinit ialidlestate,andscheduled onotherworkernodes.Ifaworkerlosesconnectionwithgrou ndstation,itretries withsameordifferentgroundstation.Ifitcannotconnectt oanygroundstationwithin acertainamountoftime,itsignalsfailuretomaster.Maste rmarkstheworkerasa temporaryfailednode.Iftheworkercannotconnecttothegr oundstation,Mastermarks theworkerasfailednodeandreschedulesthedownlinkingjo bassignedtothefailed workertoanotherworker.6.4.4TaskGranularity MasterdividestheletobedownloadedintoCchunks.Ideall y,Cshouldbemuch largerthanthenumberofworkermachines.Havingeachworke rdownloadmany differentchunksimprovesdynamicloadbalancing.However ,asCincreases,sodoes theamountofcontroltrafcanddelaysresultingfromexcha ngeofcontrolinformation. 72

PAGE 73

Inordertobalancetheadvantagesofgranularitywiththeov erheadincurreddueto controltrafc,Cischosentobeabout64KB.6.4.5TailEffectandBackupDownloads Somenodestakesunusuallylongtimetodownlinkachunk.Thes enodesare calledstragglers.Reasonsbehindthemcouldbeabadantenn aoraverylowspeed link.Tomitigatetheriskofslowdownofdownlinkingorupli nkingofalebystragglers, CubeSatTorrentusesbackupdownloads.Whenaledownlinkoru plinkoperationis closetocompletion,themasterschedulesbackupdownlinki ngtasksfortheremaining in-progresschunks.Thechunkismarkedasdownlinkedwhene vereithertheprimary orthebackupworkernishesdownlinking.Thisisonlyadesi gnfeatureandisnot implemented. 6.5SimulationResultsandSummaryofCubeSatTorrent WesimulatedCubeSatTorrentonaCubeSatclusterconsistingo fonemaster, 5-25workersand5-25groundstations.EachCubeSathasaproce ssingspeedof 1GHz,1GBRAM,32GBashstorageandgroundcommunicationdat aspeed9.6 kbps.CubeSatsintheclusterareconnectedtoeachotherthro ugh1Mbpshighspeed inter-clustercommunicationlinks.Oursimulationresult sindicatethatCubeSatTorrent, withclustersizesintherangeof5-25CubeSats,enables4.71 -22.93timesfaster (comparedtoasingleCubeSat)downlinkingofremotesensing data.CubeSatTorrent canpotentiallyspeedupCubeSatmissionsrequiringremotes ensingdatadownlinking byafactorofsizeofthecluster. CubeSatTorrentdemonstratestheessentialqualitiesfordo wnlinkingoflargesize remotesensingdataforCubeSatclusters.Itisfaulttoleran tandscalable.Itprovides faulttolerancebyconstantmonitoring,replicatingcruci aldata,andfastandautomatic recovery.Optimalchunksizebalancesamountofoverheadfr omcontrolmessage trafcandadvantagesofgranularity.Checksummingisused todetectdatacorruption. Proposeddesigndelivershighaggregatethroughputwhichis requiredforavariety 73

PAGE 74

ofmissions.Weachievethisbysplittingtheleintochunks anddownlinkingthem inparallelfromworkerstogroundstations.Simplieddesig nandminimalmetadata operationsresultinverylowoverhead. 74

PAGE 75

CHAPTER7 SIMULATOR,EMULATORANDPERFORMANCEANALYSIS ForsimulatingandmeasuringperformanceofCubeSatcloud,w ecreated aCubeSatCloudsimulator.Forverifyingthesimulationresu lts,wealsocreated aCubeSatCloudtestbedconsistingof5CubeSats.WeusedRaspb erryPimini single-boardcomputerforemulatingaCubeSatanddesktopco mputerforemulatingthe Serverandgroundstations.Belowisadetaileddescriptionof CubeSatCloudsimulator andemulator. 7.1HardwareandSoftwareofMasterandWorkerCubeSatsforEmul ator MasterandWorkerareemulatedusingRaspberryPi.Raspberry Piisamini single-boardcomputerdevelopedbytheRaspberryPiFoundat ion.Figure 7-1 shows variouscomponentsofRaspberryPi.IthasaBroadcomBCM2835sy stemonachip (SoC),has512MBofRAMandusesanSDcardforbootingandlong-te rmstorage. DebianandArchLinuxARMdistributionsareavailableforrunn ingonRaspberry Pi.Pythonistheprimaryadvocatedprogramminglanguagetobe usedwiththe platform,althoughsupportforBBCBASIC,C,andPerlisthere.Belo waremore detailedspecicationsofaRaspberryPiModelBsingle-boar dcomputer.Weprocessed imagesusingde-noise,entropy,peakdetection,segmentat ionandSobeledgedetection algorithms.WeusedScikitPythonimageprocessinglibraryfo rprocessingtheimages. • Processor:RaspberryPirunsonBroadcomBCM2835SoCchip.Broadco mchip includesARM1176JZFSprocessorclockedat700MHz,oatingp ointunit,and VideoCore4GPU. • Graphics:WithVideoCoreGPU,Raspberryenableshardware-acc elerated graphicscapableofrendering1Gpixel/s • SDRAM:ModelBcomeswith512MBRAM.512MBissharedwithGPU.RAMisgenrallyclockedfrom400MHzto500MHz. • Storage:Thereisnobootableashdisk,insteadbootsfrompl uggableSDcard.A minimumof2GBisrequired,butmorethen4GBissuggested. 75

PAGE 76

• Powerratings:RaspberryPidrawsabout300mA(1W)inidlepowe rmodeand about700mA(2.2W)whenallperipheralsareactive. • Ports:Raspberrycomeswith10/100BaseTEthernet,HDMIand2U SBports.Itis poweredusingmicroUSBinterface.Itssizeisroughlyabout9 x6x2cm. • Low-levelperipherals:Ithas8GeneralPurposeIO(GPIO)pins ,aUART,an I 2 C bus,aSPIbuswithtwochipselectsand I 2 S audio. Figure7-1.RaspberryPiminicomputerImagecourtesyofMatthewMurray Thespecicationsintermsofprocessingpowerandmemoryar everysimilartothat ofaCubeSat.So,weusedRaspberryPitoemulateaCubeSatintheCu beSatCloud testbed. 76

PAGE 77

7.2HardwareandSoftwareofServerandGroundStationforEmulat or ServerandgroundstationhardwareareimplementedusingDel lOptiplex755model desktopcomputers.Belowarethespecicationsofthesemach ines: • Processor:ItcomeswithtwoIntelCore2DuoCPUE8400,clockeda t3.00GHz. • Memory:Itcomeswith4GiBofRAM. • Graphics:ItispoweredbyVESARV610graphicscard. • OSType:ItisconguredtorunUbuntuLTS12.04.03PrecisePan golin,32-bit version. • Disk:Ithas240GBofstoragediskforOSandpermanentstorag e. WeusedtheopensourceUbuntu12.04.03LongTermSupport(LTS) versionasour baseOperatingSystemforServerandgroundstation.Forrunni ngtwistedapplications, weusedPython2.7.3versionasPythonversionsabove3.0didno thavefullsupport forTwistedframework.WeusedPythonpython-twisted11.1.0 builtforUbuntuusingthe python-twistedpackage. 7.3NetworkProgrammingFrameworks InordertodevelopCubeSatCloudframework,weresearchedav ailablenetwork programmingframeworksinPython,includingTwisted,Eventl et,PyEv,asynccore, Tornado.Belowisabriefdescriptionoftheeachofthesefram eworks. 7.3.1Twisted Twistedisconsideredasthebestreactorframeworksavaila bleinPython.Itisalittle bitcomplexandhasasteeplearningcurve,butiselegantand providesallnecessary featuresrequiredfordevelopingasynchronousapplicatio ns. 7.3.2Eventlet EventletwasdevelopedbyLindenLab.ItisbasedonGreenletf ramework,which isgearedtowardsasynchronousnetworkapplications.Itis non-pep8compliant tough.Loggingmechanismisnotimplementedtothefullexte nt,theAPIissomewhat inconsistent. 77

PAGE 78

7.3.3PyEv PyEvisbasedonlibeventframework.Itneedstobedevelopedlo tmoretobe consideredasaseriouscompetitorwithothernetworkprogr ammingframeworks.There doesnotseemtobebigcompaniesusingthisframework,asofn ow. 7.3.4Asyncore Asyncoreisbasedonstdlibandisaverylow-levelframework. Thereisnotmuch supportforhigh-levelnetworkoperations,soalotofboile rcodeneedstobewrittenjust togetstartedwithnetworkapplications.7.3.5Tornado Tornadoisaverysimplepythonservermeantfordevelopingd ynamicwebsites. ItfeaturesasyncHTTPclientandasimpleioloop.Itssimple ,butnotproviderequired callbackfeaturestobeconsideredacandidateforimplemen tingCubeSatCloud. 7.3.6Concurrence Concurrenceisanetworkingframeworkforcreatingmassive lyconcurrent networkapplicationsinPython.Itexposesahigh-levelsync hronousAPItolow-level asynchronousIOusinglibevent.ItrunsusingeitherStackle ssPythonorGreenlets. AllblockingnetworkI/Oistransparentlymadeasynchronous throughasinglelibevent loop,soitisnearlyasefcientasarealasynchronousserve r.ItissimilartoEventletin thisway.ThedownsideisthatitsAPIisquitedifferentfromPyt hon'ssockets/threading modules. 7.4TwistedFramework Oftheframeworks,weresearchedintoTwistedwasbestsuite dforourjob,sinceit providedhandyfeatureslikecallbacks,deferreds,etc.al ongwithastrongcommunity support.Itisanasynchronouseventbasednetworkprogramm ingframework.Itis implementedinPythonprogramminglanguageandlicensedund eropensourceMIT license.CallbacksarethecorepartoftheTwistedframewor k.Userswritecallbacks 78

PAGE 79

andregisterthemtobecalledwheneventshappen(asaconnec tionismade,a messageisreceived,orconnectionislost). 7.5NetworkConguration CubeSattogroundstationcommunicationlinkismodelledwit hdatarateof9600 bps,adelayof2mswithajitterof200usfollowingnormaldis tribution.Wemodelled theCubeSatClustercommunicationlinksusingthespecicat ionsofRelNAV.Datarate is1Mbps,linkcommunicationdelayof0.1ms.Packetlossrat ewassetat0.3%,with a25%losscorrelationinordertosimulatepacketburstlose s.WeusedHierarchical TokenBucket(HTB)andtcnetworkingtoolonLinuxtoshapethen etworktrafctoour requirements. 7.6CubeSatCloudEmulatorSetup CubeSatCloudemulatorconsistsofoneServer,oneMaster,5Wo rkernodes and5Groundstations.CubeSatCloudemulatorisshownintheF igure 7-2 .Master andworkerCubeSatsareemulatedusingRaspberryPi,sincethe CubeSatsfootprint (processingpowerandRAM)matcheswiththatofRaspberryPi's .Serverandground stationsareemulatedusingDellOptiplexcomputer.Allthec omponentsareconnected usingaGigabitEthernetswitch.CubeSattoCubeSatandCubeSatt ogroundstation communicationlinksareconguredasdescribedinthesecti on 7.5 7.7CubeSatCloudSimulatorSetup CubeSatCloudsimulatorconsistsofoneServer,oneMaster,525Workernodes andgroundstations.SystemarchitectureofCubeSatCloudsim ulatorisshowninthe Figure 7-3 .SimulationisrunontheDellOptiplexcomputerdescribedin section 7.2 MasterandWorkerCubeSataresimulatedusingtheprolingre sultsobtainedfrom theemulator.ComponentscommunicatetoeachusingTCP/IPso cketsoflocalhost interface.CubeSattoCubeSatandCubeSattogroundstationcom municationlinksare conguredasdescribedinthesection 7.5 .Simulationresultsarepresentedbelow. 79

PAGE 80

Figure7-2.CubeSatCloudemulator80

PAGE 81

Figure7-3.CubeSatCloudsimulator81

PAGE 82

7.8CubeSatReliabilityModel Datareliabilityisachievedthroughreplication.Eachremo tesensingimageis splitintochunksanddistributedtoworkernodes.Eachchunk isreplicatedonmultiple CubeSats,sothatifsomeCubeSatsfail,dataisstillavailabl eonotherCubeSats. Numberofreplicasperchunkisprimarilygovernedbyrequir edavailabilityofdataand nodefailurerate.Availabilityofanimage(A)isgivenby, A=(1 f R ) C 100 Where,fistheprobabilityoffailureofanode,Risthenumber ofreplicasofeach chunkandCisnumberofchunksofthele.TondtheCubeSatfai lureprobability, wecollecteddataaboutlifetimesoftheCubeSatsthatarelau nchedsofar.Figure 7-4 showsasummaryofthelifetimeoflaunchedCubeSats.Moredet ailsaboutCubeSats launchedsofarcanbeobtainedfrom”ASurveyofCommunicatio nSub-systemsfor Inter-satelliteLinkedSystemsandCubeSatMissions”[ 34 ].Usingtheabovedata,we calculatedthatthemeanlifetimeofaCubeSatisabout1204da ys.Anddependingon thedownlinkspeedsandmissiondatasize(about100MB),arem otesensingmission cantakeabout1day.SotheprobabilityoffailureofCubeSatdu ringamission(f)is about 10 3 .Typicalnumberofchunksperle(C)isabout1000.Witharedu ndancy of1(2replicasforeachchunk)CDFSprovidesanavailabilit yof99.98andwitha redundancyof2(3replicas)CDFSprovidesanavailabilityo f99.9999.Wetargetedan availabilityof99.9999.Soeachchunkneedstobereplicated 3times. 7.9SimulationandEmulationResults 7.9.1ProlingReadingandWritingofRemoteSensingDataChunk sonRaspberryPi Inordertobuildasimulationframework,wedidprolingofr eadingchunksfrom ashstorageandwritingchunkstoashstorageofremotesen singdatachunkson RaspberryPisingleboardmini-computer.Prolingresultsar ereportedinFigure 7-5 Averagereadingandwritingtimesforachunkofsize64KBare4 .91and15.66ms. 82

PAGE 83

Figure7-4.LifetimesofCubeSats Thisshowsthat,readingandwritingaleof100MBwilltakea bout8and25seconds. Comparedtothis,processinganddownlinkingachunkwillta keorderofhours.So readingandwritingtimesarenegligiblecomparedtotimeta kenforprocessingand downlinkingaremotesensingimage.7.9.2Processing,CubeSattoCubeSatandCubeSattoGroundStatio nChunk CommunicationTime Wedidprolingofprocessingtime,CubeSattoCubeSatcommuni cationtime andCubeSattoGroundstationcommunicationtime.Weprocess edimagesusing de-noise,entropy,peakdetection,segmentationandSobele dgedetectionalgorithms. WeusedScikitPythonimageprocessinglibraryforprocessing theimages.Processing timeistheaverageoftimetakenbyRaspberryPitoprocessach unkusingtheabove mentionedimageprocessingalgorithms.Communicationlin ksaresimulatedusing theparametersspeciedinsection8.5NetworkConguratio n.Prolingresultsare reportedinFigure 7-6 .AverageCubeSattoCubeSatchunkcommunicationtimeis 83

PAGE 84

Figure7-5.Readandwritetimesofachunk about1.19seconds,processingtimeis15.62secondsandCub eSattoGroundStation communicationtimeisabout68.29seconds.Thisresultshow sthatchunkprocessing timeandchunkcommunicationtimefromCubeSattogroundstat ionaremorethan anorderofmagnitudelargerthanchunkcommunicationtimeb etweenCubeSatto CubeSat.Asaresult,distributingaleontheclusterwillbem uchfastercompared toprocessinganddownlinkingthele.Theseresultsalsoin dicatethatprocessingan imageofsize100MBonasingleCubeSatwilltakeabout7hoursa nddownlinking thesamewilltakeabout30hours.Henceweneedtoparalleliz eprocessingand downlinkingofremotesensingimages.7.9.3StoringRemoteSensingImagesusingCubeSatCloud Figure 7-7 showsthetimetakenforstoring(splittingamimageintochu nksand distributingthechunkstotheworkernodesinthecluster)a nimageontheCubeSat 84

PAGE 85

Figure7-6.CubeSattoCubeSatandCubeSattogroundstationchu nkcommunication proling clusterforvariousclusterandimagesizes.Forclustersiz eof1(asingleCubeSat),le storingtimeisalmostzero(11secondsforaleofsize100MB) ,sincethelesonly needstobesplitintochunksanddoesnotneedstobedistribu tedoverthenetwork. Averagelestoringtimefor100MBleonclusterofsize10is about12.96minutes. Sincelestoringtimeisonlyfewminutes,itisnegligibleco mparedtoleprocessing andledownlinkingtime,whichareinhours.7.9.4ProcessingRemoteSensingImagesusingCubeSatCloud Figure 7-8 showstheimageprocessingtimesforvariousclusterandima gesizes. Weprocessedimagesusingde-noise,entropy,peakdetectio n,segmentationand Sobeledgedetectionalgorithms.WeusedScikitPythonimagepr ocessinglibraryfor processingtheimages.Processingtimeistheaverageoftime takenbyCubeSatCloud toprocesstheremotesensingimagesusingtheabovemention edimageprocessing algorithms.Forclustersizeof1(asingleCubeSat),leproc essingtimeis448minutes. Averageleprocessingtimefor100MBleonclustersofsize 10and25isabout47 and19minutesrespectively.Thisresultsinasavingsof401 minutesforprocessinga leonclusterofsize10and429minutesonaclusterofsize25 .CubeSatMapMerge 85

PAGE 86

Figure7-7.Filedistributiontimeforvariouslesizesand clustersizes reducestheprocessingtimefromabout8hourstolessthanan hourandthusis attractiveforprocessinglargesizeremotesensingimages Figure7-8.Fileprocessingtimeforvariouslesizesandcl ustersizes 7.9.5SpeedupandEfciencyofCubeSatMapMerge WestudiedthevariationofspeedupandefciencyofCubeSatM apMergewith variationinclustersize.Speedupisdenedastheratioofti metakenbythecluster 86

PAGE 87

toprocessimagetothetimetakenbyasingleCubeSattoproces sthesameimage. Efciencyisdenedasratioofspeedupoftheclustertothecl ustersizeexpressedin percentage.Figure 7-9 showsthevariationofprocessingspeedupwithclustersize forlargeles( > 10MB).Forclustersizesof10and25thespeedupis9.54and23. 40 respectively.Figure 7-10 showsthevariationofprocessingefciencywithclustersi ze forlargeles( > 10MB).Forclustersizesof10and25theefciencyis95.38and 93.61 respectively. Figure7-9.SpeedupofCubeSatMapMerge 7.9.6DownlinkingRemoteSensingImagesUsingCubeSatCloud Figure 7-11 showstheimagedownlinkingtimeforvariousclusterandle sizes. DownlinkingtimeisthetimetakenbytheaCubeSatorCubeSatCl ustertodownlink aremotesensingimagetotheServer.AsingleCubeSattakes1da y6hoursof connectivitytimetodownlinkaleofsize100MB.Comparedt othataveragele 87

PAGE 88

Figure7-10.EfciencyofCubeSatMapMerge downlinkingtimefor100MBleonclusterofsize10needsonl yabout3hours 13minutesofconnectivity.Thisresultsinasavingsofabou t27hoursoftimefor downlinkingaleof100MB.CubeSatTorrentreducesimagedow nlinkingtime approximatelybythefactorofthesizeofthecluster.7.9.7SpeedupandEfciencyofCubeSatTorrent WestudiedthevariationofspeedupandefciencyofCubeSatT orrentwithvariation inclustersize.Speedupisdenedastheratiooftimetakenby theclustertodownlink animagetothetimetakenbyasingleCubeSattodownlinkthesa meimage.Efciency isdenedasratiooftotaleffectivedataspeedofthecluste rtothetotalrawdataspeed oftheclusterexpressedinpercentage.Figure 7-12 showsthevariationofprocessing speedupwithclustersizeforlargeles( > 10MB).Forclustersizesof10and25the speedupis9.35and22.93respectively.Figure 7-13 showsthevariationofprocessing 88

PAGE 89

Figure7-11.Filedownlinkingtimeforvariouslesizesand clustersizes efciencywithclustersizeforlargeles( > 10MB).Forclustersizesof10and25the efciencyis71.95and70.59respectively.7.9.8CopyOnTransmitOverhead Figure 7-14 showsthebandwidthoverheadduetoreplicationusingCopyOn-Transmit forvariousclustersizes.Energyoverheadissameasbandwid thoverhead.Bandwidth overheadforCopyOnTransmitforclustersizesof10and25is 35.71and9.61 respectively.CopyOnTransmitleadsto200%storageoverhe ad,asitcreatestwo explicitreplicas.7.9.9SourceCodingOverhead Figure 7-15 showsthebandwidthoverheadforsingleanddoubleredundan cy duetoSourceCodingforvariousclustersizes.Withsinglered undancy,datacan berecoveredincaseofonefailedCubeSat.Usingdoubleredun dancy,datacanbe recovered,eveniftwoCubeSatsfail.BandwidthoverheadforSo urceCodingforcluster sizesof10and25variesfromabout5-25%dependingthenumbe rofredundant chunksandclustersize.Energyoverheadissameasbandwidth overhead. 89

PAGE 90

Figure7-12.SpeedupofCubeSatTorrent 7.9.10MetadataandControlTrafcOverhead Figure 7-16 showsthebandwidthoverheadduetometadataandothercontr ol informationforvariousclusterandlesizes.Bandwidthove rheadisabout0.4-1%. Bandwidthandoverheadpercentageismostlyindependentoft helesizeandvaries primarywiththeclustersize.7.9.11ComparisonofCDFSwithGFSandHDFS BandwidthandenergyareverylimitedonCubeSatcluster.CDFS usesseveral enhancementslikeusingMasternodeassuperreplicanode,C opy-on-transmitandliner blocksourcecodingforreducingtheenergyandbandwidthco nsumption.Figure 7-17 showsthebandwidthrequiredbyCDFSandGFS(aswellasHDFS)f orwritingaleof 100MBtothecluster.CDFSconsumesabout35-40%lessbandwi dthcomparedGFS andHDFS.Figure 7-18 showsthetimetakenbyCDFSandGFS(aswellasHDFS) 90

PAGE 91

Figure7-13.EfciencyofCubeSatTorrent forwritingaleof100MBtothecluster.CDFSwritesareabou t50%fasterthanGFS andHDFSbecauseofsuperreplicanodeandreducedbandwidth requirements.Figure 7-19 showstheenergyrequiredbyCDFSandGFS(aswellasHDFS)forw ritingale of100MBtothecluster.CDFSconsumesabout40%lessenergyc omparedGFSand HDFS.7.9.12SimulatorvsEmulator Figure 7-20 showsthetimerequiredforwriting,processinganddownlin kingremote sensingimageofsize100MB.Simulatorresultsareabout5-12 %morethenemulator results.Thisdiscrepancymightbeattributedtoduedelays inthesimulationframework becauseoflargenumberofthreadsrunningsimultaneously. 91

PAGE 92

Figure7-14.Bandwidthoverheadduetoreplication Figure7-15.Bandwidthoverheadduetosourcecoding 92

PAGE 93

Figure7-16.Bandwidthandenergyoverhead Figure7-17.BandwidthconsumptionofCDFSvsGFSandHDFS 93

PAGE 94

Figure7-18.WritetimeofCDFSvsGFSandHDFS Figure7-19.EnergyconsumptionofCDFSvsGFSandHDFS 94

PAGE 95

Figure7-20.Simulatorvsemulator 7.10SummaryofSimulationResults WesimulatedCubeSatCloudframeworkonCubeSatCloudtestbed .CubeSat CloudframeworkwasdevelopedusingPythonprogramminglang uage.Wesimulated CubeSatTorrentonaCubeSatclusterconsistingofonemaster, 5-25workersand5 -25groundstations.EachCubeSathasaprocessorrunningat1G Hz,1GBRAM,32 GBnon-volatilememory,1Mbpsinter-clustercommunicatio nlinkand9.6kbpsground stationdatarate.Serverandgroundstationsareconnectedt oeachotherviaInternet through10Mbpsdataratecommunicationlinks. WesimulatedCubeSatCloudwithvariousclustersizes.Oursi mulationresults indicatethatforclustersizesinrangeof5to25CubeSats,as peedupof4.75-23.15 timesfaster(comparedtoasingleCubeSat)processinganddo wnlinkingofremote sensingimagescanbeachieved.Simulationresultscloselym atchwithresultsfromthe testbed. 95

PAGE 96

CHAPTER8 SUMMARYANDFUTUREWORK Weight,powerandgeometryconstraintsseverelylimitproc essingandcommunication capabilities.ACubeSathasabout1GHzprocessingcapabilit y,1GBRAM,32-64 GBofashmemoryandCubeSattogroundstationcommunication datarateof9.6 kbps.Asaresult,processingandcommunicationintensivere motemissions,which generateabout100MBpersensingoperation,cannotbecompl etedinameaningful amountoftime.Processingaremotesensingimageofsize100M Btakesabout8 hoursanddownlinkingtakesadayandquarterwithcurrentin frastructure.Weconsider thepossibilityofusingdistributedstorage,processinga ndcommunicationsforfaster executionofremotesensingmissions. Wepropose,CubeSatCloud,aframeworkfordistributedstora ge,processing andcommunicationofremotesensingdataonCubeSatClusters .CubeSatCloud isoptimizedforstoring,processinganddownlinkingoflar gesizedremotesensing datawhichisoforderofhundredsofmegabytes.CubeSatCloud usesCubeSat DistributedFileSystemforstoringremotesensingdataindi stributedfashiononthe cluster.CubeSatDistributedFileSystemsplitsthelargesiz eremotesensingdatain chunksanddistributesthemtotheworkernodesinthecluste r.Metadataconsistingof letochunkmappingandchunktoworkernodemappingisstore dwithMasternode. ForprocessingdistributeddataCubeSatCloudusesCubeSatMa pMerge.Worker nodesprocessthechunksstoredwiththemandstoretheresul tsobtainedonthelocal lesystem.Oncethechunksareprocessed,theyaredownlink edtoServerusing CubeSatTorrent.Serverstitchesthepartialsolutionsintof ullsolution.Component andlinkfailuresaretreatedasnorminsteadofexceptions. Failuresaredetectedusing Heartbeatmechanismandsystemistoleranttocomponentand linkfailures.CubeSat Cloudimplementsseveralenhancementsincludingcopy-ontransmitandlinearblock sourcecodingtoreduceconsumptionofscarceresourceslik epowerandbandwidth. 96

PAGE 97

ForsimulatingCubeSatcloudwecreated,CubeSatCloudtestbe d.Wesimulated CubeSatsusingRaspberryPisandtestbediswrittenusingPytho n-twisted,anevent basedasynchronousnetworkprogrammingframework.Simulat ionresultsindicate thatCubeSatMapMergeandCubeSatTorrent,withclustersizes inrangeof5-25 CubeSats,enables4.75-23.15timesfaster(comparedtoasin gleCubeSat)processing anddownlinkingoflargesizedremotesensingdata.Allthiss peedisachievedatalmost negligiblebandwidthandmemoryoverhead(1%).Theseresul tsindicatethatCubeSat Cloudcanspeedupremotesensingmissionsbyafactorofsize ofcluster. 8.1Futurework BelowisanoverviewofthefutureworkasanextensiontoCubeSa tCloud. LaunchinganddeployingoftheCubeSatsintoCubeSatclustera ndmaintainingthe clusterforlongtimeperiodsneedstobelookedinto.CubeSat Cloudwasdesigned usingPythonprogramminglanguageinordertosupportrapidp rototyping.Aightready systemcanbebuiltusingC++andthenetworkstackcanbeopti mizedforCubeSat communicationchannelcharacteristics.Linklayercommun icationprotocolcanbe integratedintoCubeSatTorrenttoimprovetheefciencyoft hedownloads.From CubeSatsubsystemsperspective,alightweightCubeSattoCub eSatlowdistancehigh speedLASERcommunicationmodulewillsignicantlyenhanceth eefciencyofthe systemandleadtoreducedenergyconsumption. 97

PAGE 98

REFERENCES [1] H.Heidt,J.Puig-Suari,A.MooreandR.Twiggs,“Cubesat:AnewG eneration ofPicosatelliteforEducationandIndustryLow-CostSpaceExpe rimentation,” ProceedingsoftheUtahStateUniversitySmallSatelliteConfer ence,Logan,UT, Citeseer ,p.12,2001. [2] AndrewE.Kalman(2010,Jan15),“CubeSatKit:CommercialOfftheShelfComponentsforCuebsats,” RetrievedJuly16,2012,from http://www.cubesatkit.com/docs/datasheet/ [3] J.Gozalvez,“SmartphonesSentIntoSpace[MobileRadio],” VehicularTechnology Magazine,IEEE ,vol.8,no.3,pp.13–18,2013. [4] ObulapathiN.ChallaandJaniseY.McNair,“DistributedCom putingonCubesat ClustersusingMapreduce,” iCubeSat,TheInterplanetaryCubeSatWorkshop 2012. [5] ObulapathiN.ChallaandJaniseY.McNair,“CubeSatTorrent: Torrentlike DistributedCommunicationsforCubeSatSatelliteClusters, ” MilitaryCommunicationsConference ,pp.1–6,2012. [6] D.E.KoelleandR.Janovsky,“DevelopmentandTransportatio ncostsofSpace LaunchSystems,” DGLR/CEASEuropeanAirandSpaceConference ,2007. [7] KirkWoellertandPascaleEhrenfreundandAntonioJ.RiccoandH enryHertzfeld, “Cubesats:Cost-effectiveScienceandTechnologyPlatforms forEmergingand DevelopingNations,” AdvancesinSpaceResearch ,vol.47,no.4,pp.663–684, 2011. [8] JeffreyDeanandSanjayGhemawat,“MapReduce:SimpliedData Processingon LargeClusters,” CommunicationsoftheACM ,vol.51,no.1,pp.107–113,2008. [9] S.LeeandJ.Puig-Suari, CoordinationofMultipleCubeSatsontheDneprLaunch Vehicle,M.S.Thesis .CaliforniaPolytechnicStateUniversity,December2006. [10] B.Klofas,J.Anderson,K.Leveque,“ASurveyofCubeSatCommunica tion Systems,” 5thAnnualCubeSatWorkshop-CalPoly ,2008. [11] MoreDBsteamatUniversityofCalPoly,“MassiveOperations, Recording,and ExperimentationDatabaseSystem(2011,April15).,” RetrievedJuly16,2012,from http://moredbs.atl.calpoly.edu/ ,2008. [12] NormanG.Fitz-Coy,“SpaceSystemsGroup(ssg)(2008,aug26). ,” RetrievedJuly 16,2012,fromhttp://www2.mae.u.edu/ssg/ [13] JaniseY.McNair,“WirelessandMobileSystemsLaboratory(wa m)(2008,aug 26).,” RetrievedJuly16,2012,fromhttp://www.wam.ece.u.edu/ 98

PAGE 99

[14] TzuYu.Lin,TakashiHiramatsu,NarendranSivasubramaniana ndNormanG. Fitz-Coy,“T-c3:Acloudcomputingarchitectureforspacec rafttelemetrycollection,” RetrievedJuly16,2012,fromhttp://www.swampsat.com/tc 3 ,2011. [15] GENSOConsortium,“GlobalEducationalNetworkforSatelliteOp erations(2009, jun20).,” RetrievedJuly16,2012,fromhttp://www.genso.org/ ,2009. [16] R.Scrofano,P.R.Anderson,J.P.Seidel,J.D.Train,G.H.Wang, L.R.Abramowitz, J.A.BannisterandD.Borgeson,“Space-basedlocalareanetwork ,” Military CommunicationsConference,2009. ,pp.1–7,2009. [17] NestorVoronka,TyrelNewton,AlanChandlerandPeterGagnon ,“Improving CubeSatCommunications,” CubeSatDevelopersWorkshop,CalPoly ,2013. [18] SanjayGhemawat,HowardGobioffandShun-TakLeung,“TheGoog leFile System,” SIGOPSOperatingSystemsReview ,vol.37,no.5,pp.29–43,2003. [19] K.Shvachko,HairongKuang,S.RadiaandR.Chansler,“TheHado opDistributed FileSystem,” IEEESymposiumonMassStorageSystemsandTechnologies (MSST) ,pp.1–10,2010. [20] MahadevSatyanarayanan,J.J.Kistler,P.Kumar,M.E.Okasaki, E.H.Siegeland D.C.Steere,“Coda:AHighlyAvailableFileSystemforaDistri butedWorkstation Environment,” IEEETransactionsonComputers ,pp.447–459,1990. [21] SunJian,LiZhan-huaiandZhangXiao,“ThePerformanceOptimi zationofLustre FileSystem,” 7thInternationalConferenceonComputerScienceEducation (ICCSE) ,pp.214–217,2012. [22] ApacheSoftwareFoundation,“ApacheThrift,” RetrievedJuly16,2012,from http://thrift.apache.org/ ,January2012. [23] ApacheSoftwareFoundation(2012,Feb6).,“HDFS:HadoopDistr ibutedFile System,” RetrievedJuly16,2012,fromhttp://hadoop.apache.org/ ,June2012. [24] “FloridaUniversitySATelliteV(FUNSATV)Competition,” RetrievedJuly16,2012, fromhttps://vivo.u.edu/display/n958538186 ,2009. [25] “TethersSDR(2012):SoftwareDenedRadio(SWIFTSDR)BasedCommunicationDownlinksforCubeSats,” RetrievedJuly16,2012,from http://goo.gl/Q5fut ,2012. [26] “RelNav:RelativeNavigation,TimingandDataCommunicati ons forCubeSatClusters,” RetrievedJuly16,2012,from http://www.tethers.com/SpecSheets/RelNavSheet.pdf [27] PaulMuri,ObulapathiN.ChallaandJaniseY.McNair,“Enhanc ingSmallSatellite CommunicationThroughEffectiveAntennaSystemDesign,” MilitaryCommunicationsConference,2010 ,pp.347–352,2010. 99

PAGE 100

[28] R.Russell,D.QuinlanandC.Yeoh,“FilesystemHierarchySta ndard,” Retrieved July16,2012,fromhttp://refspecs.linuxfoundation.org /FHS 2.3/fhs-2.3.pdf ,January 2003. [29] A.William,Beech,D.E.NielsenandJ.Taylor,“AX.25LinkAccessProtocolforAmateurPacketRadio,” RetrievedJuly16,2012,from http://www.tapr.org/pdf/AX25.2.2.pdf ,1998. [30] “CubeSatSpaceProtocol:ASmallNetwork-layerDeliveryProtoco lDesignedfor CubeSats,” RetrievedJuly16,2012,fromhttps://github.com/GomSpace /libcsp April2010. [31] B.Cohen,“TheBitTorrentProtocolSpecicationStandard,” RetrievedJuly16, 2012,fromhttp://www.bittorrent.org/beps/bep 0003.html ,January2008. [32] ObulapathiN.ChallaandJaniseY.McNair,“DistributedDat aStorageonCubeSat Clusters,” AdvancesinComputing ,pp.36–49,2013. [33] GokulBhat,ObulapathiChalla,PaulMuriandJaniseMcNair,“ Robust CommunicationsforCubeSatClusterusingNetworkCoding,” 3rdInterplanetary CubeSatWorkshop ,2013. [34] PaulMuriandJaniseMcNair,“ASurveyofCommunicationSub-sy stems forIntersatelliteLinkedSystemsandCubeSatMissions,” JCM ,vol.7,no.4, pp.290–308,2012. 100

PAGE 101

BIOGRAPHICALSKETCH Dr.ObulapathiN.ChallawasbornandbroughtupinIndia.Her eceivedaB.S.in InformationandCommunicationTechnologyfromDA-IICTinIn dia,aM.S.inComputer EngineeringandaPh.D.inCloudComputingfromtheUniversity ofFlorida.Heworked asaResearchAssistantwithDr.JaniseMcNairandwasapartof WirelessandMobile Laboratory,andSmallSatelliteGroupatUniversityofFlorid a.Hisinterestsinclude CloudComputing,BigData,SmallSatellites,OpenSourceandDis tributedSystems. 101