Automated evaluation of physical therapy exercises using multi-template dynamic time warping on wearable sensor signals

j ou rn a l h om epa ge :w w w . i n t l . e l s e v i e r h e a l th . c o m / j o u r n a l s / c m p b

Automated

evaluation

of

physical

therapy

exercises

using

multi-template

dynamic

time

warping

on

wearable

sensor

signals

Aras

Yurtman,

Billur

Barshan

DepartmentofElectricalandElectronicsEngineering,BilkentUniversity,Bilkent,TR-06800Ankara,Turkey

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received15October2013

Receivedinrevisedform9July2014 Accepted14July2014

Keywords:

Physicaltherapy Motionsensors Inertialsensors Dynamictimewarping Patternrecognition Motioncapture

a

b

s

t

r

a

c

t

Wedevelopanautonomoussystemtodetectandevaluatephysicaltherapyexercisesusing wearablemotionsensors.Weproposethemulti-templatemulti-matchdynamictimewarping

(MTMM-DTW)algorithmasanaturalextensionofDTWtodetectmultipleoccurrencesof morethanoneexercisetypeintherecordingofaphysicaltherapysession.Whileallowing somedistortion(warping)intime,thealgorithmprovidesaquantitativemeasureof sim-ilaritybetweenanexerciseexecutionandpreviouslyrecordedtemplates,basedonDTW distance.Itcandetectandclassifytheexercisetypes,andcountandevaluatetheexercises ascorrectly/incorrectlyperformed,identifyingtheerrortype,ifany.Toevaluatethe algo-rithm’sperformance,werecordadatasetconsistingofonereferencetemplateand10test executionsofthreeexecutiontypesofeightexercisesperformedbyfivesubjects.Wethus recordatotalof120and1200exerciseexecutionsinthereferenceandtestsets,respectively. Thetestsequencesalsocontainidletimeintervals.Theaccuracyoftheproposedalgorithm is93.46%forexerciseclassificationonlyand88.65%forsimultaneousexerciseandexecution typeclassification.Thealgorithmmisses8.58%oftheexerciseexecutionsanddemonstrates afalsealarmrateof4.91%,causedbysomeidletimeintervalsbeingincorrectlyrecognizedas exerciseexecutions.Totesttherobustnessofthesystemtounknownexercises,weemploy leave-one-exercise-outcrossvalidation.Thisresultsinafalsealarmratelowerthan1%, demonstratingtherobustnessofthesystemtounknownmovements.Theproposedsystem canbeusedforassessingtheeffectivenessofaphysicaltherapysessionandforproviding feedbacktothepatient.

©2014ElsevierIrelandLtd.Allrightsreserved.

1.

Introduction

and

background

Physical therapy isan important type ofrehabilitation for patientswithvariousdisorders.Cardiopulmonarymedicine, neurology, orthopedics, and pediatrics are branches of medicinethatmaybenefitfromphysicaltherapy[1,2].Physical

∗ _{Correspondingauthor.Tel.:+903122902161;fax:+903122664192.}

E-mailaddresses:yurtman@ee.bilkent.edu.tr(A.Yurtman),billur@ee.bilkent.edu.tr(B.Barshan).

therapyusuallyrequiresexercisinginahospitalora rehabili-tationcenterunderthesupervisionofaspecialistwhoassigns oneormoreexercisestothepatient[3].Afterlearninghowto performtheexercisescorrectly,patientsusuallyneedto con-tinueexercisingathome,wheretheyreceivenofeedback[4]. Even atthe hospital,specialistscannotfolloweach patient continuously during their exercise sessions because the

http://dx.doi.org/10.1016/j.cmpb.2014.07.003

formeroftenalternatebetweenseveralpatientsorhaveother taskstodo,asituationthatcanresultininsufficient, inaccu-rate,andoftensubjectivefeedback[5,6].

Animportantissueinphysicaltherapyistoevaluatethe exercisesandassesstheeffectivenessofanexercisesession. Previousstudies consider energy expenditure [7], the total duration ofthe exercisesession,or thetime periodduring whichthepatientisphysicallyactive[8].Thesystemin[3] generatesa warningwheneverrelevantparameters exceed predefinedthresholdstoreducetheriskofover-exercising. Theseapproachesmaybemisleadingorfailifthepatient per-formstheexercisesincorrectlyoratanunusualpace.Previous studieshavenotconsidereddetectingsequentialexercise exe-cutionsandprovidinganobjectiveevaluationoftheiraccuracy toassesstheeffectivenessofaphysicaltherapysession.Such aprocess isalsodifficultandtedious foraspecialist, even whenheisresponsibleforonlyasinglepatient,and impos-siblewhenheismonitoringseveral.Patientsandspecialists wouldhighlybenefitfromanautomatedsystem.

Several differentsensor modalities are usedin physical rehabilitation,includinginertial[9–12],visual[13–15],strain [4,16],medical[6],physiological,kinetic,andenvironmental sensors[17],orsomecombinationofthese[18,19].Automatic monitoringofthepeopleperformingphysicaltherapy exer-cisesshouldbedonewithoutrestrictingtheirindependence, intrudingontheirprivacy,ordegradingtheirqualityoflife. Acommonlyusedapproachistofixcamerasinthe environ-ment,whichintrudesonprivacyandusuallyhasarelatively highinstallationcost.Themainadvantageofthisapproach isthatthepersondoesnothavetowearorcarryanysensors ordevicesontheirbody.Thisapproachmayalsoeliminate problemsrelatedtomisplacingsensorsonthebody,although somevision-based systemsrequirewearing/pasting special tagsormarkers.Thismethodmaybeacceptablewhenthe personalwaysperformsexercisesatthesameplace,butwhen theexercisesareperformedindifferentplaces,e.g.,indoors andoutdoors,thisapproachbecomesunsuitable.Webelieve thatwearablesensorsaresuperiortocamerasystemsinthese respects.

In[20],participants3–9monthspost-strokewithmildto moderatemotorimpairmentofonearmworean accelerom-eteroneacharmoutsidethelaboratoryforthreedaysbefore andaftertreatmentoranequivalentno-treatmentperiod.The useofthemoreimpairedarmindailylifewasassessedusing low-passfilteredaccelerometerrecordings.Otherstudiesthat focusonpost-strokerehabilitationare[21–23].

Innumerousstudies,a3Dreal-timehumanbodymodel isconstructedtoobservemovements[5,24,25].In[24,26,27], patients perform the givenexercises to complete tasks in video-game-likevirtualenvironments,makingexercise ses-sionsmoreenjoyable.

Another approach to observing movements is to use

biofeedbackdevicesthattransformsensormeasurementsfrom thebodyintosound,ablinkingLED,oranobservableshapeon ascreen[28–30].Biofeedbackprovidesdetailed information regarding the lengthening, shortening, and physical exer-tion of a muscle. It also allows comparing the data of a healthymuscle toanon-healthy onewhenperformingthe sameexercise.Biofeedbackdevicesaresometimescombined withelectromyography.Althoughthesedeviceshavebecome

portablerecently,olderdevicesareimmobileandcostly,and are mostly used inhospitals or rehabilitation centers [31]. Inaddition, mostdeviceseitherdonotevaluateapatient’s performanceortheyevaluatetheresultsusingsimple thresh-olding. Hence, they require a specialist or the patient to evaluate the feedback [31], both of whom may be highly subjective.Therefore,biofeedbackdevicescannotreplacean attendantspecialistmostofthetime.

Below,weprovideasummaryofstudiesaimingtoassess theaccuracyofphysicaltherapyexercisesorclassifythemas correct/incorrect:

Fergusetal.[5]proposeatele-rehabilitationsystemthat collectsand storesthepatient’smotiondata,utilizing body areaandsensornetworks,includinginertialsensors.The sys-tem virtuallysimulatesthepatient’s bodymotionsona3D humanbodymodelinrealtimeorusingthestoreddata.The physicianorphysicaltherapistmonitorsthemotionsremotely to evaluate the patient’s progress. This proposed solution isimpracticalanddoesnotsignificantlyimproveinspection timebecausethesystemprovidesnoinformationregarding the patient’s movement capability, movement accuracy, or progress.

Usingfivebody-worntri-axialaccelerometers,Tayloretal. [11]buildaclassifierthatlabelsincorrectlyperformed exer-cises forknee osteoarthritis, adegenerative disease of the kneejoint.Threeexercisesareperformedbyninehealthy sub-jects.Theexercisesareperformedinthecorrectwayaswell aswithparticularerrors.Severalfeaturesextractedfromthe accelerometerdataareusedintheAdaBoostclassifierto clas-sifytheexercisesascorrectorashavingaparticularerrortype. However,multipleerrorsarenotallowedbythemethodology and theclassificationaccuracyisabout70%inmostcases, whichisnotveryhigh.

In[12],anAndroidapplicationestimatingtheaccuracy(i.e., score)ofbalanceboardexercisesisdeveloped,usinga smart-phone’sinternalaccelerometerandmagnetometer.Acomplex rule-basedalgorithmisproposedtoobtainascorevalue clos-est tothe scoregivenbyanexpert;thedifferencebetween humanand automaticscoresisfound tobefewerthan 10 pointsinmorethan75%oftheexercisesona0to100scale. However,theproposedmethodologydoesnotyieldan opti-malsolution,anddifferentrule-basedscoringalgorithmsare usedfordifferentexercisetypes,whichmakesitdifficultto addanewexercisetothesystem.

In the MyHeart system[16], the accuracyofarm move-mentsforpost-strokerehabilitationisdeterminedusingstrain sensors.Healthysubjectswearingtight-fittinggarmentswith printedstrainsensorsimitatehowpost-strokepatientsmight performeachofsevenexercisetypescorrectlyandincorrectly underthesupervisionofaphysicianortherapist.Anexercise isconsideredcorrectlyperformedifthesimilaritybetweenthe recordedsignalandapre-recordedtemplate,calculatedbythe open-endDTW(seeAppendixA.2),exceedsathreshold[12]. Thesystemprovidesreal-timefeedbacktothepatientwithan averageclassificationaccuracyof85%.Themaindisadvantage ofthesystemisthedifficultyofputtingonthegarmentfora post-strokepatient,evenwithhelp.

In[4],strainsensorswornonthearmareusedtoprovide real-timefeedbacktopatientsundergoingmotor rehabilita-tion.Sevenexercisesareexecutedcorrectly andincorrectly

Fig.1–ComparisonoftheEuclideanandDTWdistancemeasures.(a)TheEuclideanmeasurecomparesthesamplesatthe sametimeinstants,whereas(b)theDTWmeasurecomparessampleswithsimilarshapestominimizethedistance. Retrievedfromhttp://upload.wikimedia.org/wikipedia/commons/6/69/EuclideanvsDTW.jpg.

atvariousspeedsbyahealthysubjectwearingaleft-handed sensorizedlong-sleevedshirt.Thesystemcheckswhetherthe measuredsignals“matchatmostonceaprefixofoneof sev-eralstoredreferencetemplates”[4]todetectwhichexerciseis beingperformed.Thedistanceobtainedbytheopen-endDTW algorithmisusedasasimilaritymeasurebetweenthestrain signalsandfedtoaone-nearest-neighborclassifierbothfor exerciserecognitionandcorrectness/errortypeclassification. Them-Physioplatform[10]classifiesphysical rehabilita-tion exercises using anaccelerometer. A smartphone with atri-axialaccelerometerisfixedtothepatient’s legorarm dependingonthetypeofexerciseheperforms.Thepatient firstperformstheexercisescorrectlyunderthesupervisionof aspecialistwhorecordsthereferencetemplates.Thesignals recordedfromtestexercisesarecomparedwiththereference templates using the ordinary DTW algorithm. One disad-vantageofthem-Physioplatformistheneedtodetermine thevaluesoffourparametersintheexercise-capturephase: theminimumandmaximumdurationofeach exercise,the samplingfrequencyoftheaccelerometer,andtheamountof smoothingappliedtothesignals.Thespecialistsetsthevalues bytrialanderror,whichmayeasilyalterthesystem perfor-mance.Whenusingthesystem,thepatientneedstotouch thescreentomarkthebeginningandtheendofeach exer-ciseexecution,whichisanotherdisadvantagebecausesome patients(e.g.,elderly,disabled,and/orstrokepatients)maynot beabletotouchthescreeninatimelymannerormayforget todoso.Thesystemprovidesfeedbackascorrect,incorrect,too short/longintimeforeachexecution.Thepatient’sstatisticsare uploadedtoacentralizeddatabasesothatthespecialistcan remotelycheckhisprogressthroughawebinterface.

In the previous studies,exercise executionsare usually croppedmanuallyandconsideredinanisolatedfashion;the subjectneedstomarkeach executionbypressingabutton orperformeach exercisewheninformed bythesystem by asoundoron-screennotification.Sequentialexecutionsand idletimeperiodsarenotconsidered.

Weproposeacompletelyautonomoussystemtodetectthe sequentialexecutionsofoneormultipleexercisetypesduring anexercisesession,classifytheexercisetype,evaluateeach executionascorrectly/incorrectlyperformed,andidentifythe errortypeandidletimeperiods,ifany[32].Duringthephysical therapysession,thepatientwearssmall,inexpensivemotion

sensorunits.Thesensorsarelightweightandcanbeeasily wornandcarried,makinghome-basedrehabilitationpossible. Thepatientfirstexecutestheexercisesunderthesupervision ofaspecialistwhorecordsreferencetemplates.Thepatient canthenperformtheexercisesanywhere,providedthathe properlywearsthesensors.Hedoesnotneedtopressa but-tontomarkthebeginningorendofanexecutionorselectthe exercisetypeheintendstoperform.Thesystemcomparesthe detectedexecutionswiththetemplatesrecordedwhileunder supervision,and quantifiestheir similarity.Wedevelopthe

multi-templatemulti-matchdynamictimewarping(MTMM-DTW) algorithmasanaturalextensionofDTW.UsingMTMM-DTW, multiple templatesequences ofdifferent durations can be searched ina test sequence ofany duration based on the DTWdissimilaritymeasure.Thesystemcanprovide statisti-calinformationabouttheexercisesessionatanydesiredlevel ofdetailtothespecialistandprovidefeedbacktothepatient. Theprimaryapplicabilityofthesystemistothe rehabilita-tionoforthopedicpatients.Thisisanearlyproof-of-principle studyinthepre-clinicalstageandthemethodologyhasnot yetbeendemonstratedinclinicalsettingsortobeclinically efficacious.

Thispaperisorganizedasfollows:InSection2,wedescribe

the MTMM-DTW algorithm. We present the

experimen-tal methodology and the results in Section3. In Section4, we provide a discussion ofthe results and related issues. Finally,wedrawconclusionsandindicatedirectionsforfuture researchinSection5.Weprovidebackgroundinformationon theDTWalgorithmanditsvariationsinAppendix.

2.

Multi-template

multi-match

DTW

algorithm

WedevelopanextensionoftheDTWalgorithmfordetecting multipleoccurrencesofmultipleexercisetypesinasequence recorded during a physical therapy session. This method makesitpossibletoclassifyexercisetypes,identifywhether anexerciseisperformedcorrectly(whileidentifyingthetype oferror,ifany),andcountthenumberofexercisesperformed. WeuseanapproachbasedonDTWwhichhasmore flex-ibilitythanusingabsoluteandEuclideandistancemeasures becauseitallowstimewarpingincomparingandmatching

Fig.2–TheflowchartoftheMTMM-DTWalgorithm.

similarparts oftwosequences (seeFig. 1foranexample), whichmaybebeneficialwhentimevariationsindifferent exe-cutionsofthesamephysicaltherapyexerciseareconsidered. Thepatientmayperformtheexerciseatdifferentpaces,which shouldbetoleratedtosomeextent.Ontheotherhand,the distancemeasureshouldnottoleratesignificantdifferences inamplitudethatmayoccur;forexample,ifthepatient per-formstheexercisesatalowamplitudeorholdsforsometime inthewrongposition.IfabsoluteorEuclideandistance meas-uresareused,bothtimeandamplitudevariationsaffectthe measureddistanceanditisnotpossibletoallowoneofthem whilepenalizing theother.In contrast,the DTWalgorithm naturallycompensatesfortimevariationsbutnotforchanges inamplitude,bothofwhicharemoredesirableinthisscheme. TheordinaryDTW algorithmmatches thefirst and last samplesoftwosequences, xand y(not necessarilyofthe samelength),toeachother,possiblywarpingthetimeaxesin betweentoobtainmaximumoverallsimilarity.Ontheother

hand,thesubsequenceDTW(SDTW)algorithmmatchesthe subsequence ofthe testsequenceythatismostsimilarto thetemplatesequencex.Ifthetemplatesequencemayoccur morethanonceinthetestsequence,itmaybedesirableto detect allofthe subsequencesofythat resemblex. Inthe

single-templatemulti-matchDTW(STMM-DTW)algorithmthat wedeveloped forthis purpose[33],the SDTWalgorithmis executedmorethan oncetodetectpossiblymultiple occur-rencesofasingletemplatesequencexinthetestsequencey.

Insomeapplications,itmaybenecessarytosearchfor multi-pletemplatesinatestsequence.Todetectpossiblymultiple occurrencesofeachoftheKtemplatesx(1)_,x(2)_,...,x(K)_iny,

weproposetheMTMM-DTWalgorithm(seeFig.2).

Algorithm1. MTMM-DTWalgorithm

1: M←length(y)

2: fork=1→Kdo

3: N(k)_←length(x(k)₎

4: Mleft(k)←M {Mleftisthemaximumnumber

ofsuccessiveunmatchedsamplesinyforthe

kthtemplatex(k)_}

5: ˜y(k)←y {thetestsequenceyreplicatedas

˜y(k)tobeusedforeachtemplatex(k)_}

6: endfor

7: whileMleft(k)≥˛N(k),∃k∈{1,...,K}do

8: compute SDTW(x(k)_{, ˜y}(k)_{) and save the}

SDTW(k)_distance,m(k) 1 ,andm

(k)

2 ∀k(m1andm2

arethefirstandthelastsampleindicesofthe matchedsubsequenceofy(k)₎

9: k∗←argmin_{k∈{1,...,K}} 1

N(k)SDTW(k) {find

matchingsubsequenceofthetemplatehaving theminimumDTWdistancepersampleofthe templatebynormalizingDTWdistanceswith respecttothetemplatelengths}

10: ifm(k₂∗)−m₁(k∗)+1≥˛N(k∗)_{{thelastmatched}

subsequenceissufficientlylong}

11: thenaddthelastmatchwithtemplate

num-berk*_tothelist

12: ˜y(k)(m(k₁∗):m₂(k∗))←∞,∀k {prevent the matchedsamplesfrombeingmatchedinthe DTWexecutionsthatfollowforalltemplates}

13: elseignorethelastmatch{thelastmatched

subsequenceistooshort}

14: ˜y(k∗)(m(k₁∗):m(k₂∗))←∞ {prevent the last matchedsubsequencefrombeingmatchedto thesametemplateintheDTWexecutionsthat follow}

15: endif

16: for k=1→Kdo

17: Mleft(k)← the maximum number of

suc-cessivefinite-valuedsamplesinyforthekth

templatex(k)

18: endfor

19: endwhile

Becausethedurationofeachtemplateis,ingeneral, dif-ferent,andtheDTWdistanceisthecumulativesumofthe

pairwisecostsbetweenthesamplesofthewarpedtemplate andtestsequences,theDTWdistancesneedtobenormalized bythelengthofthetemplatetobeabletomakeafair compar-ison.Forthisreason,whenweexecutetheSDTWalgorithm foragiventemplate(line8),wescaletheDTWdistancesof thesubsequencesmatchingthattemplatebythelengthofthe template(line9).1_{Then,weselectthesubsequencewiththe}

minimumnormalizedDTWdistance.

Anoptionalconditionmaybeimposedonthematchesif desired:Thelengthofthematchingsubsequencemustbeat leastsomefraction˛ofthelengthofthematchingtemplate

N(k)_{(intimesamples).Ifamatchedsubsequencesatisfiesthis}

lengthcriterion,wesavethematchingtemplatenumber,the correspondingsampleintervalinthetestsequence,andthe DTWdistanceinthelistofmatchedsubsequences.Then,we setthematchedsamplesofthetestsequenceyto∞to pre-ventthemfrommatchingagaintoanytemplateintheSDTW executionsthatfollow(line12).Ifthesubsequencedoesnot satisfythelengthcriterion,wedonotsaveit,butstillsetthe samplesiny corresponding tothis subsequence to∞,but

onlyforthemostrecentlyattemptedtemplatetopreventthesame subsequence from matchingto that template in the DTW executionsthatfollow(line14).Otherwise,exactlythesame subsequencewouldbematchedtothesametemplateinallof thefollowingiterationsbecauseSDTWalwaysfindsthebest matchingsubsequenceanddoesnotcontainanyrandomness. Thus,theavailablesamplerangeofyisdifferentforeach tem-plateandkeepschangingasweexecutethealgorithm.If a subsequencedoesnotsatisfythelengthcriterion,thereare twopossibilitiesinthefollowingDTWexecution:(1)another subsequencematchingthesametemplateatadifferent loca-tioniny(satisfyingornotsatisfyingthelengthcriterion)may befound;or(2)asubsequencematchinganothertemplate(at thesameordifferentlocation)inymaybefound.Therefore, thesubsequencethatdidnotsatisfythelengthcriterionfor aparticulartemplatemust beinvisibletothattemplateso thatthe templatemay match withsubsequencesat differ-entpositions[case(1)],butatthesametimevisibletoother templatestoallowfindinganothermatchatthesame posi-tion(becausethepreviousmatchisignored)[case (2)].This procedureisrepeateduntil,foreachtemplate,themaximum numberofsuccessivelyavailable(finite-valued)samplesiny

becomessmallerthansomefraction˛ofthe lengthofthe template.Afterthispoint,itisnotpossibleforanyremaining subsequencetosatisfythelengthcriterionevenifitmatches. IntheMTMM-DTWalgorithm,thematchedsubsequences (associatedeitherwiththesameordifferenttemplates)iny

arenotnormallyallowedtooverlap.Thisrestrictionmaybe relaxedtoallowsomeoverlapbyreplacingthesamplerange (m₁(k∗):m₂(k∗))inlines12and14by( ˜m(k₁∗): ˜m₂(k∗)),where ˜m(k₁∗)= (1−ˇ)m(k₁∗)+ˇm₂(k∗)and ˜m(k₂∗)=ˇm₁(k∗)+(1−ˇ)m₂(k∗),withˇ∈(0, 1] being the ratio of the matched subsequences that are allowedtooverlapinthebeginningandattheendwithother subsequences.Ifˇ=0,overlappingisnotallowed.

1 _{Thenormalizationisapproximatebecauseitdependsonthe}

steppatternandthevaluesofthelocalweights,wh,wv,andwd,

usedinDTW[34],seeEq.(9)inAppendix.

Fig.3–XsensMTxunit[36].

The advantagesofthe MTMM-DTWalgorithm are that: (1) thenumber oftemplates,the number ofsubsequences, theirpositionsonthesampleaxis,andthelengthofthetest sequenceyneednotbeknown;(2)thetemplatesandthetest sequencexandymaybemulti-dimensional(seeAppendix A.1);(3) trivialfalsematcheswithasubsequenceoflength muchshorterthanthematchingtemplatemaybeeliminated byimposingaconditiononlength;(4)theamount of over-lapbetweenthematchedsubsequencescanbeadjustedas desired;and (5)alloccurrencesofthetemplatesinthetest sequencecanbedetectedandclassified.Thealgorithmcan beusedforseveraldifferentpurposes:classifyingasequence givenmultipleexercisetemplates(exercisepattern recogni-tion),detectingtheoccurrencesofsingleormultipletemplates inthetestsequencewiththeirtimeinstancesanddurations, estimatingthenumberofrepetitionsofalltemplatesinthe test sequence (counting the exercises),or all three.Values ofthelocalweightswh,wv,and wd (seeAppendix)and the

parameters˛andˇoftheMTMM-DTWalgorithmcanbetuned totheapplication.

Arelatedalgorithmisproposedin[35]torecognize multi-pletemplatesinsequentialrecordingsofdancemotions.The datasetisacquiredusingamarker-basedopticalmotion cap-turesystem.Itisassumedthattherecordingsconsistofdance figuresthatfollowoneanother,wherethereareneitheridle periodsnorunmatchedintervals.Therecordingissegmented usingtheopen-endDTWthatfindsthebest-matching tem-plateandterminatesthesegment.Thus,thatalgorithmisnot asflexibleastheMTMM-DTWalgorithmproposedhere.

3.

Experimental

verification

3.1. Experimentalsetup

Weemployedwearablemotionsensorunits tocapturethe exercises performed by five subjects who were free from movement disorders. This study was approved by Bilkent University EthicsCommitteeforResearchInvolvingHuman Participantsandtheparticipantsgaveinformedconsenttothe work.WeusedfiveMTxunits(Fig.3)manufacturedbyXsens Technologies[36],eachcontainingthreetri-axialdevices:an accelerometer, a gyroscope, and a magnetometer. We cal-ibrated the sensors using the system’s default calibration

Fig.4–Sensorplacementonthehumanbody.(a)Thefirstand(b)secondconfigurationsaredesignedforlegandarm exercises,respectively.Thearrowperpendiculartothesensorunitindicatesthezdirection.Thecablegoesintothesensor unitinthedirectionofthexaxis.Theyaxiscanbeeasilydeterminedgiventhatright-handedcoordinatesystemsareused.

procedureand selectedasamplingfrequencyof25Hz.The operating ranges ofthree of the accelerometers (units 1–3 inFig.4)were±18g,andtwoofthem(units4and 5)were ±5g,whereg=9.807m/s2 _{isthe gravitationalconstant.The}

operatingrangesofthegyroscopesandmagnetometerswere ±1200◦_{/sand±75␮T,respectively.}

Becausetheexercisesweconsiderinthisstudyinvolveonly armoronlylegmovements,weusedtwosuitablydesigned sensorconfigurationstocapturethesemotions(seeFig.4for details).Thesensorunitscan befreelyconfigured to prop-erlycapturetheexercisemovementsandunitscanbeeasily addedorremovedwhenneeded.Thesystemdoesnotrequire prior knowledgeof the sensor configuration; however, the sameconfigurationshouldbeusedwhilerecordingthe tem-platesandwhileexercising.Thesystemishighlyflexibleand modularintheserespects,especiallywhen comparedwith rule-basedsystemsthatmodelthehumanbodytoevaluate theexercises.

Because each unit contains three tri-axial devices, 45(=9axes×5units)discrete-timesequenceswererecorded

during each experiment. We normalized the acquired

sequencesseparatelyforeachsensortypesothat,for exam-ple,accelerometersequencesinthewholedatasethaveunit variance.Thesameholdsforthegyroscopeand magnetome-tersequences.

3.2. Physicaltherapyexercises

Theeightexercisesweconsiderinthisstudyweresuggested and approved by a physical therapy specialist [37]. They are the most commonlyassigned exercisesto patients for

orthopedic rehabilitation(seeFig. 5). Abrief descriptionof eachisprovidedbelow:

1. Sittingonahighflatsurface,raisetherightleg,holdfor 5s,keepingtherightkneestraight,andreturntotheinitial position.

2. Sittinguprightonastoolwitharmshangingdownwards, bendtheupperbody30◦ forward,holdfor5s,andreturn totheinitialposition.

3. Lying flat on the back on a flat surface, raise the right legfromthehipjoint,keepingtherightkneeandleftleg straight,holdfor5s,andreturntotheinitialposition. 4. Lyingflatontheleftsideonaflatsurface,raisetheright

legfromthehipjoint,keepingtherightkneeandleftleg straight,holdfor5s,andreturntotheinitialposition. 5. Lyingfacedownonaflatsurface,raisetherightlegfromthe

hipjoint,keepingtherightkneeandtheleftlegstraight, holdfor5s,andreturntotheinitialposition.

6. Sittingonachair,holdinga1kgweightintherighthand, extend the right arminfront ofthe body tojust above therightkneewiththepalmfacingupwards.Bendingthe elbowjoint,raisetheweightuntiltheforearmis perpen-dicularwiththethigh,holdfor5s,andreturntotheinitial position.

7. Standinguprightwiththerightarmholdinga1kgweight andhangingstraightdown,raisetheweighttotheright sidefromtheshoulderjointtoahorizontalpositionwhile keepingtheelbowjointstraight,holdfor5s,thenreturnto theinitialposition.

8. Lying facedown on a raisedflat surface, hang the right armoverthesideattheelbow.Raisetherightforearmto

Fig.5–(a–h)Theeightphysicaltherapyexercises,labeled1–8,consideredinthisstudy.Ineachexercise,thesubjectmoves hisleg/armfromtheinitialposition(insolidlines)tothepositionindottedlines,holdsthepositionfor5s,andreturnsto theinitialposition.

straightentheelbow,holdfor5s,thenreturntotheinitial position.

3.3. Experimentalprocedure

Wedesignedtheexperimentstotestwhethertheproposed MTMM-DTWalgorithmisabletodetecttheexerciseswithin along recordingofa typicalexercisesession while identi-fyingidletimeperiods,classifyeachexercisetype,estimate thenumberofrepetitionsperexercise,determinewhetheran exerciseisperformedcorrectly,andidentifytheerrortype,if any.

Two common typesoferrorsthat patientsmakeduring exercisesessionsare:

• Performingthemovementstoofast;patientsdonothold the position for the necessary amount of time because theywanttoquicklycompletethe numberofrepetitions required.

• Performingtheexercisesatalowamplitudewithout com-pletelyexecutingthemovement.Thiserrormaybecaused byphysicalincapability(suchasafterastroke)orby negli-gence,sloppiness,carelessness,etc.

Table1–Physicalpropertiesofthesubjectswho participatedintheexperiments.

Subject Gender Age Weight(kg) Height(cm)

1 Female 55 73 169 2 Male 61 85 180 3 Male 23 95 180 4 Female 48 55 158 5 Male 53 98 175 Average 60%male 48 81.2 172.4

These errorswill bereferred toas type-1and type-2 errors,

respectively.Therefore,threeexecutiontypesareconsidered ineachexperiment:(1)correct,(2)withtype-1error,and(3) withtype-2error.

First, all three execution types of each exercise were recorded for each subject while under supervision. These recordings constitute our reference data set. Altogether,

K=24(=3executiontypes×8exercisetypes)templateswere recorded,denotedbyx(1)_,x(2)_,...,x(K)_{,foreachsubject.The}

exercisesthatapatientperformsnotundersupervisionwill becomparedwiththesetemplates.If apatientisunableto perform the exercises properly because his muscles have notsufficientlydevelopedorifhefeelstoomuchpain,the specialistcanapplyexternalforcestohelphimexecutethe exercises[37].

Systematicexperimentsconductedinthetestphase sim-ulateatypicalreal-worldexercisesession.Eachofthe five subjects(seeTable1fortheirphysicalproperties),performed eightexperiments, each corresponding toone ofthe eight exercise types. A testsequence y is a recordingof a sub-ject performing oneofthe assigned exercises. Thesubject repeatstheexercisecorrectly10times, andthenwaitsuntil the100thsecondoftheexperiment.Whilewaiting,the sub-jectneitherperformsanexercisenormovestoomuch,hence, isconsideredtobeidle.Atthe100thsecond,hestarts repeat-ingthesameexercise10timeswithatype-1error,andagain waitsidly,this timeuntilthe160thsecond.Then,the sub-jectexecutestheexercise10timeswithatype-2error,and theexperimentterminateswithnomoreidletimeperiods. Therefore,anexperimentinthetestsetconsistsof10 execu-tionsofanexerciseforeach executiontype(30 executions in total), and two idle time intervals of variable duration betweenthethreegroupsof10exercises.Becausethereare 5×8=40experiments,eachcontaining30executions,thetest setcontains5×8×30=1200exerciseexecutionstogetherwith 80(=2×40)idletimeperiods.

Fig. 6 illustrates typical reference and test recordings. Part (a) ofthe figureshows the outputs ofthe sensors in unit 2 belonging to the templates for the three execution types.Manuallyselected templatesinaseparate recording arehighlighted.Part(b)depictstheexperimentforexercise 1,performedbythethirdsubject.Thetwoidletimeperiods areobservedbetweentheactiveperiods.

Someexecutionsintherecordeddatasetmaynotexactly belongtooneofthethreeexecutiontypes.Forexample,an executionofanexercisewithatype-1errormaynotbe suffi-cientlyfasttobeclassifiedintothiscategory,butatthesame timeitmaynotbeexecutedsufficientlyaccuratelytobe clas-sifiedascorrect;thatis,somesamplesinthedatasetmaynot

belongtoanyclass.Furthermore,thesubjectsmaynotalways performthecompletesetofexercisesproperlyduetofatigue, lackofconcentrationorinterest,etc.Wedidourbesttoreduce theseproblemsbyrepeatingthewholeexperimentifsuchan executionerrorwasclearlynoticeable.

3.4. Movementdetectionandclassificationusing

MTMM-DTW

WeappliedtheMTMM-DTWalgorithmdescribedinSection2 to the acquired data set to detect, classify, and evaluate physical therapy exercises in the recorded test sequences. Toperform thetests foragivensubject, from whomeight testsequenceswererecorded,weconsiderthe24templates recordedfromthatsubjectasreference.Weuseduniformlocal weights(i.e.,wh=wv=wd=1;seeAppendix)andselectedthe

parameter˛=0.5toimposetherestrictionthateachmatched subsequenceisatleasthalfthelengthofthematching tem-plate.Wesettheˇparameter(describedinSection2)to0.05 toallowthematchedsubsequencestooverlaptoupto5%of theirdurationsinthebeginningandattheend.Inaddition, wediscardedmatchingsubsequenceswithanormalizedDTW distancelargerthan10(persampleofthematchingtemplate) because theywere not sufficientlysimilar tothe matching template.

3.5. Experimentalresults

WeevaluatedtheperformanceoftheproposedMTMM-DTW algorithmindifferentways:thepercentageofexecutionsthe algorithmdetected,thealgorithm’saccuracyinclassifyingthe exerciseand/ortheexecutiontypes,thesensitivityand speci-ficityvalues,thefalsealarm(FA)andmisseddetection(MD) rates.Thenumberoftrueandfalsepositivesandnegativesare neededtocalculatethesequantities[32].Thenumberoftrue positivesandfalsenegativescanbedirectlyobtainedbecause forthesecases,thecorrectclassistrue;i.e.,thereexistsan exerciseexecution.Thenumberoffalsepositives(incorrectly recognizingexerciseexecution(s)duringanidletimeinterval) issimplythesameasthenumberofFAs.However,thenumber oftruenegatives(idletimeintervalsthatarenotrecognized asanexerciseexecution)isnotdirectlyavailablebecauseidle timeintervalsarenotcountable.Thenumberofsamplesin theidleintervalscanbeestimatedbydividingtheinterval’s durationbythedurationofthecorrectlyexecutedtemplate oftheexerciseineachexperiment,obtainingthenumberof negative(idle)samples.Then,thenumberoftruenegatives iscalculatedbysubtractingthenumberofFAsfromthetotal numberofnegatives.

InFig.7,thedetectedexecutionsintheeightexperiments (eachactuallycontaining30executionsofanexerciseaswell astwoidletimeintervals),areshownasbarsalongthetime axisforoneofthesubjects.Thewidthsofthebarsindicatethe durationsoftheexecutions,andtheheightsshowthe normal-izedDTWdistancebetweentheexecutionsandthematching template.Thatis,the shorterthebar,themoresimilarthe matchedsubsequenceandthematchingtemplate.FourFAs occur inthe idletimeintervals:twointhefirst,oneinthe fifth,andoneintheseventhexperiment.

0 5 10 15 20 25 30 35 40 45 50 −20

0 20

subject 3, exercise 1 (templates), unit 2

time (s) acceleration (m/s 2) 0 5 10 15 20 25 30 35 40 45 50 −5 0 5 time (s)

angular rate (rad/s)

x y z 0 5 10 15 20 25 30 35 40 45 50 −1 0 1 time (s)

magnetic field (relative)

(a)

0 50 100 150 200

−20 0 20

subject 3, exercise 1, unit 2

time (s) acceleration (m/s 2) 0 50 100 150 200 −5 0 5 time (s)

angular rate (rad/s)

x y z 0 50 100 150 200 −1 0 1 time (s)

magnetic field (relative)

(b)

Fig.6–Recordingofthetemplatesandtheexperimentforexercise1performedbysubject3.(a)Thethreetemplates (highlightedwiththicklines)forcorrect,type-1error,andtype-2errorexecutiontypesofexercise1,(b)theexperiment consistingof10repetitionsofexercise1forthethreeexecutiontypesandtwoidletimeperiodsinbetween.Onlythe sensoroutputsofunit2,themostimportantoneforthisexercise,areshown.

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 1 1 7 ( 2 0 1 4 ) 189–207

Table2–Cumulativeconfusionmatrix,whichcontainsthethreeexecutiontypes(A:correct,B:type-1error,C:type-2error)ofalleightexercises(1–8)summedupfor allfivesubjects.ThenumberofMDsandFAsareshowninanadditionalcolumnandrow,respectively.Theelementsinthe3×3blocksonthediagonalcorrespondto correctexercisetypeclassifications.

classes Estimated 8C 8B 8A 7C 7B 7A 6C 6B 6A 5C 5B 5A 4C 4B 4A 3C 3B 3A 2C 2B 2A 1C 1B 1A MD total True  1A 50 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 50 1B 2 39 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 50 1C 0 0 50 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 50 2A 0 0 0 44 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 50 2B 0 0 0 3 45 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 50 2C 0 0 0 7 4 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9 50 3A 0 0 0 0 0 0 43 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 50 3B 0 0 0 0 0 0 3 43 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 50 3C 0 0 0 0 0 0 0 6 34 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 50 4A 0 0 0 0 0 0 0 0 0 41 7 0 0 0 0 0 0 0 0 0 0 0 0 0 2 50 4B 0 0 0 0 0 0 0 0 0 1 49 0 0 0 0 0 0 0 0 0 0 0 0 0 0 50 4C 0 0 0 0 0 0 0 0 0 5 6 17 0 0 0 0 0 0 0 0 0 0 0 0 22 50 5A 0 0 0 0 0 0 0 0 0 0 0 0 42 4 1 0 0 0 0 0 0 0 0 0 3 50 5B 0 0 0 0 0 0 0 0 0 0 0 0 5 21 0 0 0 0 0 0 0 0 0 0 24 50 5C 0 0 0 0 0 0 0 0 0 0 0 0 5 7 25 0 0 0 0 0 0 0 0 0 13 50 6A 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 48 2 0 0 0 0 0 0 0 0 50 6B 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 45 0 0 0 0 0 0 0 2 50 6C 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 1 40 0 0 0 0 0 0 2 50 7A 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 49 1 0 0 0 0 0 50 7B 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 47 0 0 0 0 0 50 7C 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 1 39 0 0 0 0 50 8A 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 50 0 0 0 50 8B 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 37 0 7 50 8C 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9 2 39 0 50 FA 4 6 0 0 16 3 1 4 1 1 2 0 9 2 4 5 4 0 1 2 0 6 0 3 74 Total 56 45 59 54 70 34 47 55 37 48 64 17 61 34 30 63 52 40 63 51 39 71 39 42 103 1274 classes

50 100 150 200 0 2 4 6 8 10 time (s) DTW distance Experiment 1 of subject 5 50 100 150 200 0 2 4 6 8 10 time (s) DTW distance Experiment 2 of subject 5 50 100 150 200 0 2 4 6 8 10 time (s) DTW distance Experiment 3 of subject 5 50 100 150 200 0 2 4 6 8 10 time (s) DTW distance Experiment 4 of subject 5 50 100 150 200 0 2 4 6 8 10 time (s) DTW distance Experiment 5 of subject 5 50 100 150 200 0 2 4 6 8 10 time (s) DTW distance Experiment 6 of subject 5 50 100 150 200 0 2 4 6 8 10 time (s) DTW distance Experiment 7 of subject 5 50 100 150 200 0 2 4 6 8 10 time (s) DTW distance Experiment 8 of subject 5

Fig.7–Detectionandclassificationofexerciseexecutionsperformedbysubject5intheeightexperimentscorresponding totheeightexercises.Eachdetectedexecutionisshownasabarwhosewidthistheexecution’sdurationandwhoseheight isthenormalizedDTWdistancebetweenthedetectedsubsequenceandthematchingtemplate.Black/gray-filled

(black/blueonline)barsindicatecorrectly/incorrectlydetectedexecutionsofthecorrectexercisetype.Unfilled(redonline) barsindicateincorrectlydetectedexercisetypes.(Thetrueexecutions,MDs,andwhetherthematchedsubsequencesare FAsarenotillustrated.)

Table 2shows the cumulative confusion matrix forthe eightexercisetypes(1–8),eachwiththreeexecutiontypes(A, B,andC). Thismatrixisobtainedbysummingupthe con-fusionmatricesofthefivesubjects.Thelastcolumnandthe lastrowindicatethenumberofMDsandFAsineachclass, respectively.Thereare103MDsand74FAsintotal.

Althoughtheproposed system simultaneously classifies theexerciseandexecutiontypesinto24classes(1A,1B,1C, 2A,...,8C),exerciseandexecutiontypeclassificationcanbe consideredseparately.Thematrixelementscorrespondingto thecorrectexercisetypeclassificationsareemboxedinthe con-fusionmatrix.Theseelementscontaincorrectandincorrect executiontypeclassifications.Combiningthethreeexecution

types(A,B,andC)ofeachexercise,the8×8confusionmatrix oftheeightexercisesisobtainedandpresentedinTable3. Weobserve thatthesystem neverincorrectlyclassifiesthe exercisetypethattheexecutionbelongsto,butitmaymiss someexecutionsordetectsomeadditionalones,resultingin MDsandFAs.ThetrueclassoftheFAsisthenegativeclass, meaningthattheydonotbelongtoanyoftheeightexercise types.However,theyhavebeenclassifiedpositivelyasifthey dobecausewhilethepatientremainsidle,notperformingany exercises,the systemmayrecognizeanumber ofexercises becauseofthenoiseonthesequences.

TheresultsaresummarizedforthefivesubjectsinTable4 andfortheeightexercisesinTable5.Thesystemcorrectly

Table3–Cumulativeconfusionmatrixofalleightexercisetypes(1–8)summedupforallfivesubjects.Thenumberof MDsandFAsareshowninanadditionalcolumnandrow,respectively.

Estimatedclasses Total

1 2 3 4 5 6 7 8 MD 1 150 0 0 0 0 0 0 0 0 150 2 0 139 0 0 0 0 0 0 11 150 3 0 0 133 0 0 0 0 0 17 150 4 0 0 0 126 0 0 0 0 24 150 Trueclasses 5 0 0 0 0 110 0 0 0 40 150 6 0 0 0 0 0 146 0 0 4 150 7 0 0 0 0 0 0 150 0 0 150 8 0 0 0 0 0 0 0 143 7 150 FA 10 19 6 3 15 9 3 9 74 Total 160 158 139 129 125 155 153 152 103 1274

Table4–Experimentalresultssummarizedpersubject.

Subject Total

1 2 3 4 5

Numberofdetectedexecutions 194 255 216 216 244 1125

Numberofactualexecutions 240 240 240 240 240 1200

Percentage(%) Overall(%)

Accuracyofexerciseclassification 88.42 93.14 93.17 93.86 97.64 93.46

Accuracyofexerciseandexecutiontypeclassification 80.69 89.29 88.42 88.48 94.62 88.65

Sensitivity 80.00 99.58 88.75 89.17 99.58 91.42

Specificity 97.35 88.81 96.86 97.86 96.33 95.09

MDrate 20.00 0.42 11.25 10.83 0.42 8.58

FArate 2.65 11.19 3.14 2.14 3.67 4.91

Table5–Experimentalresultssummarizedperexercisetype.

Exercisetype Total

1 2 3 4 5 6 7 8

Numberofdetectedexecutions 160 142 135 127 115 149 151 146 1125

Numberofactualexecutions 150 150 150 150 150 150 150 150 1200

Percentage(%) Overall(%)

Accuracyofexerciseclassification 97.01 90.71 91.92 91.35 83.07 96.36 99.26 94.84 93.46 Accuracyofexerciseandexecutiontypeclassification 93.54 84.29 87.93 85.67 77.01 92.74 94.73 89.57 88.65

Sensitivity 100.00 92.67 88.67 84.00 73.33 97.33 100.00 95.33 91.42

Specificity 89.15 95.65 96.56 98.29 94.44 95.65 98.83 94.97 95.09

MDrate 0.00 7.33 11.33 16.00 26.67 2.67 0.00 4.67 8.58

FArate 10.85 4.35 3.44 1.71 5.56 4.35 1.17 5.03 4.91

detects1125ofthe1200executionsinthedataset, under-counting the exercises by−6.25%. As observed in Table4, the number ofdetectedexecutions foreach subjectvaries between 194 and 255, where the correct number is 240. Table5showsthatthenumberofdetectedexecutionsranges between115and160foreachoftheeightexercises,wherethe correctnumberis150.Thevariationintheexercisesiscaused by the fact that some exercises inherently contain move-mentsofloweramplitudecomparedtoothers.Forexample,in exercises4and5,thelegmovementsaresmallbecauseofthe difficultyoftheexercise,andthesystemcanonlyrecognize 85%and77%oftheexercises,respectively.Consideringthat thealgorithmtriestodetectthecorrectanderroneous move-mentsoftwotypes(executedquicklyoratalowamplitude), itismoredifficultforittorecognizeexecutionsperformedat alowamplitude.Forthisreason,notonlydoesthenumber ofMDs increase, but also the number ofFAs, because the

templatesthat belongtothe low-amplitude executionsare moresimilartothesequencesintheidletimeintervals.

Theoverallaccuracyofthesysteminexerciseclassification onlyis93.46%,whereasinsimultaneousexerciseand execu-tiontypeclassificationitis88.65%.2_{Fortheformer,FAsand}

MDs decreasethe accuracy,whereas forthelatter, in addi-tiontoFAsandMDs,incorrectexecutiontypeclassifications alsodecreasetheaccuracy.InTables4and5,weobservethat theperformanceofthesystemvariesconsiderablyamongthe subjectsandtheexercises.Theaccuracyinexerciseand exe-cution typeclassificationvaries between80.69%(subject 1) and94.62%(subject5)forthefivesubjects,and77.01% (exer-cise 5) and 94.73%(exercise6) for theeight exercises.The

2 _{Incalculatingtheaccuracyvalues,weconsiderboththe}

exerciseexecutionswhosetrueclassispositiveandtheidletime intervalswhosetrueclassisnegative.

accuracyofthesystemcanbefurtherimprovedbytuningthe parameters(suchas˛andˇintheMTMM-DTWalgorithmand thelocalweightsintheDTWalgorithm)forspecificexercise types.

Theoverallsensitivityandspecificityvaluesare91.42%and 95.09%,respectively,andtheoverallMDandFAratesare8.58% and4.91%,respectively,forthewholedataset(Tables4and5).

3.6. Testingthesystemwithunknownexercise

movements

Intheproposedsystem, weconsiderthreeexecutiontypes ofeachexercise,twoofthembeingthemostcommontypes ofincorrectexecutions[28,37].However,thepatientmay per-formtheexerciseinanincorrectwaythatmaynotbelongto oneofthesecategories,orhemayperformacompletely differ-entactivitybetweenexercises,suchasrepositioninghisbody orstretching.Whenanunknownmovementisperformed,the systemisideallyexpectednottodetectanyexerciseexecution ordetectonewitharelativelylargeDTWdistancethatcanbe easilydiscardedbysimplethresholding.Sincethereisawide varietyofmovementstoconsiderandtheoutputdependson howdifferenttheunknownmovementsarefromtherecorded templates,itisratherdifficulttotestthisaspectofthesystem. To validate the robustness of the proposed system to unknownmovementsorexecutiontypes,weusethe leave-one-exercise-out(L1EO)cross-validationapproach:Weexecute theMTMM-DTWalgorithmbyleavingoutthethreetemplates (correct, type-1,and type-2 errors)of onetype of exercise atatimeforeachsubject,andcomparingtherecordedtest sequenceofthatexercise(with30executions)withthe tem-platesoftheremainingsevenexercisetypesofthatsubject(21 templates).Asmentionedabove,ideally,thesystemshould notdetectanyexecutions.Notethatintheproposedsystem (seeSection3.4),wediscardmatchingintervalswith normal-izedDTWdistanceslargerthan10.Usingthesamethreshold valueof10inL1EOandconsideringeachdetectionwithaDTW distancebelowthislevelasanFAresultsin10falsealarmsout ofthe1200executionsinthedataset.TheverylowFArate (0.83%)demonstratestherobustnessoftheproposedsystem tounknownmovements.ThenumberofFAsistabulatedin Fig.8(a)foreachexerciseofeachsubject.Thehistogramof thenormalizedDTWdistancesofthedetectionsisprovided inFig.8(b).Thethresholdlevelof10separatesthehistogram intotwodistinctpartsandthedistancevaluesnearesttothe threshold(5.44and14.88)arerelativelyfar fromeachother andfromthethreshold,indicatingthatthenumberofFAsis notverysensitivetothethresholdlevel.

3.7. Computationalcomplexity

Thecomputational complexity ofthe DTWand the SDTW algorithms is directly proportional to the product of the lengthsofthe twosequences [42]. Thisisalsotrueforthe

STMM-DTWand MTMM-DTWalgorithmsbecause bothare

basedontheSDTWalgorithm.Thecomputational complex-ityoftheMTMM-DTWalgorithmisalsodirectlyproportional tothenumberoftemplates.However,thealgorithmrepeatsa particularprocessuntilaconditionissatisfied;thus,its com-putationalcomplexitydiffersfordifferentsequences ofthe

Fig.8–(a)NumberofFAsforeachexerciseofeachsubject inL1EOintabularform.(b)Histogramofthenormalized DTWdistancesofallthedetectionsinL1EO.Detections withDTWdistancesbelowthethresholdareconsidered FAs.

same length. Weimplemented the MTMM-DTW algorithm

onalaptopwithaquad-coreprocessorat2GHz (IntelCore i7 2630QM) andwith8GB ofRAM, running32-bitMATLAB. Although the calculations are completed with some delay becausethecomputationalcomplexityoftheproposed algo-rithmissomewhathigh,thealgorithmcanbemodifiedtorun innearrealtime.Thecomputationalefficiencywouldincrease byabout200timesiftheSDTWalgorithmwereprogrammedin C.Itisdemonstratedin[38]thatifagraphicalprocessingunit (GPU)isusedinadditiontoacentralprocessingunit(CPU),the runningtimefurtherdecreases,upto29times.Whenafield programmablegatearray(FPGA)isusedinsteadofaPC,the SDTWalgorithmwouldrunupto4500timesfasterthanits versionprogrammedinC,whichmakesreal-time implemen-tationpossibleevenonalow-costPCoraportabledevice[38]. Areal-timeimplementationofaclassificationscheme,based ontheordinaryDTW,ispresentedin[39],wheretheclassifier performanceisevaluatedbasedonexperimentaldata.

Sincethe computationalcomplexity ofthe MTMM-DTW algorithmisdirectlyproportionaltothenumberoftemplates used, tofurther improve the computational efficiency, the

numberofreferencetemplatescanbereducedbyremoving thetwoincorrectexecutiontypesoftheexercises.

4.

Discussion

Themethodologyproposedheredoesnotrequirepreviously definedrulesandisflexibleandadaptablebecause(1)thereare norestrictionsonthenumberofexerciseexecutions,number oftemplates,ordurationsofthesequences;(2)theexecutions maybeallowedtooverlapwitheachotheratadesiredlevel;(3) eachofthetemplatesequencesmaybeofanylengthbecause theDTWdistanceisnormalizedbythetemplateduration;and (4)thesequencesmaybeone-ormulti-dimensional.

Themainadvantagesoftheproposedsystemarethe fol-lowing:(1) Thepatientdoes notneed topress abutton to indicatethebeginningortheendoftheexerciseexecutions, waitforasignaltostartanexecution,orselecttheexercise typeheintendstoperform.Thisfeaturemakesthesystem usableinaphysicaltherapysession ofanyduration, theo-reticallyconsistingofanunlimitednumberofexecutions.(2) Thereisnoneedforthesystemtobeconfiguredfor differ-entexercisetypesorpatients;itdoesnotrequireinformation onthephysicalpropertiesofthesubjectsnorasetofrules todefine the exercisetypes,but onlyneeds arecordingof thetemplateexecutionsofeachexerciseperformedbyeach patient.(3)Thesystemworksindependentlyofthenumber, type,and configuration ofthe wearablesensors; thesecan bechosenasneeded.Hence,any sensorconfigurationthat canproperlycapturethemovementscanbeused.Theonly requirementisthattheconfigurationmustremainthesame whenrecordingthetemplatesequencesandwhenusingthe system.Itiseasytoaddanewexercisetype,anewsensor, orforanon-experttochangesomeorallofthesensors;i.e., thewholesystemdoesnothavetobere-programmedforsuch modifications.

Thelastfeaturerepresentsasignificantadvantageoverthe manystudiesthatemploy3Dhumanbodymodelstoevaluate exercises[5,18,24],wherethenumber,types,positions,and orientationsofthesensorsonthebodyarepre-determined. Whenthisisthecase,thesystemmustbemodified consid-erablyifachangeinsensorconfigurationoccurs.Inaddition, thesetypesofsystemsoftenrequirerule-based algorithms (or decisiontrees) to evaluatethe correctness of the exer-cise executions, which makes them extremely difficult to reconfigurefornewlyaddedexercisetypes.However,inour system,ifanewexerciseneedstobeadded,theonly require-mentistorecordthetemplatesofthepatientperformingthe differentexecutiontypesofthatexercise,whichthe physi-cian/therapistcaneasilydoinsteadofhavingtorelyonthe engineerwho developedthe system. Consideringthe wide varietyofphysical therapyexercises assigned fordifferent disorders,theunsupervised natureofouralgorithmallows flexibilityinexercisetypeandinhardwaretypeand configu-ration.Intheexperiments,weobservedthatthisautonomous methodologycauses rareMDsand FAs,whicharetolerable consideringthatthesystemismucheasiertousecompared tothesystemsdevelopedinstudiessuchas[10].

Ours is a very early proof-of-principle study in the pre-clinical stage and the methodology has not yet been

demonstrated in clinical settings or to be clinically effica-cious.Theprimaryapplicabilityofthesystemisforpatients undergoingorthopedicrehabilitationwhoperiodicallyrepeat theprescribedexercisesinthesamewayduringthe rehabil-itationprocess.Neurologicalrehabilitationusuallyrelieson one-to-one interaction between the patientand the physi-cian/therapist. The assigned exercises are more directly focusedonimprovingfunctional activities(e.g.,graspingor squeezinganobject,holdingacup)andmoreofteninvolve applicationofforcestothepatient’sbodymanuallyorusing roboticdevices.However,suchpatientsmayalsobenefitfrom the system proposed here. Whilethey may not beable to completelyexecutetheexercisemovementsatfirst,theywill likelyprogress,andastheydoso,thenormalizedDTW dis-tance betweenthe motion performedand its correctform may beusedasameasureofdiscrepancyorimprovement. Forpatientsinthissecondgroup,thesystemcanbemodified asfollows tokeeptrack oftheirprogressduring rehabilita-tion:Insteadofrecordingonlyonecorrecttemplateforeach exercisethatrepresentsthefinalgoal(andpossiblysome erro-neoustemplates),templatesthatcorrespondtointermediate goal points (suchas partial,low-amplitude, orslow execu-tions of anexercise)may be recorded.Thetemplates may be updated as needed according to the patient’s progress; forinstance,someintermediatetemplatesmayberemoved overtime.Thisway,theprogressofthepatientcanbe moni-toredquantitativelytoprovidefeedbacktothepatientandthe physician/therapist.Nothavingtopressabuttoncanalsobe advantageousforpatientsundergoingneurological rehabili-tation,especiallythosewithupperextremityproblems.Even healthypatientsmayfinditdifficulttopressthebuttonina timelymannerwhichmaydegradethesystemperformance.

Thesystemcanprovidefeedbackatanydesiredlevelof detailintheformofaprogressreport(e.g.,thetotalnumber of executionsthe patient performs, the percentage of cor-rectexecutions,theaccuracyoftheexecutions,whetherthe patient reachedthe targeted number ofexercises, and the active/idletimeintervalsinthephysicaltherapysession).To save time, feedback can alsobe inthe form ofalerts that informphysicians/therapistsonlywhenneeded,forexample, whentheactivitylevelofthepatientistoolow,themajority oftheexecutionsareincorrectortoofast,etc.Providing feed-backinthisformwilltakemuchless timeofthephysician thanhavingtomonitoreachpatientonanindividualbasis.

Ifanexerciseisperformedtooquicklyortooslowly,the detected executionwill beshorter/longer comparedto the reference template,respectively,whichwillnaturally affect theresultingcost(theDTWdistance)inthealgorithm.Ifthe exercise isperformed tooslowly,the DTW distanceis cal-culated basedoncomparingalargernumberofsamplesof thetemplateandtestsequences,andtheresultingDTW dis-tancewillbeslightlylargerduetodifferencesinthemovement andnoise.Sincewechoosethevaluesofthehorizontaland diagonallocalweights(whandwd)asthesame,wedonot

addi-tionallypenalizelongmatchesanddonotimposeanylimit onthelengthofthematchingsubsequence.Therefore,slower executionscanalsobedetectedusingthisscheme.

Iftheexerciseisperformedtooquickly,thematching sub-sequence willbeshort, andwillbeignoredif shorterthan ˛N(k)_,whereN(k)_{isthelengthofthekthtemplate,k=1,...,K.}

Bysetting˛=0.5,wediscardmatchesshorterthanN(k)_/2,as

statedinSection2.Thevalueof˛canbedecreasedfurther toallowevenshortermatches. (Obviously, ˛should beset between1/N(k) _{and 1:when it is1/N}(k)_{, weallow matching}

subsequenceswithaminimumlengthof1;thatis,weallow allmatches.Whenitis1,wediscardmatchingsubsequences shorter than the length of the template.) However, if ˛ is selectedtoosmall(say,lessthan0.2),thenthetemplatecan bematchedtosubsequencesmuchshorterthanitself,which arelikelynottocontainanyexerciseexecution.Thissituation mayincreasetheFAratebecausethesequencescontainnoise, andsomesubsequencesofarecordingmayresembleoneof thetemplatesasaresultofwarping.Weobservedthat˛=0.5 isasuitablechoiceforourdatasetbecauseexercisesshould notbeexecutedtooquicklyandbecausewehavetemplates recorded forexercisesperformed atafasterpace. Further-more,toaccomodateshortermatches,thelocalweight,wv,

fortheverticaldirectioncanbedecreased. Thus,theDTW algorithmissufficientlyflexibleinthesensethattheamount anddirectionofwarpingofthesequencescanbeadjustedas needed.

Notealsothattheinertialsensorrecordingsofanexercise execution at a faster pace may not simply be the time-contracted versions of the sequences of an execution at regular pace. For instance, agyroscope signalprovides the angularrateofchangeofsomepartofthebody,andits ampli-tudemayalsoincreasewhentheexerciseisperformedfaster. Therefore,itmaynotbetrivialtomatchveryquickorslow executionstothetemplateofanormalexecution.Whenthe purposeistodetectexercisesperformedatanunusualpace, itmaybebettertoincludetemplatesforsuchexecutionsin thesystem,asisdoneinthisworkforfasterexecutions.

Whenthepatientneedstousethesystemwhilenotunder supervision,thesensorsmustbefixedtothesamepositions onhisbodyaswhenthetemplatesequenceswererecorded. Aspartoftheirtraining,patientsmustbetaughttowearthe sensorset properlyinall environments. Thepositionsand orientationscouldbemarkedonthebodyusingpermanent markers.Illustrative/descriptivebrochures maybealso pro-videdtoguidesensorplacement.Anotherpossibilityistofix thesensorstoagarmentthatfitsthepatient’sbodytightlyto makesurethattheywillalwaysbeworninthesameway.

A possible modification to the proposed methodology wouldbetoomit recordingand including thetemplatesof incorrectexecutions.Inthiscase,onewouldneedtorecord onlyasingletemplateforeachexercisetype,corresponding tothecorrectexecutionofthatexercise.TheamountofDTW errorcanbeusedasameasureofthediscrepancybetweenthe patient’sexecutionandthereferencetemplateofthecorrect execution,andprovidealertswhennecessary.Anexecution canbeclassifiedastheexercisetypewhosetemplatematches withthesmallestDTWdistance.Incorrectexecutionswould bematchedtothemost-similarcorrectlyexecutedtemplate withsomeerror.Inthiscase,thedegreeofcorrectnessofthe matchedexecutionswouldbedeterminedbythresholdingthe corresponding normalizedDTWdistance.If the distanceis belowthe threshold,the executionissufficientlysimilar to thetemplate,andtheexecutionwouldbeconsideredcorrect. Ifthisisnotthecase,thentheexecutionisnotsufficiently similartothetemplate,anditwouldbeclassifiedasincorrect.

Thisapproachwouldalsohavealowercomputationalcost,as mentionedinSection3.7.

Other techniquessuchasthe Wii[40,41]canbeusedin physicaltherapy.However,the aimofstudiesutilizing that technologyisquitedifferentthanthepurposeofoursystem. Wii systemsare mostlybased onvirtual reality,wherethe patientexecutessomemovementswhileholdingtheWii con-trolleritselforatoolthatcontainsit,suchasatennisracket, toattaincertain goals inagame-likeenvironment by con-trollingacharacter oranobject,whichisdisplayedon the screeninrealtime.Theobjectivesareselectedspecifictothe therapy.Hence,inWiisystems,thepatientisnotguidedto performwell-definedmovements;rather,heperformssome movementshimselftoattainthedescribedgoal.Inaddition,a Wiisystemhasonlyasinglecontrollercontainingmotion sen-sors;thus,itcannotdetectbodypostureormultiplepartsof thebodyasoursystemdoes.Forthisreason,thepatientmay completetheprocessinanundesiredmanner;forinstance, hemaysatisfythegoalswhilesittinginsteadofstanding,or (ifthe purposewastostrengthenaninjuredhand) usehis healthyhand.Suchexploitationsarenotallowedinour sys-temthatusesmultiplesensors.Wiisystemsareinexpensive andeasilyaccessibletoanyonewithstandardTVconnections, andourmethodologycanbeintegratedwithaWiisystemor anothergameconsoleifnecessary.Moderngameconsolesare readilyconnectedtotheinternetandTVwithaudioandour MTMM-DTWalgorithmintegratedwithagameconsolemay beusedtoprovidevisualorvoicefeedbacktothepatient.The results canalso besent tothe specialist. Considering that GPUimplementation ofDTW ismuchfasterthan the CPU implementation [38], andthat moderngameconsoleshave fastGPUs,thehardwareofmostgameconsoleshassufficient computationalpowertoruntheMTMM-DTWalgorithm.

5.

Summary

and

conclusions

Inthisstudy,weaddressanimportantprobleminphysical therapy:automaticallydetectingtheindividualexecutionsof assignedexercisesduringaphysicaltherapysession, classify-ingthem,andevaluatingtheircorrectness/accuracytoprovide feedback tothe patientand the specialist.Wepropose the MTMM-DTWalgorithmforthispurpose,basedontheDTW dissimilaritymeasure,todetecttheoccurrencesofmultiple templatesintherecordingofaphysicaltherapysession.

To evaluate the algorithm’s performance, we conduct experimentsinvolvingeighttypesofexercisesperformedby fivesubjectsandacquireadatasetwithatotalof120and 1200exerciseexecutionsinthereferenceandtestsets, respec-tively.Theaccuracyis93.46%forexerciseclassificationonly and 88.65% for simultaneous exercise and execution type classification.Thealgorithmmisses8.58%oftheperformed executions anddetects 4.91% executionsin excess,caused byidletimeintervals.Totesttherobustnessofthesystem tounknown exercises,we employaleave-one-exercise-out cross-validation strategy. This resultsin afalse alarmrate of 0.83%, demonstrating the robustness of the system to

unknownmovements.

Considering these outcomes, the performance of the proposed system is acceptable, especially in counting and

classifying the exercises. This is valuable because manual countingofapatient’sexecutionscanbeatediousprocess. Furthermore,automaticandobjectiveevaluationofthe exer-cisesessionhasmeritbecausedirectobservationmayprovide subjectivefeedback [7,8]. Oursystem can providefeedback atanydesired level ofdetail tothe patientand the physi-cian/therapist.

This algorithm is quite flexible and highly adaptable fordifferentschemesbecause theflexibilityallowedinthe sequences,the sensitivityindetectingtheoccurrences,the amountofoverlapallowedbetweentheoccurrences,andthe lengthofthematchedsubsequencesarealladjustable.The methodmay bepotentially appliedtoother areas,suchas recognizingquasi-periodicdailyorsportsactivities.

Infuturework,weplan toextend ourstudytopatients undergoing orthopedic rehabilitation to demonstrate our methodologyinclinicalsettings.Thealgorithmmaybe mod-ified toworkinnearreal timeso thatthe patientreceives feedback after each exercise execution within a reason-abletime frame.Thesystem parameterscan beoptimized individuallyforeachexercisetype.Othersensormodalities may be used or combined with motion sensors, such as radio-frequency(RF)localization,todirectlyacquireposition informationinsteadofusinginertialsensorsthatproviderate information.

Conflicts

of

interest

Theauthorsdeclarethattheyhavenoconflictsofinterest.

Acknowledgments

Theauthorswouldliketothankphysicaltherapyspecialists Assoc.Prof.Dr. ˙IlknurTu ˘gcuandDr.ÖzlemYoleriforproviding thedescriptionsoftheexercisesandforprovidingbackground informationontheneedsofpatientsandspecialistsin physi-caltherapy.Theauthorsarealsogratefultothefivevolunteers whoparticipatedinourexperimentsfortheirefforts, dedica-tion,andtime.

Appendix

A.

Dynamic

time

warping

The ordinary DTW algorithm matches two discrete-time sequences (represented as vectors x=[x1, x2, ..., xN]T and y=[y1,y2,...,yM]T)byelasticallytransformingtheirtime(or sample)axessuchthattheyaremostsimilartoeachother.To quantizesimilarity,alocaldistance(cost)measurecanbedefined betweentwosamplesxnandymas

c(xn,ym):F×F→R≥0, (1)

whereFisthefeaturespacesuchthatxn,ym∈F∀n,m[42].The

moresimilararexnandym,thesmalleristhecostfunction c(xn,ym).Inthispaper,weselectthelocalcostasthesquare ofthedistancebetweenxnandym,c(xn,ym)=(xn−ym)2,asis usuallydone.

Tofindtheoptimalmatchbetweenxandy,onecan cal-culate thecostmatrix CofsizeN×Mbetweeneach pairof elementsofxnandymas

C=[Cn,m]=[c(xn,ym)] (2)

andfindtheoptimalwarpingpathinthecostmatrixCwiththe smallestcumulativecost.

A warping path can be represented with the sequence

p=(p1,p2,...,pL),where

pl=(nl,ml)∈[1:N]×[1:M], 1≤l≤L. (3)

Therearethreebasicconditionsforthewarpingpath[42]:

1. Boundarycondition:Thepathstartsfromtheveryfirst ele-mentofthecostmatrixandendsattheverylastelement; i.e.,p1=(1,1)andpL=(N,M).

2. Monotonicitycondition:Thepathcanproceedtothe right, to the bottom, or to the direction in between (bottom-right),butitcannotreturnback;i.e.,n1≤n2≤···≤nLand m1≤m2≤···≤mL.3

3. Step-sizecondition(continuity):4_{Thepathcanproceedtothe}

neighbor element at the right, atthe bottom,or at the bottom-right;i.e.,

pl∈{pl−1+(0,1),pl−1+(1,0),pl−1+(1,1)}, 2≤l≤L. (4)

Thetotal(cumulative)costofawarpingpathpbetweenthe sequencesxandyisdefinedsimplyasthesumofthelocal costsofthematchedelementsofxandy:

Cp(x,y)= L



l=1

c(xnl,yml). (5)

Then,theoptimalwarpingpathp∗isthepathhavingthe min-imum totalcostamongallwarpingpathsbetweenxandy

satisfyingthepathconditions:

p∗_=argmin

p Cp(x,y). (6)

TheDTWdistancebetweenxandyisthendefinedasthetotal distanceoftheoptimalwarpingpath:

DTW(x,y)=Cp∗(x,y)=min

p Cp(x,y). (7)

Unlikeitsname,theDTWdistancedoesnotsatisfythetriangle inequalityevenifthecostfunctionisametric,andhencethe DTWdistanceisnotametric[42].

Insteadofanexhaustivesearch forall possiblewarping pathsbetweenxandy,whichwouldbeextremelyinefficient, one can use an algorithm with computational complexity

3 _{TheindicesnandmoftheCmatrixincreasedownwardsand}

fromlefttoright,respectively.

4 _{SeeRef.[42]forthedifferentchoicesforthestep-size}

O(NM)basedondynamicprogramming.Forthispurpose,the

accumulatedcostmatrixofsizeN×Misdefinedas

D=[Dn,m]=[DTW(x1:n,y1:m)], (8)

wherex1:n=[x1, x2, ...,xn]and y1:m=[y1, y2,...,ym]are the prefixesofthesequencesxandywithlengthsnandm, respec-tively,for1≤n≤Nand1≤m≤M.Obviously,DN,Misthedesired DTWdistance;i.e.,DTW(x,y)=DN,M.Theaccumulatedcost matrixDcanbecomputedrelativelyefficientlywiththe fol-lowingequations[42]: Dn,1= n





⎧

⎪

⎨

⎪

⎩

Here,wh,wv,andwdarethelocalweightsforthehorizontal,

vertical,anddiagonaldirections,respectively,withdefault val-uesofone.Ifoneneedstodiscourageadvancinginacertain directioninwarpingthesequences,thelocalweightforthat directioncanbechosentobelargerthanone.

NotethatallelementsofDmustbecalculatedtoobtainthe verylastelementDN,M,whichistheDTWdistance.Byusing thismethod,theDTWdistanceiscalculatedwithoutexplicitly findingtheoptimalwarpingpathp∗.UsingD,p∗canbe calcu-latedbyinitializingp∗_L=(N,M)andprogressinginthereverse order:Ifp∗_l iscomputed,p∗_l−1iscalculatedas

p∗_l−1=

⎧

⎨

⎩

argmin{Dn−1,m,Dn,m−1,Dn−1,m−1} otherwise.

(10)

Theoptimalpathp∗terminateswithp∗1=(1,1).Inotherwords,

startingatthebottom-rightelementDN,M,theoptimal war-pingpathstepsintothenext-smallestneighborelementinD

(proceedingonlytotheleft,top,ortop-left)andfinallyends upwiththetop-leftelementD1,1.

A.1. Multi-dimensionalsequences

NotethatonecancomputetheDTWdistanceandtheoptimal warpingpathoftwosequencesxandybyknowingonlythe costmatrixC,whichcanbecomputedbyusingthelocalcost functionc(xn,ym)definedoneverysamplexnandymofxand y.Inthecaseofmulti-dimensionalsequencesxandy,thelocal costc(x_n,y

m)canbedefinedtohandlethissuchthatitsrange

isscalar,asbefore.Then,theDTWandtheoptimalwarping pathcanbecalculatedinexactlythesamewayasbeforefor one-dimensionalsequences.Inthiscase,thesamewarpingis appliedtoalldimensionsofxandy,consideringtheoverall similaritybetweenxandy.

A.2. Freeendpoints

For some applications, the sequences to be matched are croppedmanuallyandhencemaynaturallycontainaprefix orasuffixthatcontainsnovaluableormeaningful informa-tion.Then,thesequencesshouldbematchedtoeachother withsomeunmatchedpartsatthebeginningorattheendof

oneofthesequences.Fromnowon,thesequencesxandyare calledthetemplateandthetestsequences,respectively, assum-ingthetestsequencemaycontainprefixand/orsuffixparts, whereas thetemplatesequence doesnot. Thisassumption isvalidinpatternclassificationandpatternsearchproblems, wheretherearetypicallyseveraltemplatesequencesobtained forthispurposethatdonotcontainanyundesiredparts,but therearemany testsequences thatmaycontainadditional partsatthebeginningand/orattheend[42].

UsingtheordinaryDTWalgorithm,theprefixesand/or suf-fixesofthesequenceswillcauseanadditionalundesiredcost, increasingtheDTWdistance.Abetterapproachwouldbeto modifytheordinaryDTWalgorithmtoallowfreeendpointsso thattheprefixand/orsuffixpartsofthetestsequenceycan beignored[43].Thelengthoftheignoredpartsareselected optimallyinthesensethattheDTWdistancebetweenxand the matchedsubsequence ofy isminimized[4,42]. If both endpointsarefree,thealgorithmiscalledsubsequenceDTW

(SDTW) [42]oropen-beginopen-end DTW(OBE-DTW)[4], and theresultingdistanceis

SDTW(x,y)= min

m1,m2

DTW[x,y(m1:m2)], (11)

wherey(m1:m2)=[ym1,ym1+1,...,ym2]isthesubsequenceof y,with1≤m1≤m2≤M.Them1andm2valuesminimizingthe

DTWdistance,namelym∗₁and m∗₂,determinethe (optimal) matchedsubsequenceofytox.Ifm1issetto1,onlythe

suf-fixofyisexcluded. Similarly,ifm2=M,onlytheprefixofy

isexcluded.Obviously,usingm1=1andm2=Mresultsinthe

ordinaryDTWalgorithm.

Toallowfreeendpoints,theordinaryDTWalgorithmneeds tobemodifiedasfollows:

• open-beginDTW:Toexcludetheprefixofy,thefirstboundary conditionofthewarpingpathisextendedsothatp1=(1,m1)

with1≤m1≤M,allowingthebeginningpointofthewarping

pathtoresideanywhereinthefirstrowoftheaccumulated costmatrixD.Thus,thefirstm1−1samplesofyareignored.

Tothisend,thefirstmodificationintheordinaryDTW algo-rithmisinthecalculationofthefirstrowoftheaccumulated costmatrixD:

D1,m=c(x1,ym). (12)

Inthisway,thefirstrowofDconsistsofthecostsof match-ingx1toeachelementofyinsteadoftotheaccumulated

costs.Theopen-beginDTWdistanceisDN,M,asbefore.The secondmodificationisinthecalculationoftheoptimal war-pingpath.Theoptimalpathp∗iscalculatedinthereverse order,asbefore;however,thistimetheprocessterminates whenthefirstrowofDisreached;i.e.,whenp∗₁=(1,m).