Representing stuff in the human brain

Representing

stuff

in

the

human

brain

Alexandra

C

Schmid

and

Katja

Doerschner

Ourexperienceofmaterialsdoesnotmerelycomprise judgmentsofsinglepropertiessuchasglossinessorroughness butisrathermadeupofamultitudeofsimultaneous impressionsofqualities.Tounderstandtheneuralmechanisms yieldingsuchcompleximpressions,wesuggestthatitis necessarytoextendexistingexperimentalapproachesto thosethatviewmaterialperceptionasadistributedand dynamicprocess.Adistributedrepresentationsframeworknot onlyfitsbetterwithourperceptualexperienceofmaterial qualities,itiscommensuratewithrecentpsychophysicsand neuroimagingresults.

Addresses

1_{Justus-Liebig-UniversityGiessen,Germany} 2_{BilkentUniversity,Turkey}

Correspondingauthor:Doerschner,Katja (Katja.Doerschner@psychol.uni-giessen.de)

CurrentOpinioninBehavioralSciences2019,30:178–185 ThisreviewcomesfromathemedissueonVisualperception EditedbyHannahESmithsonandJohnSWerner

ForacompleteoverviewseetheIssueandtheEditorial

Availableonline4thNovember2019

https://doi.org/10.1016/j.cobeha.2019.10.007

2352-1546/ã2019TheAuthors.PublishedbyElsevierLtd.Thisisan openaccessarticleundertheCCBY-NC-NDlicense( http://creative-commons.org/licenses/by-nc-nd/4.0/).

Separate

neural

processing

of

material

and

shape

properties?

Ourvisualexperienceoftheworldisasmuchdefinedby

the material qualities of objects as it is by their shape

properties:keyslookshiny,atreetrunklooksrough,and

chocolatesouffle´ looksairy.Humanscaneasilyandnearly

instantaneously identify shapes [1,2] and properties of

materials [3] through vision alone. Although much

researchhasbeendedicatedtounderstandingtheneural

mechanisms underlyingshape (e.g. Refs.[4–6,7]) and

—morerecently—materialperception[8],forthemost

partresearchonshapeandmaterialperceptionhavenot

intersectedsubstantially.Infact,mostinterpretationsof

human neuroimaging and monkey physiology research

proposethatshape andmaterialpropertiesmaybe

pro-cessedindependentlyalongdifferentpartsoftheventral

visualstream([9–11]alsoseethereviewbyRef.[8]).In

contrast,recentpsychophysicsworkhasshownthatshape

informationcanplayquiteacriticalroleintheperception

of material properties, such as translucency ([12],

Figure1a)orgloss([13],Figure 1b).In linewiththis,

recentneuroimagingstudieshavefoundthatshape

sen-sitivecorticalareasare,infact,alsosensitivetomaterial

propertiessuchassurfacegloss[14].Here,wetakethese

recentfindingsof coupled shape-materialcomputations

as adeparturepoint to highlight thatourexperience of

materialscomprisesnotonlyjudgmentsofsingle

proper-tiessuchasglossinessorroughness;ratheritconsistsofa

multitudeof simultaneousimpressionsof qualities (e.g.

visual[15],haptic [16,17],auditory[18,19],emotional,

ormotivational[20,21]).Tounderstandtheneural

mech-anisms yielding such complex impressions, we suggest

thatitisnecessarytoextendexistingexperimental

para-digmstothosethatviewmaterialperceptionasa

distrib-utedanddynamicprocess.Specifically,wewilldiscussan

alternativeframework, inspiredbyrecent neuroimaging

workinobjectperception[22,23,24],thatpromisesto

betteridentifytheneuralcorrelatesofourexperienceof

materialqualities.First,wewillbrieflyreviewstudiesthat

investigatetheneuralsensitivitytomaterialqualitiesand

categories.Wewillthenpointoutthepotentiallimitsof

considering material quality as an independently and

locallyprocessedobjectproperty.Finally,wewilldiscuss

apotentialalternativewayofconceptualizingtheneural

representation of material properties in a distributed

networkinvolvingdirectandindirectassociations.

Neural

mechanisms

in

material

perception

Aprocessinghierarchyintheventralvisualpathway

Investigations into the neural mechanisms underlying

theperceptionofmaterialqualitieshavestartedoutonly

recently. From this work a few candidate areas have

emerged as being particularly sensitive to changes in

materialproperties:forexample,in human fMRI

stud-ies, stronger responses to glossy objects (compared to

matte)havebeenfoundfromearly(e.g.V1,V2)tolate

visualareasintheventralstream(e.g.posteriorfusiform)

[25,26]). Similarly, regions along the medial ventral

visualcortex(e.g.CollateralSulcusCoS,

Parahippocam-palgyrusPHG,LingualGyrusLG,FusiformGyrusFG,

ParahippocampalPlaceAreaPPA)showapreferencefor

texture information (e.g. granite and tree bark) over

shape, color, or orientation [27,9,11,28–33]. Ventral

streamareasalsoseemtobeimportantfortheprocessing

ofmaterialcategories(e.g.wood,stone,fabric)andtheir

properties(e.g.FG,CoS,orPHG,see[34–36]).Inlight

oftheseresultsitisperhapsnotsurprisingthatageneral

interpretationisthat‘visualinformationaboutmaterials

and surface qualities are processed and represented

mainlythroughahierarchyoftheventralvisualpathway’

[8],wherelower-levelimagestatisticsthatdifferentiate

materials are represented in earlier visual areas,

representationsoftheperceivedqualityandcategoryof

materials [35].

Cooperativecomputationsandinteractions

Althoughthislate-combination-of-cuesideacertainlyhas

acomputationalappeal[37]andhasguidedneuroimaging

research(e.g.Refs.[38,7,8]),theprocessingofmaterial

properties might not beneatly localized to one cortical

vicinitywith,say,theprocessingof shapetoanother.In

fact,recentpsychophysicsresultshavestronglysuggested

thatatleastsomecomputationsofmaterialqualityoccur

together with computations of shape. For example,

perceived3Dshapeandsurfacepropertieslikelightness,

glossandtranslucencycanmutuallyconstrainoneanother

[12,13,39–45], implying that it is computationally

unlikely that they are processed separately. Marlow

etal.[13,41]showedthatthesameluminancegradient

— even with the same bounding contour — can be

perceivedasmatteshadingorglossyshading(i.e.different

materials) depending on the perceived 3D shape

(Figure1b).Moreover,recentneuroimagingstudieshave

found that putative shape-specialized regions are also

sensitive to changes in material properties (e.g.

V3b/KO [25,46,14], LOC, [35], or V4 [47]), and,

con-versely, that putative material specialized-regions can

process shape information (e.g. CoS, [33,35], or FG

Figure1 (a) (b) Snake Snake Ribbon Ribbon

Current Opinion in Behavioral Sciences

Examplesforjointcomputationsofshapeandmaterialquality.

(a)Dependingonthestereoscopicshapeinterpretation(snakeorribbon)thesameluminancegradientappearsasatranslucentvolume

illuminatedfromwithin,orasanopaquesurfacereflectinglightfromabove(leftandrightimagesaresetupforcross-fusion).Figureadaptedfrom Ref.[12](withpermissionfromauthors).(b)Anotherexamplethatmaterialperceptiondependsonperceivedthree-dimensionalshape:the luminancegradientsintheleftandrightimagesarethesame,howeverthedifferentcontours,inducedifferentperceptsofthree-dimensional shape,andmaterial(matteontheleftandshinyontheright).FigureadaptedfromRef.[13](withpermissionfromauthors).

[32])whichsupportstheideaof(neural)joint

computa-tionsofmaterialqualityandshape.3

Shapeandmaterialpropertiesdonotjustmutually

con-strain one another, they also interact in a non-linear

manner when observers judge perceptual qualities of

objects. For example, in [48] soft substances that fell

onthegroundlookedrunnierwhentheyweretransparent

and glossy (as opposed to matte), whereas harder

sub-stances looked equally non-runny regardless of their

surfaceoptics.Inaddition,certaincombinationsofshape

and material evoke specific material qualities [48] and

categories (see Figure 2, [49,50]), suggesting that our

perception ofmaterialsis notlimited to the processing

ofimagefeaturestodeterminewhetherasurfaceisglossy

or translucent [51]; our perception also includes these

associatedmaterialqualities(e.g.seeFigure 2,[52]).In

fact,suchassociations extendbeyondourvisual

experi-ence: seeing an image of silky stuff evokes a vivid

sensation of what it would feel like to run our hands

throughthematerial(e.g.Refs.[53,48]).These

associa-tionsmayalso be relatedto task demands:judging the

softnessofavisuallypresentedmaterialwillmostlyrely

onassociationswith tactileproperties(Figure3d).

Weproposethatinteractiveprocessingofimagefeaturesand

associationsshouldbeconsideredwhenstudyingtheneural

mechanismsofourperceptualexperienceofmaterial

quali-ties.Paradigmsthatfocusonidentifyingwhatareasprocessa

specificobjectpropertyorimagefeature maymiss outon

(a) (b)

(d) (c)

specular roughness lower specular roughness higher

highlight isotropic

highlight anisotropic

‘Looks like plastic.’ ‘Looks like plastic.’

‘Looks like plastic.’ ‘Looks like silk.’

Current Opinion in Behavioral Sciences

Categoricalshiftsinmaterialappearance.

Fourpanelsshowthesameobjectilluminatedbythesamelightfield.Wemanipulatedthespecularroughness(betweenleftandrightimages)and specularhighlightanisotropy(betweentopandbottomimages).WefoundthatwhileFigures(a)–(c)haveasomewhat‘plastic-like’appearance (withmoreorlessgloss),panel(d)notonlylooksrougherbutalsochangesthematerialcategory,thatis,itlookslikesilktomostobservers[54]. Thisillustratesthatcertaincombinationsofvisualcuesevokespecificmaterialqualitiesandcategories.Investigationsoftheneuralprocessingof materialpropertiesneedtobeabletoaccountfortheseassociationeffects.

3

Ithasbeenproposedthattheinvolvementofacorticalregionina certainperceptualcomputation(shapeormaterial)mightdependonthe taskthatthevisualsystemisperforming([31,35]).Taskdemandscanbe incorporatedintoadistributedrepresentationofmaterialqualitiesas illustratedinFigure3.

Figure3

Current Opinion in Behavioral Sciences

Adistributednetworksframeworkformaterialperception.

Anexampleofhowtheperceptionofmaterialqualitiesmightarisefromdistributednetworkactivity.Thistoynetworkencompassesthecoupled computationsofdifferentproperties(suchas3Dshapeandmaterial)fromvisualsensorycues(orcuesfromothermodalities,e.g.tactile;arrows fromrectanglestoellipses),anditalsoconsidershowassociatedpropertiesmightinfluenceoractivateeachother(connectionsbetweenellipses). Rectanglesshowexamplesensorycuesthatareprocessedbythenetwork,ellipsesdenotespecificobjectandmaterialproperties(3Dshape, surfaceappearance,surfacefeel)thatmaybeevokeddirectlybysensorycuesorassociatedproperties(solid)orindirectlyviaotherroutes (dotted).Lightgraydottedlinesimplyneitherdirectnorindirectprocessingviaagivenroute.Panel(a)showsourhypotheticalnetworkandits potentialconnections.(b)Staticimagecuesaredirectlyassociatedwithrepresentationsofsurfaceappearanceand3Dshape(solidblacklines), aswellasindirectlyassociated(dottedblacklines)withsurfacefeel,mechanicalqualitiesandpotentialimagemotion(nonrigid,specularmotion). (c)Lookingatmovingdotpatternsofaclothblowinginthewind[82]changesthepatternofdirectandindirectassociations.Note,however,that similarpropertiesareactivatedasinb.Panel(d)illustrateshowtaskdemandsinfluencewhichaspectsofthenetworkaredrawnupon.Red colorsmeanthatpropertiesofconnectionsarerelevantforagiventask(solidanddottedlinesdenotedirectandindirectassociations, respectively).Inthiscaseestimatingthehardness(atactilejudgment)ofthematerialintheimagecannotbeachieveddirectlyfromthevisual input(nodirectconnectionsfromvisualinputtotheredellipses)buthastooccurvia3Dcuestoshape,oropticalpropertiesofthematerial (directroutes,redsolid)orviaindirectroutesthatbecomeactivatedbyassociationshapeandopticalcueswithaspecificmaterialcategorythat hascharacteristicnonrigidmotionproperties.Panels(e)and(f)showpropertiesdrawnuponwhenjudginganopticalpropertyandwhile performingacategorizationtask,respectively.Notethathighlightingacomponentdoesnotimplythattheseunitsareactivatedperse:itisthe

appearsto beaconsensusamong mostresearcherswhose

workwehavecitedthatknowingwhichcorticalareas

pref-erentiallyrespondtooneobjectpropertyoveranotherdoes

notnecessarilyrevealtheunderlyingcomputationscarried

out bytheseregions(e.g.Ref.[34]).Inordertomakeprogress

towardsunderstandingthecomputationsperformedbythe

braininmaterialperceptionwesuggest,inthenextsection,

thatitmaybefruitfultolooktowardsdevelopmentsinthe

objectandscenerecognitionneuroimagingliterature.

Spe-cifically,wesuggestthatthecomputationsthatmakeupour

complexperceptualexperienceofmaterialsareunlikelyto

be executed by separate specialized cortical areas, but

insteadmustbejointlycomputedandrealizedbysufficiently

complexanddistributed,interactingneuralhardware.

Moving

to

a

distributed

representations

framework

Researchinvestigatingthemechanismsunderlyingobject

andsceneperceptionhasstartedturningtotheideathat

neuralrepresentationsshouldreflectthedynamicnatureof

tasksandgoals;thatis,recognition,interaction,navigation,

andprediction(e.g.Refs.[55–57]).Thereisagrowingbody

ofliteraturesuggestingthatobjectandscene

representa-tionsaredistributedindistinctbuthighlyinteractive

net-worksorcircuitsthatextendbeyondtheventralpathway

([22,23,58–60,45,57,61]alsoseeBox1),andthatproperty

representations(suchaswhatanobjectlookslike,howit

moves,howitisused)aregroundedintheactivityofsuch

networks(e.g.Ref.[24]).Theimplicationsformaterial

perception are thatthe processingof properties such as

surfaceappearance,form, motion,tactileproperties,and

evenaction-relatedpropertiessuchas‘graspable’are

intri-cately intertwined (e.g. Ref. [62], for a review seeRef.

[24]),rather than being processedseparately andthen

integrateddownstream.Underthisframework,

represen-tationsof suchproperties(e.g.wobbling motion) canbe

activatedandaffectedbyotherassociatedproperties(e.g.

Jell-Oshaped,green,glossy,translucent),associated

con-ceptualknowledge(e.g.‘dessert’),andtaskdemands(e.g.

asking‘how gelatinousisthis object?’)(seeFigure3 for

anotherexample). Furthermore, representations of such

propertiesarenotmodality-specific[63]:theycan

poten-tially be activated through visual, tactile, and auditory

input.Forexample, somatosensoryandauditorycortices

respondwhenviewingpicturesof graspableobjects[64]

and sound-implying objects [65], respectively.

Impor-tantly,objectandmaterialrepresentations,includingboth

categoryandmaterialqualityrepresentations,arethe

dis-tributedactivationof associatedproperties andconcepts

(seeFigure3foranillustrationofthisframework).

Thinkingaboutmaterialsandtheirpropertiesintermsof

distributed activations, rather than as emerging from

separatecorticalareasspecializedforprocessing

individ-ualproperties,willhelptoconnectneuralrepresentations

of materials with our complex and multifaceted visual

experienceofthe worldand theobjects in it: Through

visual information alone we simultaneously recognise

objectsholistically (chair, spoon, cat) atdifferentlevels

ofabstraction(mycat,pet,animatebeing);werecognise

thematerialsthatthingsaremadefrom(wood,fur,glass,

plastic);experiencemultisensorymaterialqualities(hard,

cold, fluffy); and we automatically access associated

semanticconceptsandaffordances(‘cangrasp’,‘iseaten’,

‘cansiton’).Adistributedrepresentationsframeworknot

onlyfitsbetterwithourperceptualexperienceof

materi-als, it is commensurate with recent psychophysics and

neuroimagingresults.Forexample,[66]foundthatwhen

people visually discriminated photographs of different

fabrics,combinationsofthesurfacepropertiesandfolding

patternsthatwerepresentinthestimuliinfluencedhow

tactilestimuliwouldbematched.Thiscrossmodal

asso-ciationbetweenvisualandtactile propertiesisreflected

inneuroimagingresultsthatshowthattactile

discrimina-tionscanactivateandbedecodedinvisualareas[67–72],

and reciprocally, visual discrimination of rough and

smoothsurfacescanbedecodedinsomatosensorycortex

(evenwhencontrollingfortheeffectsofimagery,

mem-ory,andnon-tactilevisualcharacteristics,[14]).Sunetal.

[14]describetheirresultsas‘compatiblewithan

antici-patorysystemthatextractssurfacepropertiesfromvisual

information’. Indeed, touching and grasping objects is

somethingthattypicallyoccursafterobjectidentification,

(Figure3LegendContinued)aspectoftherepresentationthatthebrain‘paysattentionto’whenperformingthetask.As[24]putsit:‘the regionscomprisingacircuitdonotcomeonlineinpiecemealfashionastheyarerequiredtoperformaspecifictask,butratherseemtorespond inanautomatic,all-or-nonefashionasiftheywerepartoftheintrinsic,functionalneuralarchitectureofthebrain.’[24].Othermodalities(e.g. auditoryinput),andcognitiveandemotionalstatesthatarenotshowninthistoydiagrammayalsointeractwiththeprocessingofmaterial qualities.

Whatevidenceistherefordistributedrepresentationsoverthe conceptionoftheventralanddorsalpathwaysasserialstaged hierarchies?

Adistributedrepresentationsframeworkbetterreflectsevidence aboutstructuralandfunctionalconnectionsthathavebeenfound inthebrain.Thereisanatomicalandfunctionalevidencethat ventralanddorsalstreamsgiverisetomultipledistinctpathways, whereregionsfromputativeearlyandlatestagesofthehierarchy communicatedirectly[22,23].

RestingstatefMRIrevealsseveralinterconnectednetworksof brainregions(e.g.Ref.[58]).

Commonareasareactivatedforbothperceptualandconceptual tasks,suggestingthatobjectpropertiesarerepresentedina modality-independentmanner(e.g.Refs.[60]).

Thereareexamplesoftask-basedeffectsonvisualprocessing(e. g.Refs.[80,57,73]).

Thereisevidencethatbehaviordoesnotcorrelatewithpatternsof activityinputativeobject/scene-selectivebrainregions(e.g.Ref. [59,81]).

so such effects couldreflecta primingfor futureaction

[24].Itisdifficulttoaccountforcross-modalinteractions

in visualand somatosensory areasif properties are

pro-cessed in separate, independent streams before being

integrated.

Outlook

The aim of this article was to use recent findings that

highlightthemultifacetedaspectsofexperiencing

mate-rial qualitiesto sparkaparadigm shiftin related

neuro-imaging research. The questionremains about how our

representations of material properties are grounded in

these distributednetworks. That is, what are thelocal

computationsperformedinventralanddorsalregionsthat

give rise to these representations [73]? It has been

suggested that the important computational goals of

the visual system likely reflect our experience, that is,

the perceptual scission of a scene into different causal

‘layers’:shape, pigment,gloss,translucency,and

illumi-nation effects [74].These local computations, as

sug-gestedinSection2,arelikelytooccurcoupled(Figure1).

Therefore,justasimportantassearchingforareaswhere

certaincues(e.g.texturestatistics,motionflow,binocular

disparity) areprocessed isan understandingof howour

holisticimpressionsemerge,thatis,theneuralsubstrates

associatedwithcombiningthesecuestoconjointly

com-puteshape,material,illumination,andsoon.

Investigat-ing this requires moving from univariate designs and

analyses, where one type of stimulus or attention to

one stimulus dimension leads to greater activity than

another stimulus/attended dimension, to multivariate

designs and analyses ([75,76] see Ref. [77] review for

acomprehensivecomparisonofunivariateand

multivari-atetechniques,butseeRef.[78]forlimitationsof

multi-variatetechniques).Forexample,usingmultivariate

pat-tern analysis (MVPA), Sun and colleagues identified a

region that potentially integrates cues to 3D structure

(V3B/KO, [25,46,14,7]). Such multivariate methods

allow for the identification of regions where activity

reflects unique or joint representations. Furthermore,

new techniqueshavebeendevelopedtocombinefMRI

decoding with MEG decoding [79], which could help

reveal the underlying spatio-temporal dynamics — a

representation atacertain timepoint (MEG)correlates

with (hasthesame representationalstructure as) neural

activation at particular regions (fMRI) — which could

helptounravelwhenandwheredifferentrepresentations

emerge.Resultsyieldedbymultivariatetechniquesmay

thus playakey rolein deepening ourunderstandingof

the neural processes involved in material perception

because they have the potential to reveal distributed

patterns of activity that underlie joint computations of

propertiesand theirassociations.

Conflict

of

interest

statement

Nothing declared

Acknowledgements

SofjaKovalevskajaAward(‘PerceivingMaterialQualities-Brain MechanismsandDynamics’)bytheAlexandervonHumboldtFoundation, endowedbytheGermanFederalMinistryofEducationandResearch.

References

and

recommended

reading

Papersofparticularinterest,publishedwithintheperiodofreview, havebeenhighlightedas:

 ofspecialinterest ofoutstandinginterest

1. ThorpeS,FizeD,MarlotC:Speedofprocessinginthehuman visualsystem.Nature1996,381:520-522.

2. VanRullenR,ThorpeSJ:IsitaBird?IsitaPlane?Ultra-rapid visualcategorisationofnaturalandartifactualobjects. Perception2001,30:655-668.

3. SharanL,RosenholtzR,AdelsonEH:Accuracyandspeedof materialcategorizationinreal-worldimages.JVis2014, 14:1-24.

4. HandaT,MikamiA:Neuronalcorrelatesofmotion-defined shapeperceptioninprimatedorsalandventralstreams.EurJ Neurosci2018,48:3171-3185.

5. GeorgievaSS,ToddJT,PeetersR,OrbanGA:Theextractionof 3Dshapefromtextureandshadinginthehumanbrain.Cereb Cortex2008,18:2416-2438.

6. DoddJV,KrugK,CummingBG,ParkerAJ:Perceptuallybistable three-dimensionalfiguresevokehighchoiceprobabilitiesin corticalareaMT.JNeurosci2001,21:4809-4821.

7.

 AnnuWelchmanRevVisAE:SciThe2016,human2:345-376.brainindepth:howweseein3D.

Excellentreviewonstudiesofbinoculardisparityanditscombinationwith otherdepthcues.Suggeststhatthedorsalvisualcortexprocesses3D information and also integratessignals torepresent thestructure of viewedsurfaces,whereastheventralcortexmaystorerepresentations ofobjectconfigurationsandthefeaturesrequiredfortaskperformance. 8. KomatsuH,GodaN:Neuralmechanismsofmaterial

perception:questonShitsukan.Neuroscience2018, 392:329-347.

9. CantJS,GoodaleMA:Attentiontoformorsurfaceproperties modulatesdifferentregionsofhumanoccipitotemporal cortex.CerebCortex2007,17:713-731.

10. CantJS,LargeME,McCallL,GoodaleMA:Independent processingofform,colour,andtextureinobjectperception. Perception2008,37:57-78.

11. Cavina-PratesiC,KentridgeRW,HeywoodCA,MilnerAD: Separateprocessingoftextureandformintheventralstream: evidencefromfMRIandvisualagnosia.CerebCortex2010, 20:433-446.

12.

 ofMarlowsurfacePJ,opacity.KimJ,AndersonProcNatlBL:AcadPerceptionSciUSandA2017,misperception 112:13840-13845.

Demonstratesthatperceivedsurfaceglossdependsonperceived 3D shape.Suggeststhatperceived shapeconstrainsthecomputationof materialqualities.

13.

 Marlowthree-dimensionalPJ,TodorovicshapeD,Andersonandmaterial.BL:CoupledCurrBiolcomputations2015,25:221-of 222.

Demonstrates that perceivedtranslucencydepends onperceived 3D shape.Suggeststhatperceived shapeconstrainsthecomputationof materialqualities.

14.

 SunDifferentialH-C,DiprocessingLucaM,BanofH,binocularMuryyA,FlemingandmonocularRW,WelchmanglosscuesAE: inhumanvisualcortex.JNeurophysiol2016,115:2779-2790.

Identifiesbrainareasthatpreferentiallyrespondtoglossyobjects.Also findsactivationsrelatedtoglossintheposteriorfusiform(pFs)andinarea V3B/KO,which areareasknown toplaya roleinshapeperception. Suggeststhat theseareasintegrateinformation relatedtoshapeand surfaceproperties,andmightbecandidateareasforjointcomputations ofshapeandmaterialproperties.

16. AdamsWJ,KerriganIS,GrafEW:Touchinfluencesperceived gloss.SciRep2016,6:21866.

17.

 Suntouch:H-C,visualWelchmancuestoAE,surfaceChangstructureDHF,DiLucadriveM:somatosensoryLookbutdon’t cortex.Neuroimage2016,128:353-361.

Shows that visual material qualities activates somatosensorycortex. Suggeststhatvisualcuestoanobject’ssurfaceproperties(especially thoserelevantforgrasping,suchasroughness)evokeactivityinneural circuitsassociatedwithtactilestimulation.

18. KlatzkyRL,PaiDK,KrotkovEP:Perceptionofmaterialfrom contactsounds.PresenceTeleoperatorsVirtualEnviron2000, 9:399-410.

19. FujisakiW,GodaN,MotoyoshiI,KomatsuH,NishidaS: Audiovisualintegrationinthehumanperceptionofmaterials. JVis2014,1412–12.

20. FujisakiW,TokitaM,KariyaK:Perceptionofthematerial propertiesofwoodbasedonvision,audition,andtouch.Vis Res2015,109:185-200.

21. DrewingK,WeyelC,CelebiH,KayaD:Systematicrelations betweenaffectiveandsensorymaterialdimensionsintouch. IEEETransHaptics2018,11:611-622.

22. KravitzDJ,SaleemKS,BakerCI,MishkinM:Anewneural frameworkforvisuospatialprocessing.NatRevNeurosci2011, 12:217-230.

23.

 KravitzventralDJ,visualSaleempathway:KS,BakeranexpandedCI,UngerleiderneuralLG,frameworkMishkinM:forThethe processingofobjectquality.TrendsCognSci2013,17:26-49.

Proposesthattheventralpathwayisbestunderstood asarecurrent occipitotemporalnetwork containing neuralrepresentations ofobject qualityboth,utilizedandconstrainedbydistinctcorticalandsubcortical systems.Eachsystemservesitsownspecializedbehavioral,cognitive,or affectivefunction.Thisframeworkcontrastswiththeideaoftheventral visualpathwayasalargelyserialstagedhierarchyculminatinginsingular object representations and more parsimoniously incorporates atten-tional,contextual,andfeedbackeffects.

24.

 Martinperception,A:GRAPES—groundingandemotionsystems:representationshowobjectpropertiesinaction,and categoriesarerepresentedinthehumanbrain.PsychonBull Rev2016,23:979-990.

Proposesthat conceptsshould beviewedasthe productsofhighly interactiveneuralcircuitsgroundedintheaction,perception,and emo-tionsystems.Thenodesofthesecircuitsaredefinedbyregions repre-sentingspecificobjectproperties(e.g.form,color,andmotion)andthus areproperty-specific, ratherthan strictly modality-specific.Thisview breakesawayfromtheviewthatconceptscouldbehousedinprimary sensoryandmotorcorticesandthatanobjectconceptcouldbestoredin asinglebrainregion.

25. KamT-E,MannionDJ,LeeS-W,DoerschnerK,KerstenDJ: Humanvisualcorticalresponsestospecularandmatte motionflows.FrontHumNeurosci2015,9p.579.

26. WadaA,SakanoY,AndoH:Humancorticalareasinvolvedin perceptionofsurfaceglossiness.Neuroimage2014, 98:243-257.

27. BaumgartnerE,GegenfurtnerKR:Imagestatisticsandthe representationofmaterialpropertiesinthevisualcortex.Front Psychol2016,7:1-13.

28. CantJS,XuY:Objectensembleprocessinginhuman anterior-medialventralvisualcortex.JNeurosci2012,32:7685-7700.

29. CantJS,XuY:Theimpactofdensityandratioon object-ensemblerepresentationinhumananterior-medialventral visualcortex.CerebCortex2015,25:4226-4239.

30. CantJS,XuY:Thecontributionofobjectshapeandsurface propertiestoobjectensemblerepresentationin anterior-medialventralvisualcortex.JCognNeurosci2016,29:398-412.

31. CantJS,ArnottSR,GoodaleMA:fMR-adaptationreveals separateprocessingregionsfortheperceptionofformand textureinthehumanventralstream.ExpBrainRes2009, 192:391-405.

evidencefromfMRIadaptationandvisualobjectagnosia. CerebCortex2010,20:2319-2332.

33. PeuskensH,ClaeysKG,ToddJT,NormanJF,VanHP,OrbanGA: Attentionto3-Dshape,3-Dmotion,andtexturein3-D structurefrommotiondisplays.JCognNeurosci2004, 16:665-682.

34. CantJS,GoodaleMA:Scratchingbeneaththesurface:new insightsintothefunctionalpropertiesofthelateraloccipital areaandparahippocampalplacearea.JNeurosci2011, 31:8248-8258.

35. HiramatsuC,GodaN,KomatsuH:Transformationfrom image-basedtoperceptualrepresentationofmaterialsalongthe humanventralvisualpathway.Neuroimage2011,57:482-494.

36. JacobsRHAH,BaumgartnerE,GegenfurtnerKR:The representationofmaterialcategoriesinthebrain.Front Psychol2014,5p.146.

37. LandyMS,MaloneyLT,JohnstonEB,YoungM:Measurement andmodelingofdepthcuecombination:indefenseofweak fusion.VisRes1995,35:389-412.

38. WelchmanAE,DeubeliusA,ConradV,Bu¨lthoffHH,KourtziZ:3D shapeperceptionfromcombineddepthcuesinhumanvisual cortex.NatNeurosci2005,8:820-827.

39. KnillDC,KerstenD:Apparentsurfacecurvatureaffects lightnessperception.Nature1991,351:228-230.

40. SchmidAC,AndersonBL:Perceptualdimensionsunderlying lightnessperceptioninhomogeneouscenter-surround displays.JVis2017,17p.6.

41. MarlowPJ,AndersonBL:Materialpropertiesderivedfrom three-dimensionalshaperepresentations.VisRes2015, 115:199-208.

42. MarlowPJ,AndersonBL:Motionandtextureshapecues modulateperceivedmaterialproperties.JVis2016,16:1-14.

43. MarlowPJ,MooneySWJ,AndersonBL:Photogeometriccuesto perceivedsurfaceshading.Curr.Biol.2019,29:306-311.

44. MooneySWJ,AndersonBL,TaylorG,BuildingA:Specularimage structuremodulatestheperceptionofthree-dimensional shape.CurrBiol2014,24:2737-2742.

45. NormanJF,ToddJT,OrbanGA:Perceptionof three-dimensionalshapefromspecularhighlights,deformationsof shading,andothertypesofvisualinformation.PsycholSci 2004,15:565-570.

46. SunH-C,BanH,DiLucaM,WelchmanAE:fMRIevidencefor areasthatprocesssurfaceglossinthehumanvisualcortex. VisRes2015,109:149-157.

47. KimT,BairW,PasupathyA:Neuralcodingforshapeandtexture inmacaqueareaV4.J.Neurosci.2019,39:4760-4774.

48. SchmidAC,DoerschnerK:Shatterandsplatter:the contributionofmechanicalandopticalpropertiestothe perceptionofsoftandhardbreakingmaterials.JVis2018, 18:1-14.

49. vanAssenJJR,FlemingRW:Influenceofopticalmaterial propertiesontheperceptionofliquids.JVis2016,16:1-12.

50. AliagaC,O’sullivanC,GutierrezD,TamstorfR:Sackclothorsilk? Theimpactofappearancevsdynamicsontheperceptionof animatedcloth.Proc.oftheACMSymposiumonApplied Perception2015:41-46.

51. ChadwickAC,KentridgeRW:Theperceptionofgloss:areview. VisRes2015,109:221-235.

52. FlemingRW:Materialperception.AnnuRevVisSci2017, 3:365-388.

53. BaumgartnerE,WiebelCB,GegenfurtnerKR:Visualandhaptic representationsofmaterialproperties.MultisensRes2013, 26:429-455.

54. SchmidAC,DoerschnerK:Chocolate,chrome,orcloth?The appearanceofspecularhighlightsdeterminesperceived materialcategory.J.Vis.,19thAnnualVisionSciencesSociety Conference(VSS);StPeteBeach,FL,USA:2019.

55. FischerJ,MikhaelJG,TenenbaumJB,KanwisherN:Functional neuroanatomyofintuitivephysicalinference.ProcNatlAcad SciUSA2016,113:5072-5081.

56. GroenIIA,SilsonEH,BakerCI:Contributionsoflow-and high-levelpropertiestoneuralprocessingofvisualscenesinthe humanbrain.PhilosTransR.SocB2017,372:1-11.

57. HebartMN,BanksonBB,HarelA,BakerCI,CichyRM:The representationaldynamicsoftaskandobjectprocessingin humans.eLife2018,7:1-21.

58. SimmonsWK,MartinA:Spontaneousresting-stateBOLD fluctuationsrevealpersistentdomain-specificneural networks.SocCognAffectNeurosci2012,7:467-475.

59. GroenII,GreeneMR,BaldassanoC,Fei-FeiL,BeckDM,BakerCI: Distinctcontributionsoffunctionalanddeepneuralnetwork featurestorepresentationalsimilarityofscenesinhuman brainandbehavior.eLife2018,7:1-26.

60. HsuNS,FranklandSM,Thompson-SchillSL:Chromaticityof colorperceptionandobjectcolorknowledge.

Neuropsychologia2012,50:327-333.

61. AlmeidaJ,MahonBZ,NakayamaK,CaramazzaA:Unconscious processingdissociatesalongcategoricallines.ProcNatlAcad SciUSA2008,105:15214-15218.

62. ChouinardPA,GoodaleMA:Category-specificneural processingfornamingpicturesofanimalsandnaming picturesoftools:anALEmeta-analysis.Neuropsychologia 2010,48:409-418.

63. MahonBZ:Whatisembodiedaboutcognition? LangCogn Neurosci2015,30:420-429.

64. SmithFW,GoodaleMA:Decodingvisualobjectcategoriesin earlysomatosensorycortex.CerebCortex2015,25:1020-1031.

65. MeyerK,KaplanJT,EssexR,WebberC,DamasioH,DamasioA: Predictingvisualstimulionthebasisofactivityinauditory cortices.NatNeurosci2010,13:667-668.

66. XiaoB,BiW,JiaX,WeiH,AdelsonEH:Canyouseewhatyou feel?Colorandfoldingpropertiesaffectvisual–tactilematerial discriminationoffabrics.JVis2016,16:1-34.

67. AmediA,vonKriegsteinK,vanAtteveldtNM,BeauchampMS, NaumerMJ:Functionalimagingofhumancrossmodal identificationandobjectrecognition.ExpBrainRes2005, 166:559-571.

68. LaceyS,SathianK:Visuo-hapticmultisensoryobject recognition,categorization,andrepresentation.FrontPsychol 2014,5:1-15.

69. ManK,DamasioA,MeyerK,KaplanJT:Convergentand invariantobjectrepresentationsforsight,sound,andtouch. HumBrainMapp2015,36:3629-3640.

70. PodrebaracSK,GoodaleMA,SnowJC:Arevisual texture-selectiveareasrecruitedduringhaptictexturediscrimination? Neuroimage2014,94:129-137.

71. SathianKetal.:Dualpathwaysforhapticandvisualperception ofspatialandtextureinformation.Neuroimage2011, 57:462-475.

72. StillaR,SathianK:Selectivevisuo-hapticprocessingofshape andtexture.HumBrainMapp2008,29:1123-1138.

73.

 Bracciobject-S,andDanielscategory-relatedN,OpdeBeeckrepresentationalH:Taskcontextcontentoverrulesinthe humanparietalcortex.CerebCortex2017,27:310-321.

Human fMRI studythat shows that human parietalcortex doesnot preferentiallyrepresentparticularobjectpropertiesirrespectiveoftask, buttogetherwithfrontalareasispartofamultiple-demandand content-richcorticalnetworkrepresentingtask-relevantobjectproperties.The resultspaintsamoredetailedpictureoftherolesofventralanddorsal streamintheprocessingofobjectproperties.

74.

 AndersonBiol2011,BL:21:978-983.Visualperceptionofmaterialsandsurfaces.Curr

Thisprimersummarizesthechallengesinmaterialperception, under-standing howmidlevelvisionorganizes imagemeasurements intoa coherent representationofsurfacesandmaterials.It suggeststhat a simplelinear,progressiveflowfromlow-levelfeaturedetectionto high-levelsceneanalysisfailstocapturethemassiverecurrencethatoccurs throughoutthevisualsystem.Here,‘mid-levelvision’referstothe col-lectiveprocessesthatareinvolvedinmakinginformationaboutsurfaces and materials explicit, rather than a particular region of cortical processing.

75. HaxbyJV,GobbiniMI,FureyML,IshaiA,SchoutenJL,PietriniP: Distributedandoverlappingrepresentationsoffacesand objectsinventraltemporalcortex.Science2001, 293:2425-2430.

76.

 Kriegeskorteanalysis-connectingN,MurM,theBandettinibranchesP:Representationalofsystemsneuroscience.similarity FrontSystNeurosci2008,2:1-28.

Anapproachgrowinginpopularityforinvestigationsofobjectandscene representationsistoexploremultivariatestimulusspaces,forexample, usingstimulussetsthatvaryinmultipledimensionsandcomparingthe similaritystructureofneuralactivationspaceswiththosederivedfrom differentfeaturesaboutthestimulusspace,orhumanbehavior. 77. HebartMN,BakerCI:Deconstructingmultivariatedecodingfor

thestudyofbrainfunction.Neuroimage2017,180:4-18.

78. RitchieJB,KaplanDM,KleinC:Decodingthebrain:neural representationandthelimitsofmultivariatepatternanalysisin cognitiveneuroscience.BrJPhilosSci2017,70:1-27.

79. CichyRM,PantazisD,OlivaA:Similarity-basedfusionofMEG andfMRIrevealsspatio-temporaldynamicsinhumancortex duringvisualobjectrecognition.CerebCortex2016, 26:3563-3579.

80. HarelA,KravitzDJ,BakerCI:Taskcontextimpactsvisualobject processingdifferentiallyacrossthecortex.ProcNatlAcadSci USA2014,111:962-971.

81. KingML,GroenIIA,SteelA,KravitzDJ,BakerCI:Similarity judgmentsandcorticalvisualresponsesreflectdifferent propertiesofobjectandscenecategoriesinnaturalistic images.Neuroimage2019,197:368-382.

82. SchmidAC,BoyaciH,DoerschnerK:DifferentialfMRI responsestomaterialmotioncomparedtoothermotion types.Perception2019,48:205-206http://dx.doi.org/10.1177/ 0301006618824879.