You've way overcomplicated that. Write it with if statements instead like this:
if(liCount == 0)
setLayoutState('start');
else if(liCount<=5)
setLayoutState('upload1Row');
else if(liCount<=10)
setLayoutState('upload2Rows');
$('#UploadList').data('jsp').reinitialise();
Or, if ChaosPandion is trying to optimize as much as possible:
setLayoutState(liCount == 0 ? 'start' :
liCount <= 5 ? 'upload1Row' :
liCount <= 10 ? 'upload2Rows' :
null);
$('#UploadList').data('jsp').reinitialise();