I found a solution wich is to do:
<span class="visible-sm"> your code without col </span>
<span class="visible-xs"> your code with col </span>
It's not very optimized but it works. Did you find something better? It really miss a class like col-sm-0 to apply colons just to the xs size...