:vsp
or :sp
- splits vim into two instance but you cannot use :shell in only one of them.
Why not display another tab of the terminal not another tab of vim. If you like the idea you can try it: Ctrl-shift-t.
and move between them with Ctrl - pageup
and Ctrl - pagedown
If you want just a few shell commands you can make any shell command in vim using !
For example :!./a.out
.