There are different types of shells. The SSH command execution shell is a non-interactive shell, whereas your normal shell is either a login shell or an interactive shell. Description follows, from man bash:
A login shell is one whose first character of argument
zero is a -, or one started with the --login option.
An interactive shell is one started without non-option
arguments and without the -c option whose standard input
and error are both connected to terminals (as determined
by isatty(3)), or one started with the -i option. PS1 is
set and $- includes i if bash is interactive, allowing a
shell script or a startup file to test this state.
The following paragraphs describe how bash executes its
startup files. If any of the files exist but cannot be
read, bash reports an error. Tildes are expanded in file
names as described below under Tilde Expansion in the
EXPANSION section.
When bash is invoked as an interactive login shell, or as
a non-interactive shell with the --login option, it first
reads and executes commands from the file /etc/profile, if
that file exists. After reading that file, it looks for
~/.bash_profile, ~/.bash_login, and ~/.profile, in that
order, and reads and executes commands from the first one
that exists and is readable. The --noprofile option may
be used when the shell is started to inhibit this behav
ior.
When a login shell exits, bash reads and executes commands
from the file ~/.bash_logout, if it exists.
When an interactive shell that is not a login shell is
started, bash reads and executes commands from ~/.bashrc,
if that file exists. This may be inhibited by using the
--norc option. The --rcfile file option will force bash
to read and execute commands from file instead of
~/.bashrc.
When bash is started non-interactively, to run a shell
script, for example, it looks for the variable BASH_ENV in
the environment, expands its value if it appears there,
and uses the expanded value as the name of a file to read
and execute. Bash behaves as if the following command
were executed:
if [ -n "$BASH_ENV" ]; then . "$BASH_ENV"; fi
but the value of the PATH variable is not used to search
for the file name.