You need to add a start anchor ^
as:
^\d{4}$
Your regex \d{4}$
will match strings that end with 4 digits. So input like -1234
will be accepted.
By adding the start anchor you match only those strings that begin and end with 4 digits, which effectively means they must contain only 4 digits.