Yes, this is possible. You don't have to be the current process owning the socket to close it. Consider for a moment that the remote machine, the network card, the network cable, and your OS can all cause the socket to close.
Consider also that Fiddler and Desktop VPN software can insert themselves into the network stack and show you all your traffic or reroute all your traffic.
So all you really need is either for Windows to provide an API that allows this directly, or for someone to have written a program that operates somewhat like a VPN or Fiddler and gives you a way to close sockets that pass through it.
There is at least one program (CurrPorts) that does exactly this and I used it today for the purpose of closing specific sockets on a process that was started before CurrPorts was started. To do this you must run it as administrator, of course.
Note that it is probably not easily possible to cause a program to not listen on a port (well, it is possible but that capability is referred to as a firewall...), but I don't think that was being asked here. I believe the question is "how do I selectively close one active connection (socket) to the port my program is listening on?". The wording of the question is a bit off because a port number for the undesired inbound client connection is given and it was referred to as "port" but it's pretty clear that it was a reference to that one socket and not the listening port.