You can't make window.open
modal and I strongly recommend you not to go that way.
Instead you can use something like jQuery UI's dialog widget.
UPDATE:
You can use load()
method:
$("#dialog").load("resource.php").dialog({options});
This way it would be faster but the markup will merge into your main document so any submit will be applied on the main window.
And you can use an IFRAME:
$("#dialog").append($("<iframe></iframe>").attr("src", "resource.php")).dialog({options});
This is slower, but will submit independently.