To answer the initial question (after your edit), you need to unset($images[1][$key]);
Now some more infos how PHP works: You can safely unset elements of the array in foreach loop, and it doesn't matter if you have & or not for the array item. See this code:
$a=[1,2,3,4,5];
foreach($a as $key=>$val)
{
if ($key==3) unset($a[$key]);
}
print_r($a);
This prints:
Array
(
[0] => 1
[1] => 2
[2] => 3
[4] => 5
)
So as you can see, if you unset correct thing within the foreach loop, everything works fine.