First, I'd make sure that this really is a performance bottleneck, otherwise I'd go with the solution that is cleanest and most expressive.
If it IS a performance bottleneck, just try the different strategies and see what's the quickest. My bet is on creating a new ArrayList and puting the desired objects in that one, discarding the old ArrayList.