This should work, at least according to the Search Filter Syntax article on MSDN network.
The "hang-up" you have noticed is probably just a delay. Try running the same query with narrower scope (for example the specific OU where the test object is located), as it may take very long time for processing if you run it against all AD objects.
You may also try separating the filter into two parts:
(|(displayName=*searchstring)(displayName=searchstring*))