I took another approach inspired by @john-rees above - especially when his approach started to fail for me at some point. Basically recurse the subtree and sort files by their path-length - delete from longest to the shortest
Get-ChildItem $tfsLocalPath -Recurse | #Find all children
Select-Object FullName,@{Name='PathLength';Expression={($_.FullName.Length)}} | #Calculate the length of their path
Sort-Object PathLength -Descending | #sort by path length descending
%{ Get-Item -LiteralPath $_.FullName } |
Remove-Item -Force
Regarding the -LiteralPath magic, here's another gotchya that may be hitting you: https://superuser.com/q/212808