Can Powershell Run Commands in Parallel

Question

I have a powershell script to do some batch processing on a bunch of images and I d like to do some parallel processing   Powershell seems to have some background processing options such as start-job  wait-job  etc  but the only good resource I found for doing parallel work was writing the text of a script out and running those  PowerShell Multithreading   Ideally  I d like something akin to parallel foreach in  net 4   Something pretty seemless like   foreach-parallel -threads 4   file in  Get-ChildItem  dir           Do Work     Maybe I d be better off just dropping down to c

User · Answer

There s so many answers to this these days    jobs  or threadjobs in PS 6 7 or the module  start-process workflows powershell api with another runspace invoke-command with multiple computers  which can all be localhost  have to be admin  multiple session  runspace  tabs in the ISE  or remote powershell ISE tabs  Powershell 7 has a foreach-object -parallel as an alternative for  4   Here s workflows with literally a foreach -parallel   workflow work     foreach -parallel   i in 1  3         sleep 5        i done          work  3 done 1 done 2 done   Or a workflow with a parallel block   function sleepfor  time    sleep  time   sleepfor  time done    workflow work     parallel       sleepfor 3     sleepfor 2     sleepfor 1        hi     work   sleepfor 1 done sleepfor 2 done sleepfor 3 done hi   Here s an api with runspaces example    a     PowerShell   Create   AddScript sleep 5  a done    b     PowerShell   Create   AddScript sleep 5  b done    c     PowerShell   Create   AddScript sleep 5  c done    r1  r2  r3     a  b  c  begininvoke     run in background  a EndInvoke  r1    b EndInvoke  r2    c EndInvoke  r3    wait   a  b  c  streams error   check for errors   a  b  c  dispose     clean  a done b done c done

User · Answer

You can execute parallel jobs in Powershell 2 using Background Jobs  Check out Start-Job and the other job cmdlets     Loop through the server list Get-Content  ServerList txt            Define what each job does    ScriptBlock         param  pipelinePassIn       Test-Path     pipelinePassIn c   Something      Start-Sleep 60          Execute the jobs in parallel   Start-Job  ScriptBlock -ArgumentList       Get-Job    Wait for it all to complete While  Get-Job -State  Running       Start-Sleep 10      Getting the information back from the jobs Get-Job   Receive-Job

User · Answer

The answer from Steve Townsend is correct in theory but not in practice as  likwid pointed out  My revised code takes into account the job-context barrier--nothing crosses that barrier by default  The automatic    variable can thus be used in the loop but cannot be used directly within the script block because it is inside a separate context created by the job   To pass variables from the parent context to the child context  use the -ArgumentList parameter on Start-Job to send it and use param inside the script block to receive it    cls   Send in two root directory names  one that exists and one that does not    Should then get a  True  and a  False  result out the end   temp    foo           ScriptBlock           accept the loop variable across the job-context barrier     param  name         Show the loop variable has made it through      Write-Host   processing   name  inside the job         Execute a command     Test-Path    name        Just wait for a bit        Start-Sleep 5          Show the loop variable here is correct   Write-Host  processing             pass the loop variable across the job-context barrier   Start-Job  ScriptBlock -ArgumentList         Wait for all to complete While  Get-Job -State  Running     Start-Sleep 2      Display output from all jobs Get-Job   Receive-Job    Cleanup Remove-Job      I generally like to provide a reference to the PowerShell documentation as supporting evidence but  alas  my search has been fruitless  If you happen to know where context separation is documented  post a comment here to let me know

User · Answer

Backgrounds jobs are expensive to setup and are not reusable  PowerShell MVP Oisin Grehan has a good example of PowerShell multi-threading    10 25 2010 site is down  but accessible via the Web Archive    I e used adapted Oisin script for use in a data loading routine here   http   rsdd codeplex com SourceControl changeset view a6cd657ea2be Invoke-RSDDThreaded ps1

User · Answer

http   gallery technet microsoft com scriptcenter Invoke-Async-Allows-you-to-83b0c9f0  i created an invoke-async which allows you do run multiple script blocks cmdlets functions at the same time  this is great for small jobs  subnet scan or wmi query against 100 s of machines  because the overhead for creating a runspace vs the startup time of start-job is pretty drastic  It can be used like so   with scriptblock    sb    scriptblock   param  system  gwmi win32 operatingsystem -ComputerName  system   select csname caption     servers   Get-Content servers txt    rtn   Invoke-Async -Set  server -SetParam system  -ScriptBlock  sb   just cmdlet function   servers   Get-Content servers txt    rtn   Invoke-Async -Set  servers -SetParam computername -Params   count 1  -Cmdlet Test-Connection -ThreadCount 50

User · Answer

This has been answered thoroughly  Just want to post this method i have created based on Powershell-Jobs as a reference  Jobs are passed on as a list of script-blocks  They can be parameterized  Output of the jobs is color-coded and prefixed with a job-index  just like in a vs-build-process  as this will be used in a build  Can be used to startup multiple servers at a time or running build steps in parallel or so   function Start-Parallel       param           ScriptBlock             Parameter Position   0            ScriptBlock            Object             Alias  quot arguments quot             parameters             jobs    ScriptBlock   ForEach-Object   Start-Job -ScriptBlock    -ArgumentList  parameters        colors    quot Blue quot    quot Red quot    quot Cyan quot    quot Green quot    quot Magenta quot       colorCount    colors Length      try           while    jobs   Where-Object      State -ieq  quot running quot      Measure-Object  Count -gt 0                 jobs   ForEach-Object    i   1                      fgColor    colors   i - 1     colorCount                   out        Receive-Job                  out    out -split  System Environment   NewLine                  out   ForEach-Object                       Write-Host  quot  i gt   quot -NoNewline -ForegroundColor  fgColor                     Write-Host                                                        i                                 finally           Write-Host  quot Stopping Parallel Jobs     quot  -NoNewline          jobs   Stop-Job          jobs   Remove-Job -Force         Write-Host  quot  done  quot           sample output

User · Answer

To complete previous answers  you can also use Wait-Job to wait for all jobs to complete   For   i 1   i -le 3   i           ScriptBlock             Param                string   Parameter Mandatory  true    increment                    Write-Host  increment            Start-Job  ScriptBlock -ArgumentList  i    Get-Job   Wait-Job   Receive-Job

User · Answer

If you re using latest cross platform powershell  which you should btw  https   github com powershell powershell get-powershell  you can add single  amp  to run parallel scripts   Use   to run sequentially  In my case I needed to run 2 npm scripts in parallel  npm run hotReload  amp  npm run dev  You can also setup npm to use powershell for its scripts  by default it uses cmd on windows   Run from project root folder  npm config set script-shell pwsh --userconfig    npmrc and then use single npm script command  npm run start  quot start quot   quot npm run hotReload  amp  npm run dev quot

User · Answer

In Powershell 7 you can use ForEach-Object -Parallel   Message    Output   Get-ChildItem  dir   ForEach-Object -Parallel         using Message       -ThrottleLimit 4

[multithreading] Can Powershell Run Commands in Parallel?

Examples related to multithreading

Examples related to powershell

Examples related to parallel-processing