Move Your Lync/Skype Users Faster

Hypothesis and Assumption

The other night during an upgrade, I had to move 35,000 Lync users from one pool to another Skype for Business pool.  Because it was an upgrade, this wasn’t a simple Invoke-CSPoolFailover situation.  Running a Get-CSUser  | Move-CSUser command suggested the move time was going to be just short of 8 hours, well outside of our maintenance window and well into the morning where users would arrive and potentially notice.

I’ve long suspected that multi-threading the move would help, and I typically run the move across many servers out of hope and superstition, but I could never say for sure if it was real or my imagination that users seemed to move faster.  Since the back end is mirrored, all of these threads on all of these servers are really ultimately beating on the same back end server, was that the bottleneck?  This night was a good night to find out if it was faster for sure.

Testing

I have six servers in the new Skype for Business pool, breaking the users equally across all of these servers, running 4 threads per server I was able to complete all moves within 2 hours.  A HUGE 4x improvement.

I was still curious, was it because I split it across six servers, or was splitting it across multiple threads enough?  I set up a lab with a similar number of users and ran the moves there.   There were four groups of users with names starting with A, B, C, or D respectively.

Running the following command would take about 40 minutes in each direction:

Download Measure1.txt
1
measure-command {Get-CsUser -Filter {Registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com}

However splitting the command up on the same server into four threads sped the move time up so the moves completed on an average of 14 minutes total, nearly 3x faster.

Download Measure2.txt
1
2
3
4
measure-command {Get-CsUser -Filter {FirstName -like 'a*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com}
measure-command {Get-CsUser -Filter {FirstName -like 'b*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com}
measure-command {Get-CsUser -Filter {FirstName -like 'c*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com}
measure-command {Get-CsUser -Filter {FirstName -like 'd*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com}

Splitting the move across four servers didn’t seem to improve the performance much over the existing tests, though in fairness my lab is limited and runs on a single virtual host.  I’d love someone to test further with a real environment, and I will myself, but these large moves don’t happen every night so I’ll need some time before another very large test can occur.

Conclusion

  • Is it worth splitting user moves into multiple processes?  YES!
  • Is it worth splitting across multiple servers?  I strongly suspect so.
  • Is it worth trying to find the point of diminishing returns?  Probably not, we’re talking about hours of time and not days, every system is different so while it’s a fun exercise, it’s somewhat fruitless and therefore not published here.
  • Is it worth creating a script that can programmaticly divide the users to be moved into perfectly equal parts for each thread because you’re a PowerShell nerd?  Not likely, I fought this with myself… we’re again talking about hours of time here, not days.  That said, here’s a sample that splits it up by first name and opens threads in new PowerShell windows, just cut and paste a chunk into an Administrative Skype for Business Management Shell window and let it do the rest of the work for you.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'a*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'b*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'c*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'd*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'e*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'f*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'g*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'h*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'i*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'j*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'k*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'l*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'm*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'n*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'o*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'p*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'q*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'r*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 's*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 't*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'u*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'v*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'w*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'x*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'y*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"
start-process powershell -ArgumentList "Get-CsUser -Filter {FirstName -like 'z*' -and registrarpool -eq 'sourcepool.domain.com'}|move-csuser -target destpool.domain.com;pause"

And for fun, here’s an image of me running the measure-command with results in my own lab:

Capture

7 thoughts on “Move Your Lync/Skype Users Faster

  1. Pingback: NeWay Technologies – Weekly Newsletter #170 – October 22, 2015 | NeWay

  2. Pingback: NeWay Technologies – Weekly Newsletter #170 – October 23, 2015 | NeWay

  3. Peter Van Gils

    Works great, I just migrated 14000 users this way. Perhaps worth mentioning that 10 PowerShell sessions in parallel, evenly spread over two 6-core servers, are enough to keep them running at 95-100% CPU all the time, so don’t run them all at once! Also, you forgot usernames that start with a number. Other than that, this is a great script, thanks a lot!

  4. Jeff Covlin

    So the command in your MultipleThreads.txt will not run, as you would have to confirm all moves. When I try to add the -Confirm:$false, it fails to process the move-csuser command. I also want to append the -moveconferencedata, but have not had any luck getting this to work. Any feedback is appreciated.

    Cheers,
    Jeffcsp

Comments are closed.