Windows365 Performance benchmarking

Introduction

This article looks at the performance metrics of Windows 365, aka the cloud PC. The desired outcome of these tests was to understand the overall performance metrics, including limitations for a Cloud PC user.

Benchmark testing can provide an insight into how applications would behave and perform in a production environment. For example, the overall impact to compute resources, including the Virtual Central Processing unit (vCPU), Random Access Memory (RAM), and Disk bandwidth.

Finally, for my understanding, I wanted to understand the user experience and whether the CloudPC (Business version) could be successfully used as a day-to-day solution for the SME market as this would offer a stepping stone solution for SME’s to enter the Virtual Desktop marketspace.

To complete the tests, I used PCMARK 10 From UL. You can find their details at the bottom of the article. I also wanted to thank RDAnalyzer for their community version of Remote Display Analyzer that I used to understand if reverse connect (TCP) was being used as well as the overall round trip latency of the CloudPC .

Here are some of the screenshots from the benchmark testing I completed:

Test Scope:

Testing looked at three specific Windows 365 Cloud PC’s and there was a series of benchmark tests completed. The specs are as follows:

Cloud PC Business 4vcpu/16GB/128GB

Cloud PC Business 2vcpu/8GB/128GB

Cloud PC Business 2vcpu/4GB/128GB

I also wanted to callout the pricing at the time of writing so you as the reader can see the difference in cost between the three different Business version Cloud PCs.

You will note that the cost difference between the 2vCPU/4GB/128GB and 4vCPU/16GB/128GB is approximately £ 29.80 per month. Its suggested that you do size carefully to ensure you select the most suitable spec for both performance and cost. This theme will become more apparent as you progress through the article.

Tests completed on each Cloud PC

This summarises the tests completed to understand what the Cloud PC has to offer.

Essential (basic) Benchmarks
- App Start-up
- Video Conferencing
- Web Browsing
Productivity Benchmarks
- Spreadsheets
- Writing
- Loading Documents
- Saving Documents
- Copy/Paste
- Cut and paste.
Disk Drive Benchmark
- Bandwidth
- Average Access Time
Internet Speeds

It is also important to note that these Cloud PC’s are using Intel CPU’s and not AMD.

Optimisations

Before we dive into the test results I did want to share that each Cloud PC has basic optimisations applied. That’s 83 with a remaining additional 427 possible optimisations you could apply. In most cases I see optimisations as a density benefit by reducing the machine’s in use CPU threads and handles. I will look at optimisation for the Cloud PC at a later date.

The Results:

In this section we look at the results of benchmarks / testing of all three cloud PC specs as detailed in the pervious section “Test Scope”.

Basic Benchmark Testing Completed

Basic benchmark testing looks at three areas, application start-up, Video Conferencing, and Web browsing. To provide a little more context to what these tests means I have listed out the tests in detail:

Application start-up tests both cold and warm application start-up. The differences between the two is cold means the application has not been launched before (since the device reboot or power on).

Video Conferencing looks at the playback Frames per second (FPS) using the CPU. its important to note that I did disable hardware acceleration tests as no GPU is present. All tests we conducted with no HW Acceleration.

Web Browsing benchmarking tests social media page load, shop image view, map zooming, video using H.264 and VP9.

The following Graph shows the results from the basic tests. You will note there is very little difference between the 2vCPU 4GB RAM and 2vCPU 8GB RAM cloud PC’s.

The next section takes a look at productivity benchmark tests that were completed.

Productivity Benchmark testing completed

In this benchmark we looked at using Spreadsheets and Word processing applications to understand how the cloud PC performs for general workload users.

Spreadsheet testing carries out a number of tasks including opening documents, completing recalculation tasks and copying of formulas ect.

Writing testing carries out similar tests associated with Word processing, including adding pictures and cut / copy and past.

You will note from the graph, that it looks like a significant difference between the three Cloud PC’s. However if you take a closer look at the scale you will note the integer differences are not large. Meaning all three Cloud PC’s perform at a similar performance baseline.

In the next section we look at the Disk Benchmark tests completed.

Disk Benchmarks

All three cloud PC’s have the same disk capacity of 128GB. Two tests were completed: Bandwidth, which tests the MB/s on the cloud PC and the second test is average access time in microseconds.

The Bandwidth test shows both the 4/8GB Cloud PC’s offer the same bandwidth, whereas the larger VM 4vCPU’s provides 75.38MB/s. I believe the bandwidth should be much higher, especially for larger applications/ other workloads. I also note that from my own testing, copying files can be slow at times.

When we look at the disk average access time graph you can see there is a difference of 185 microseconds between the 2vCPU4GB/128GB and the 4vCPU16GB/128GB Cloud PC’s.

I also had a quick look at Disk read and write speeds using CrystalDiskMark for the 2vCPU/4GB/128GB Cloud PC:

As you can see they are quite good however the previous tests using PCMARK10 test bandwidth rather than typical read/writes testing.

We now move on to looking at some specific work load tests completed on all three Cloud PC’s.

Workload Testing

The following Graph shows a number of the tests completed on each Cloud PC. These tests include cold application start-up as well as work load testing specifically on Excel.

You will note that each Cloud PC offers similar results, which raises some questions. For example, when we look at the test “Excel Recalculate stock history”, you will note that the 8GB and 16GB RAM cloud PCs show similar results. You will also note all tests show similar tests between the two. So my question is, what is the real difference besides the price.

I am also somewhat surprised that AMD CPU’s have not been introduced to the Cloud PC offering from the outset as the previous testing with Azure Virtual Desktop and in production use has proven AMD has offered better performance outcomes for none GPU virtual desktops.

The next section takes a brief look at internet speeds.

Internet speed tests:

The results of the internet were interesting as I noted that the 2vCPU/8GB/128GB is in the UK which shows higher speeds where as the 4vCPU/16GB/128GB and 2vCPU/4GB/128GB are in the Netherlands. Not sure why two Cloud PCs deployed in the Netherlands however I have reached out for a response.

Cloud PC Spec	Download (Mbps)	Upload (Mbps)	Jitter (ms)
Cloud PC Business 4vcpu/16GB/128GB	439.56	288.99	0.43
Cloud PC Business 2vcpu/8GB/128GB	901.74	807.37	0.48
Cloud PC Business 2vcpu/4GB/128GB	387.33	277.67	0.44

Testing of internet speeds is on going but this gives you an idea of what to expect.

Test Conclusion:

In my opinion, The user experience is good. When test driving, I found the UI to be responsive and would use this when away from the works computer or if I was traveling.

In regards to the Benchmark testing, This showed that there is little difference three different versions, apart from apps that would consume large amounts of RAM or apps that require multiple CPUs to function correctly. I have to say I was a little disappointed on the Disk Bandwidth results as I believe they should be higher. Poor performing disks can impact the overall user experience including RAM and CPU. Using disk intensive applications could have an impact to the Cloud PC.

I am also not sure why AMD CPU’s were not offered at the launch however, I am sure Microsoft will roll these out as we move closer to the next quarter.

I would consider the application you use day to day to understand the RAM requirement. Most users would get away with no more than 4GB of RAM on a Virtual Desktop for basic workloads which raises the question “if you can achieve the desired user experience and performance requirements, why pay more? … Make sure you test your apps.

Finally, I note the sizing recommendations from Microsoft do suggest larger specs for most users however, from testing I don’t believe this is the case. Microsoft do state “ensure experience expectations are met, you should test your deployment with simulation tools.” So it could be suggested that guidance is provided as a baseline.

As this is an initial release and Microsoft have reached trial capacity, I will test again in a few months time to see if any changes have been made.

Useful Links:

Windows 365 size recommendations | Microsoft Docs

Windows 365 planning guide | Microsoft Docs

Windows 365 deployment overview | Microsoft Docs

Use Azure AD to access the Windows 365 APIs in Microsoft Graph | Microsoft Docs

Summary

In this article, we took a look at testing three of the many Cloud PC versions available from launch. The test results show that customers should really get to understand their needs before purchasing cloud PCs to make sure they don’t under spec or over spec the Cloud PC. I do believe there are some improvements needed however, This is a great offering for those who are looking for a simple cost affective access anywhere solution.

One final note, is I would have not been able to complete this research/testing without the help of UL aka PCMark and wanted to say Thank you for allowing me to use their software/others to complete this peace of work.

I hope this helps you with sizing and understanding the differences between the different spec Cloud PC’s.

PCMARK 10 from UL

I wanted to thank UL for providing me with a press licence that allows me to complete this testing. Some may be familiar with UL when building custom PC’s or gaming. Some of the Products include PCMARK 10, 3DMARK and SERVERMARK.

A bit of information about PCMARK 10.

PCMark 10 is the latest version in UL’s series of industry standard
PC benchmarks. PCMark benchmarks measure complete system
performance using tests based on real-world apps and activities. In
PCMark 10, these tests reflect the common tasks performed in the
modern workplace. This makes PCMark 10 an ideal, vendor-neutral
choice for governments and enterprise organizations that buy PCs
in high volumes.

Benchmark test used for this article:

PCMark 10 Express is a shorter benchmark focused on basic work
tasks. It is a good choice for organizations tendering for PCs for
general office use. The PCMark 10 Express benchmark focuses on
the performance needs of a typical office worker. It is a less
demanding test than the main PCMark 10 benchmark

You can find out more about PCMARK 10 from UL here: https://benchmarks.ul.com/

The test results shown in this article are from using PCMark 10 Express. The testing areas included the following: