So its time to fire-up the second part of my blog on developing a Testing approach to being a Windows Insider. Part one found Here talks about being a Casual tester or hardcore as well as being on the fast or slow rings.
Part 2 is specifically going to address
- Bare metal or Upgrade testing or both
Bare metal testing is when you start your Machine or VM from a formatted disk - nothing is on the machine at all - The image is usually laid down from an ISO or built up from a reference image using a product like MDT or SCCM - As a Windows Insider we don't always get an ISO with every new release so bare metal testing isn't always a reality that makes this possible.
When an ISO is released, I always test a bare metal install on a Physical and virtual machine separately - this tests a couple different items differently than an upgrade does. Discovering hardware on a new install sometimes will bring out unknown devices, compatibility issues and basic install issues. When I do a bare metal I connect:
1) one build to an AD Domain for business testing
2) one build to an Azure AD tenant
3) one build as standalone or connected only to a Live account
In testing that I am testing to see if the newest build has any issues connecting to these core components - you wouldn't expect to see any problems but as my testing here has proven things break in new builds - Its our job as #WindowsInsiders to find that prior to the masses finding that out!
I always take the ISO and integrate it with the ADK (If available!) into the Microsoft Deployment Tool kit - sometimes this is not possible, but a lot of the time it does work. If you want to get serious about imaging MDT is one of the most flexible tools I have used - there are others gaining some popularity including Windows Autopilot via Intune - I will discuss this in a future blog post!
After a bare metal install the first thing that needs to be done after logging in and connecting to what ever back end (as I have listed) is to check the old School Device manager.
After every install the device manager will hold a number of good indicators as to what isn't going to be working - a great reference to start your testing with - If it looks all clear - you move on to your next tests but starting here is critical.
Expect your device manager to show up everything at the root level to be clear
This looks like 100% success but don't be tricked by it - expose each sub component to see what actually is broken, quite often there will be something that is not functioning as it was intended. Most likely you will see these failures on actual hardware devices associated with physical machines - In the case of the device manger to the right, the Bluetooth built into this Lenovo X230 device - which appears to be working here - once exposed shows an issue
On bare metal testing physical devices like Bluetooth, Card readers, sound devices and video cards are usually subject to not show until manufacturer device drivers are located. Microsoft default drivers usually work but in the case of newer Video cards getting the newest driver is always important to support the newest functionality.
Upgrade Testing is the most common form of testing that I am involved in as a #WindowsInsider - Upgrading an operating system is actual more difficult they laying down an OS on bare metal - there are lots of things that can go wrong - The whole process is laid out well in this Microsoft Article - Essentially the following steps are followed:
1) Check for and download a new build via Windows updates
2) Backup settings and user configurations and test for incompatible software
3) Prepare the new WIM that comes from the new build that was just downloaded
4) Reboot
5) Applies the new WIM over top the existing operating system replacing the old
6) Migrate drivers and other core components to the OS
7) Restore the user content back to its locations
8) Finalize and reboot to OS
Microsoft in their article clearly documents and articulates this process - review the link for some very detailed steps!
After the upgrade after I login (I ensure I am connected to Azure AD as my testing has exposed in 19H1 builds it is disconnecting some times) and like the bare metal, I check the Device manager to ensure everything looks good. Before I go any further I always check the Windows Insider Blog on the new build to ensure I know exactly what is new, has changed and more importantly what isn't working! No sense in reporting a known issue or if you are actually being blocked by a known issue.
On each build I have done, I login to the feedback hub and rate the build - The insider team put this in here for a reason and your immediate feedback on the upgrade is extremely important!
I always go through the new features and validate that the new items are correctly working and test out the new added items or functions to ensure there are no hiccups - if I run into an issue I will try and replicated it on one of my other builds, if not I replicate it after a reboot - if you can easily replicate it, this needs to go into the feedback hub ASAP. If you cant replicate it, consider asking some public forums (MS Answers Insiders) or other about what you observed.
One of the biggest issues with upgrade testing is running into incompatible software - If the build is virgin, with no new software installed there is a high probability you will not run into too many issues - when you add software you add a level of complexity (and lets face it, who uses a stock only windows box!)
Software that affects the OS at the core level is liable to cause problems during the upgrade - 3rd party Antivirus software
- 3rd party Encryption Software
- Device Drivers
- Gaming Anti-Cheat software
Testing Bare metal and upgrade testing are both valid and important for you as a #WindowsInsider to perform- make sure you are looking at both - it will improve your overall knowledge and help the process work out any kinks that you find.
Next Part 3 on testing
- x86 or x64 or both (Its still more important then you think!)
Thanks for reading and get ready to do more Great Testing!
Murray